Re: [Declude.JunkMail] method for reducing CPU load
This is an excellent suggestion. I can't wait to see it implemented. In the mean time, it's worth taking a look at WeightGate.exe (FREE). http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.Tools I am personally using it with Message Sniffer and invURIBL with great success. - Chris -- Midtown Micro, Inc. (TM) Programming and Web Hosting http://www.MidtownMicro.com Toll Free: 1-800-442-2447 Voice: (916) 442-2447 - Original Message - From: Scott Fisher To: Declude.JunkMail@declude.com Cc: Support - Scott Sent: Tuesday, November 28, 2006 7:43 AM Subject: [Declude.JunkMail] method for reducing CPU load I've been mulling this one over as I watch my spam filtering CPU time slowly taking over the email server. And I don't expect the number of emails to go down. For external programs and filters I think it would be a good idea to add two optional fields to the global.cfg definition line: a minweight and a maxweight. These would be the last two arguments and optional so existing configs would not need to be changed. For an external program: INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 would become INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 -50 300 in this case invuribl would only get run if the current weight was between -50 and 300. For a filter: ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 would become ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 -50 300 in this case the attachment-gif filter would only get processed if the current weight was between -50 and 300 Here's why I think this is a good idea: Declude could check the weights before launching the external program. If it is over/under weight the external program would not be launched. 2 if statements to avoid launching a program. That seems like a CPU time saver. Especially when multiplied by 10,000s of emails per day. I use 6 external programs. I believe over half of the program launches would be avoided because of stuff that has already been declared obvious ham or obvious spam. My final of the 6 programs, gets weight skipped over 90% of the time. At 10,000 emails a day, avoiding 50% of the external programs would save 30,000 program launches a day. I believe my 50% to be a conservative number and I think that the percentage would average out to be even higher. Now I have about one hundred filters. The vast majority of them get triggered with the skipweight since the email is already at a high spam weight by the time it reaches the filters. But still every one of these filter files needs to be opened, read and closed for every email. Again 2 IF statements per filter could avoid opening 100 files. That seems to me to be a CPU time saver. By the time, email reaches the filters, I think 75% of it is bypassing filters by being over the skipweight. At 10,000 emails a day (small to many of us). That would mean 750,000 filter files a day would not need to be open, read and closed. From the programming side, I don't believe the coding changes to be too difficult. Weight verification/processing code already exists in the Declude program. It would just need to be relocated. I'm a pretty small user here, getting about 14,000 spams on a weekday. Imagine the potential CPU savings for scaling this up to an ISP with 100,000 emails per day. I don't know if this would have an impact on saving my CPU or not, but it has to help even if it is a little. Please consider this. - Scott Fisher Director of IT Farm Progress Companies 191 S Gary Ave Carol Stream, IL 60188 630-462-2323 This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Although Farm Progress Companies has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] method for reducing CPU load
Anyway, this idea would probably help out a great deal. PirateFish sounds good, although it looks like you are just buying support and an ebook based on http://www.howtoforge.com/linux_spam_filter_mail_gateway "The Piratefish system is a set of instructions on how to construct an anti-spam gateway system using a free computer operating system called Linux. The instructions will walk you through downloading and creating a Linux OS installation CD, then using that CD to create an anti-spam, anti-virus email gateway system. As you build the Piratefish, you also learn about Linux, and about how all the various open-source programs work together to protect your network from spam." Len's IMGate is very good too if you have a spare machine and he can be contracted to configure it for you. The opensource ASSP can also be put on each of your existing Imail servers or on a spare machine running any *nix, Mac, or Windows OS. Putting any sort of gateway that rejects email to invalid addresses and employs greylisting (or delaying ala ASSP) will take a large chunk of the load off your Imail/Declude installations. Doug Traylor --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Declude v2.06 and Imail 2006.1
MessageAs Matt said - Imail 8.22+ requires Declude 3+. So if you end up trying to use 2.x under 2006 you may or may not have issues. Darrell Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. - Original Message - From: Sharyn Schmidt To: declude.junkmail@declude.com Sent: Tuesday, November 28, 2006 3:05 PM Subject: RE: [Declude.JunkMail] Declude v2.06 and Imail 2006.1 Um, I did... (in the subject line) Decluded v2.06 and Imail 2006.1 Sharyn, You should specify what version of Declude you are asking about. FYI, IMail 8.2+ requires Declude 3+. Some claim that older versions of Declude will work, however there are also widely reported problems with IMail 8.2+ and it is no doubt safest to run Declude 3+. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Declude v2.06 and Imail 2006.1
Um, I did... (in the subject line) Decluded v2.06 and Imail 2006.1 Sharyn, You should specify what version of Declude you are asking about. FYI, IMail 8.2+ requires Declude 3+. Some claim that older versions of Declude will work, however there are also widely reported problems with IMail 8.2+ and it is no doubt safest to run Declude 3+. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Declude v2.06 and Imail 2006.1
Sharyn, You should specify what version of Declude you are asking about. FYI, IMail 8.2+ requires Declude 3+. Some claim that older versions of Declude will work, however there are also widely reported problems with IMail 8.2+ and it is no doubt safest to run Declude 3+. Matt Sharyn Schmidt wrote: Will my old version of Declude work with the new version of Imail? TIA, Sharyn --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] method for reducing CPU load
I think this would be fantastic. Twice now this week I've had to comment out sniffer and invuribl because the proc folder was swelling. Some of my customers are grumbling because the service I've provided for years isn't as good anymore. Right now I'm thinking of putting a PirateFish type postfix server in front of my two declude powered mail servers to try and lessen the work they do. Anyone here use PirateFish? Anyway, this idea would probably help out a great deal. -- Michael Cummins > Matt: > This is exactly what I have wanted Declude to do for over two > years now all the way down to the spec. >> Scott Fisher wrote: >> I've been mulling this one over as I watch my spam filtering CPU >> time slowly taking over the email server. And I don't expect the >> number of emails to go down. >> >> For external programs and filters I think it would be a good idea to >> add two optional fields to the global.cfg definition line: a >> minweight and a maxweight. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] method for reducing CPU load
Ditto. I've been wanting the same functionality. Darin. - Original Message - From: Matt To: declude.junkmail@declude.com Sent: Tuesday, November 28, 2006 1:34 PM Subject: Re: [Declude.JunkMail] method for reducing CPU load Scott, This is exactly what I have wanted Declude to do for over two years now all the way down to the spec. This would add low and high skip weight functionality to all filters, external apps and anything else that might be skippable, and do so by using the config in such a way that could easily be backwards compatible when not specified (you would assume null null to be 0 0 and consider that disabled). This would save tremendously in loading filter files and launching external apps. I for instance have this functionality in one of my external apps, but it still has to be run in order for the weight skipping mechanism to operate. Other external apps have no weight skipping built into them and this would add the much needed functionality to save resources. Matt Scott Fisher wrote: I've been mulling this one over as I watch my spam filtering CPU time slowly taking over the email server. And I don't expect the number of emails to go down. For external programs and filters I think it would be a good idea to add two optional fields to the global.cfg definition line: a minweight and a maxweight. These would be the last two arguments and optional so existing configs would not need to be changed. For an external program: INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 would become INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 -50 300 in this case invuribl would only get run if the current weight was between -50 and 300. For a filter: ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 would become ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 -50 300 in this case the attachment-gif filter would only get processed if the current weight was between -50 and 300 Here's why I think this is a good idea: Declude could check the weights before launching the external program. If it is over/under weight the external program would not be launched. 2 if statements to avoid launching a program. That seems like a CPU time saver. Especially when multiplied by 10,000s of emails per day. I use 6 external programs. I believe over half of the program launches would be avoided because of stuff that has already been declared obvious ham or obvious spam. My final of the 6 programs, gets weight skipped over 90% of the time. At 10,000 emails a day, avoiding 50% of the external programs would save 30,000 program launches a day. I believe my 50% to be a conservative number and I think that the percentage would average out to be even higher. Now I have about one hundred filters. The vast majority of them get triggered with the skipweight since the email is already at a high spam weight by the time it reaches the filters. But still every one of these filter files needs to be opened, read and closed for every email. Again 2 IF statements per filter could avoid opening 100 files. That seems to me to be a CPU time saver. By the time, email reaches the filters, I think 75% of it is bypassing filters by being over the skipweight. At 10,000 emails a day (small to many of us). That would mean 750,000 filter files a day would not need to be open, read and closed. From the programming side, I don't believe the coding changes to be too difficult. Weight verification/processing code already exists in the Declude program. It would just need to be relocated. I'm a pretty small user here, getting about 14,000 spams on a weekday. Imagine the potential CPU savings for scaling this up to an ISP with 100,000 emails per day. I don't know if this would have an impact on saving my CPU or not, but it has to help even if it is a little. Please consider this. - Scott Fisher Director of IT Farm Progress Companies 191 S Gary Ave Carol Stream, IL 60188 630-462-2323 This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Although Farm Progress Companies has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came f
Re: [Declude.JunkMail] method for reducing CPU load
Scott, This is _exactly_ what I have wanted Declude to do for over two years now all the way down to the spec. This would add low and high skip weight functionality to all filters, external apps and anything else that might be skippable, and do so by using the config in such a way that could easily be backwards compatible when not specified (you would assume null null to be 0 0 and consider that disabled). This would save tremendously in loading filter files and launching external apps. I for instance have this functionality in one of my external apps, but it still has to be run in order for the weight skipping mechanism to operate. Other external apps have no weight skipping built into them and this would add the much needed functionality to save resources. Matt Scott Fisher wrote: I've been mulling this one over as I watch my spam filtering CPU time slowly taking over the email server. And I don't expect the number of emails to go down. For external programs and filters I think it would be a good idea to add two optional fields to the global.cfg definition line: a minweight and a maxweight. These would be the last two arguments and optional so existing configs would not need to be changed. For an external program: INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 would become INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 -50 300 in this case invuribl would only get run if the current weight was between -50 and 300. For a filter: ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 would become ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 -50 300 in this case the attachment-gif filter would only get processed if the current weight was between -50 and 300 Here's why I think this is a good idea: Declude could check the weights before launching the external program. If it is over/under weight the external program would not be launched. 2 if statements to avoid launching a program. That seems like a CPU time saver. Especially when multiplied by 10,000s of emails per day. I use 6 external programs. I believe over half of the program launches would be avoided because of stuff that has already been declared obvious ham or obvious spam. My final of the 6 programs, gets weight skipped over 90% of the time. At 10,000 emails a day, avoiding 50% of the external programs would save 30,000 program launches a day. I believe my 50% to be a conservative number and I think that the percentage would average out to be even higher. Now I have about one hundred filters. The vast majority of them get triggered with the skipweight since the email is already at a high spam weight by the time it reaches the filters. But still every one of these filter files needs to be opened, read and closed for every email. Again 2 IF statements per filter could avoid opening 100 files. That seems to me to be a CPU time saver. By the time, email reaches the filters, I think 75% of it is bypassing filters by being over the skipweight. At 10,000 emails a day (small to many of us). That would mean 750,000 filter files a day would not need to be open, read and closed. From the programming side, I don't believe the coding changes to be too difficult. Weight verification/processing code already exists in the Declude program. It would just need to be relocated. I'm a pretty small user here, getting about 14,000 spams on a weekday. Imagine the potential CPU savings for scaling this up to an ISP with 100,000 emails per day. I don't know if this would have an impact on saving my CPU or not, but it has to help even if it is a little. Please consider this. - Scott Fisher Director of IT Farm Progress Companies 191 S Gary Ave Carol Stream, IL 60188 630-462-2323 This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Although Farm Progress Companies has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
[Declude.JunkMail] method for reducing CPU load
I've been mulling this one over as I watch my spam filtering CPU time slowly taking over the email server. And I don't expect the number of emails to go down. For external programs and filters I think it would be a good idea to add two optional fields to the global.cfg definition line: a minweight and a maxweight. These would be the last two arguments and optional so existing configs would not need to be changed. For an external program: INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 would become INV-URIBL external 25 "D:\INVURIBL.exe %WEIGHT% %REMOTEIP%" 25 0 -50 300 in this case invuribl would only get run if the current weight was between -50 and 300. For a filter: ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 would become ATTACHMENT-GIF filterD:\ATTACHMENT-GIF.txt x 0 0 -50 300 in this case the attachment-gif filter would only get processed if the current weight was between -50 and 300 Here's why I think this is a good idea: Declude could check the weights before launching the external program. If it is over/under weight the external program would not be launched. 2 if statements to avoid launching a program. That seems like a CPU time saver. Especially when multiplied by 10,000s of emails per day. I use 6 external programs. I believe over half of the program launches would be avoided because of stuff that has already been declared obvious ham or obvious spam. My final of the 6 programs, gets weight skipped over 90% of the time. At 10,000 emails a day, avoiding 50% of the external programs would save 30,000 program launches a day. I believe my 50% to be a conservative number and I think that the percentage would average out to be even higher. Now I have about one hundred filters. The vast majority of them get triggered with the skipweight since the email is already at a high spam weight by the time it reaches the filters. But still every one of these filter files needs to be opened, read and closed for every email. Again 2 IF statements per filter could avoid opening 100 files. That seems to me to be a CPU time saver. By the time, email reaches the filters, I think 75% of it is bypassing filters by being over the skipweight. At 10,000 emails a day (small to many of us). That would mean 750,000 filter files a day would not need to be open, read and closed. >From the programming side, I don't believe the coding changes to be too >difficult. Weight verification/processing code already exists in the Declude >program. It would just need to be relocated. I'm a pretty small user here, getting about 14,000 spams on a weekday. Imagine the potential CPU savings for scaling this up to an ISP with 100,000 emails per day. I don't know if this would have an impact on saving my CPU or not, but it has to help even if it is a little. Please consider this. - Scott Fisher Director of IT Farm Progress Companies 191 S Gary Ave Carol Stream, IL 60188 630-462-2323 This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Although Farm Progress Companies has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
