[Declude.JunkMail] method for reducing CPU load

2006-11-28 Thread Scott Fisher
I've been mulling this one over as I watch my spam filtering CPU time slowly 
taking over the email server. And I don't expect the number of emails to go 
down.

For external programs and filters I think it would be a good idea to add two 
optional fields to the global.cfg definition line: a minweight and a maxweight. 
These would be the last two arguments and optional so existing configs would 
not need to be changed.

For an external program:
INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0
would become
INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0  -50  300
in this case invuribl would only get run if the current weight was between -50 
and 300.

For a filter:
ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0 
would become
ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0   -50  300
in this case the attachment-gif filter would only get processed if the current 
weight was between -50 and 300

Here's why I think this is a good idea:
Declude could check the weights before launching the external program. If it is 
over/under weight the external program would not be launched.
2 if statements to avoid launching a program. That seems like a CPU time saver. 
Especially when multiplied by 10,000s of emails per day.
I use 6 external programs. I believe over half of the program launches would be 
avoided because of stuff that has already been declared obvious ham or obvious 
spam.
My final of the 6 programs, gets weight skipped over 90% of the time. 
At 10,000 emails a day, avoiding 50% of the external programs would save 30,000 
program launches a day. I believe my 50% to be a conservative number and I 
think that the percentage would average out to be even higher.

Now I have about one hundred filters. The vast majority of them get triggered 
with the skipweight since the email is already at a high spam weight by the 
time it reaches the filters.
But still every one of these filter files needs to be opened, read and closed 
for every email.
Again 2 IF statements per filter could avoid opening 100 files. That seems to 
me to be a CPU time saver.
By the time, email reaches the filters, I think 75% of it is bypassing filters 
by being over the skipweight. At 10,000 emails a day (small to many of us). 
That would mean 750,000 filter files a day would not need to be open, read and 
closed.

From the programming side, I don't believe the coding changes to be too 
difficult. Weight verification/processing code already exists in the Declude 
program. It would just need to be relocated.

I'm a pretty small user here, getting about 14,000 spams on a weekday. Imagine 
the potential CPU savings for scaling this up to an ISP with 100,000 emails per 
day.

I don't know if this would have an impact on saving my CPU or not, but it has 
to help even if it is a little.
Please consider this.

-
Scott Fisher
Director of IT
Farm Progress Companies
191 S Gary Ave
Carol Stream, IL 60188
630-462-2323

This email message, including any attachments, is for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
Any unauthorized review, use, disclosure or distribution is prohibited. If you 
are not the intended recipient, please contact the sender by reply email and 
destroy all copies of the original message. Although Farm Progress Companies 
has taken reasonable precautions to ensure no viruses are present in this 
email, the company cannot accept responsibility for any loss or damage arising 
from the use of this email or attachments.



---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.


Re: [Declude.JunkMail] method for reducing CPU load

2006-11-28 Thread Matt

Scott,

This is _exactly_ what I have wanted Declude to do for over two years 
now all the way down to the spec.  This would add low and high skip 
weight functionality to all filters, external apps and anything else 
that might be skippable, and do so by using the config in such a way 
that could easily be backwards compatible when not specified (you would 
assume null null to be 0 0 and consider that disabled).


This would save tremendously in loading filter files and launching 
external apps.  I for instance have this functionality in one of my 
external apps, but it still has to be run in order for the weight 
skipping mechanism to operate.  Other external apps have no weight 
skipping built into them and this would add the much needed 
functionality to save resources.


Matt



Scott Fisher wrote:
I've been mulling this one over as I watch my spam filtering CPU time 
slowly taking over the email server. And I don't expect the number of 
emails to go down.
 
For external programs and filters I think it would be a good idea to 
add two optional fields to the global.cfg definition line: a minweight 
and a maxweight. These would be the last two arguments and optional so 
existing configs would not need to be changed.
 
For an external program:

INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0
would become
INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0  
-50  300
in this case invuribl would only get run if the current weight was 
between -50 and 300.

For a filter:
ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0 
would become

ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0   -50  300
in this case the attachment-gif filter would only get processed if the 
current weight was between -50 and 300


Here's why I think this is a good idea:
Declude could check the weights before launching the external program. 
If it is over/under weight the external program would not be launched.
2 if statements to avoid launching a program. That seems like a CPU 
time saver. Especially when multiplied by 10,000s of emails per day.
I use 6 external programs. I believe over half of the program launches 
would be avoided because of stuff that has already been declared 
obvious ham or obvious spam.

My final of the 6 programs, gets weight skipped over 90% of the time.
At 10,000 emails a day, avoiding 50% of the external programs would 
save 30,000 program launches a day. I believe my 50% to be a 
conservative number and I think that the percentage would average out 
to be even higher.
 
Now I have about one hundred filters. The vast majority of them get 
triggered with the skipweight since the email is already at a high 
spam weight by the time it reaches the filters.
But still every one of these filter files needs to be opened, read and 
closed for every email.
Again 2 IF statements per filter could avoid opening 100 files. That 
seems to me to be a CPU time saver.
By the time, email reaches the filters, I think 75% of it is bypassing 
filters by being over the skipweight. At 10,000 emails a day (small to 
many of us). That would mean 750,000 filter files a day would not need 
to be open, read and closed.
 
From the programming side, I don't believe the coding changes to be 
too difficult. Weight verification/processing code already exists in 
the Declude program. It would just need to be relocated.
 
I'm a pretty small user here, getting about 14,000 spams on a weekday. 
Imagine the potential CPU savings for scaling this up to an ISP with 
100,000 emails per day.
 
I don't know if this would have an impact on saving my CPU or not, but 
it has to help even if it is a little.

Please consider this.

-
Scott Fisher
Director of IT
Farm Progress Companies
191 S Gary Ave
Carol Stream, IL 60188
630-462-2323
 
This email message, including any attachments, is for the sole use of 
the intended recipient(s) and may contain confidential and privileged 
information. Any unauthorized review, use, disclosure or distribution 
is prohibited. If you are not the intended recipient, please contact 
the sender by reply email and destroy all copies of the original 
message. Although Farm Progress Companies has taken reasonable 
precautions to ensure no viruses are present in this email, the 
company cannot accept responsibility for any loss or damage arising 
from the use of this email or attachments.
 
 


---
This E-mail came from the Declude.JunkMail mailing list. To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail. The archives can be found
at http://www.mail-archive.com. 



---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.

Re: [Declude.JunkMail] method for reducing CPU load

2006-11-28 Thread Darin Cox
Ditto.  I've been wanting the same functionality.

Darin.


- Original Message - 
From: Matt 
To: declude.junkmail@declude.com 
Sent: Tuesday, November 28, 2006 1:34 PM
Subject: Re: [Declude.JunkMail] method for reducing CPU load


Scott,

This is exactly what I have wanted Declude to do for over two years now all the 
way down to the spec.  This would add low and high skip weight functionality to 
all filters, external apps and anything else that might be skippable, and do so 
by using the config in such a way that could easily be backwards compatible 
when not specified (you would assume null null to be 0 0 and consider that 
disabled).

This would save tremendously in loading filter files and launching external 
apps.  I for instance have this functionality in one of my external apps, but 
it still has to be run in order for the weight skipping mechanism to operate.  
Other external apps have no weight skipping built into them and this would add 
the much needed functionality to save resources.

Matt



Scott Fisher wrote: 
  I've been mulling this one over as I watch my spam filtering CPU time slowly 
taking over the email server. And I don't expect the number of emails to go 
down.

  For external programs and filters I think it would be a good idea to add two 
optional fields to the global.cfg definition line: a minweight and a maxweight. 
These would be the last two arguments and optional so existing configs would 
not need to be changed.

  For an external program:
  INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0
  would become
  INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0  -50  300
  in this case invuribl would only get run if the current weight was between 
-50 and 300.

  For a filter:
  ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0 
  would become
  ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0   -50  300
  in this case the attachment-gif filter would only get processed if the 
current weight was between -50 and 300

  Here's why I think this is a good idea:
  Declude could check the weights before launching the external program. If it 
is over/under weight the external program would not be launched.
  2 if statements to avoid launching a program. That seems like a CPU time 
saver. Especially when multiplied by 10,000s of emails per day.
  I use 6 external programs. I believe over half of the program launches would 
be avoided because of stuff that has already been declared obvious ham or 
obvious spam.
  My final of the 6 programs, gets weight skipped over 90% of the time. 
  At 10,000 emails a day, avoiding 50% of the external programs would save 
30,000 program launches a day. I believe my 50% to be a conservative number and 
I think that the percentage would average out to be even higher.

  Now I have about one hundred filters. The vast majority of them get triggered 
with the skipweight since the email is already at a high spam weight by the 
time it reaches the filters.
  But still every one of these filter files needs to be opened, read and closed 
for every email.
  Again 2 IF statements per filter could avoid opening 100 files. That seems to 
me to be a CPU time saver.
  By the time, email reaches the filters, I think 75% of it is bypassing 
filters by being over the skipweight. At 10,000 emails a day (small to many of 
us). That would mean 750,000 filter files a day would not need to be open, read 
and closed.

  From the programming side, I don't believe the coding changes to be too 
difficult. Weight verification/processing code already exists in the Declude 
program. It would just need to be relocated.

  I'm a pretty small user here, getting about 14,000 spams on a weekday. 
Imagine the potential CPU savings for scaling this up to an ISP with 100,000 
emails per day.

  I don't know if this would have an impact on saving my CPU or not, but it has 
to help even if it is a little.
  Please consider this.

  -
  Scott Fisher
  Director of IT
  Farm Progress Companies
  191 S Gary Ave
  Carol Stream, IL 60188
  630-462-2323

  This email message, including any attachments, is for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
Any unauthorized review, use, disclosure or distribution is prohibited. If you 
are not the intended recipient, please contact the sender by reply email and 
destroy all copies of the original message. Although Farm Progress Companies 
has taken reasonable precautions to ensure no viruses are present in this 
email, the company cannot accept responsibility for any loss or damage arising 
from the use of this email or attachments.



  ---
  This E-mail came from the Declude.JunkMail mailing list. To
  unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
  type unsubscribe Declude.JunkMail. The archives can be found
  at http://www.mail-archive.com. 

---
This E-mail came from 

RE: [Declude.JunkMail] method for reducing CPU load

2006-11-28 Thread Michael Cummins
I think this would be fantastic.  Twice now this week I've had to comment
out sniffer and invuribl because the proc folder was swelling.  Some of my
customers are grumbling because the service I've provided for years isn't as
good anymore.  Right now I'm thinking of putting a PirateFish type postfix
server in front of my two declude powered mail servers to try and lessen the
work they do.

Anyone here use PirateFish?

Anyway, this idea would probably help out a great deal.

-- Michael Cummins


 Matt:
 This is exactly what I have wanted Declude to do for over two 
 years now all the way down to the spec. 

 Scott Fisher wrote: 
 I've been mulling this one over as I watch my spam filtering CPU 
 time slowly taking over the email server. And I don't expect the 
 number of emails to go down.
  
 For external programs and filters I think it would be a good idea to 
 add two optional fields to the global.cfg definition line: a 
 minweight and a maxweight. 



---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.



Re: [Declude.JunkMail] Declude v2.06 and Imail 2006.1

2006-11-28 Thread Matt

Sharyn,

You should specify what version of Declude you are asking about.  FYI, 
IMail 8.2+ requires Declude 3+.  Some claim that older versions of 
Declude will work, however there are also widely reported problems with 
IMail 8.2+ and it is no doubt safest to run Declude 3+.


Matt



Sharyn Schmidt wrote:


Will my old version of Declude work with the new version of Imail?

TIA,
Sharyn


---
This E-mail came from the Declude.JunkMail mailing list. To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail. The archives can be found
at http://www.mail-archive.com. 



---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.

RE: [Declude.JunkMail] Declude v2.06 and Imail 2006.1

2006-11-28 Thread Sharyn Schmidt
Um, I did... (in the subject line)
 
Decluded v2.06 and Imail 2006.1
 
 

 
Sharyn,

You should specify what version of Declude you are asking about.  FYI, IMail
8.2+ requires Declude 3+.  Some claim that older versions of Declude will
work, however there are also widely reported problems with IMail 8.2+ and it
is no doubt safest to run Declude 3+.





---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.


Re: [Declude.JunkMail] Declude v2.06 and Imail 2006.1

2006-11-28 Thread Darrell \([EMAIL PROTECTED])
MessageAs Matt said - Imail 8.22+ requires Declude 3+.  So if you end up trying 
to use 2.x under 2006 you may or may not have issues.

Darrell 

Check out http://www.invariantsystems.com for utilities for Declude And Imail.  
IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG 
Integration, and Log Parsers.
  - Original Message - 
  From: Sharyn Schmidt 
  To: declude.junkmail@declude.com 
  Sent: Tuesday, November 28, 2006 3:05 PM
  Subject: RE: [Declude.JunkMail] Declude v2.06 and Imail 2006.1


  Um, I did... (in the subject line)

  Decluded v2.06 and Imail 2006.1



Sharyn,

You should specify what version of Declude you are asking about.  FYI, 
IMail 8.2+ requires Declude 3+.  Some claim that older versions of Declude will 
work, however there are also widely reported problems with IMail 8.2+ and it is 
no doubt safest to run Declude 3+.



  ---
  This E-mail came from the Declude.JunkMail mailing list. To
  unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
  type unsubscribe Declude.JunkMail. The archives can be found
  at http://www.mail-archive.com. 

---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.


Re: [Declude.JunkMail] method for reducing CPU load

2006-11-28 Thread Doug Traylor

Anyway, this idea would probably help out a great deal.


PirateFish sounds good, although it looks like you are just buying
support and an ebook based on
http://www.howtoforge.com/linux_spam_filter_mail_gateway

The Piratefish system is a set of instructions on how to construct an
anti-spam gateway system using a free computer operating system called
Linux. The instructions will walk you through downloading and creating
a Linux OS installation CD, then using that CD to create an anti-spam,
anti-virus email gateway system. As you build the Piratefish, you also
learn about Linux, and about how all the various open-source programs
work together to protect your network from spam.

Len's IMGate is very good too if you have a spare machine and he can
be contracted to configure it for you.  The opensource ASSP can also
be put on each of your existing Imail servers or on a spare machine
running any *nix, Mac, or Windows OS.

Putting any sort of gateway that rejects email to invalid addresses
and employs greylisting (or delaying ala ASSP) will take a large chunk
of the load off your Imail/Declude installations.

Doug Traylor


---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.



Re: [Declude.JunkMail] method for reducing CPU load

2006-11-28 Thread Christopher Jaime
This is an excellent suggestion.  I can't wait to see it implemented.

In the mean time, it's worth taking a look at WeightGate.exe (FREE).
http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.Tools

I am personally using it with Message Sniffer and invURIBL with great success.

- Chris

--
Midtown Micro, Inc. (TM)
Programming and Web Hosting
http://www.MidtownMicro.com
Toll Free: 1-800-442-2447
Voice: (916) 442-2447

  - Original Message - 
  From: Scott Fisher 
  To: Declude.JunkMail@declude.com 
  Cc: Support - Scott 
  Sent: Tuesday, November 28, 2006 7:43 AM
  Subject: [Declude.JunkMail] method for reducing CPU load


  I've been mulling this one over as I watch my spam filtering CPU time slowly 
taking over the email server. And I don't expect the number of emails to go 
down.

  For external programs and filters I think it would be a good idea to add two 
optional fields to the global.cfg definition line: a minweight and a maxweight. 
These would be the last two arguments and optional so existing configs would 
not need to be changed.

  For an external program:
  INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0
  would become
  INV-URIBL external  25 D:\INVURIBL.exe %WEIGHT% %REMOTEIP%  25 0  -50  300
  in this case invuribl would only get run if the current weight was between 
-50 and 300.

  For a filter:
  ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0 
  would become
  ATTACHMENT-GIF  filterD:\ATTACHMENT-GIF.txt   x   0   0   -50  300
  in this case the attachment-gif filter would only get processed if the 
current weight was between -50 and 300

  Here's why I think this is a good idea:
  Declude could check the weights before launching the external program. If it 
is over/under weight the external program would not be launched.
  2 if statements to avoid launching a program. That seems like a CPU time 
saver. Especially when multiplied by 10,000s of emails per day.
  I use 6 external programs. I believe over half of the program launches would 
be avoided because of stuff that has already been declared obvious ham or 
obvious spam.
  My final of the 6 programs, gets weight skipped over 90% of the time. 
  At 10,000 emails a day, avoiding 50% of the external programs would save 
30,000 program launches a day. I believe my 50% to be a conservative number and 
I think that the percentage would average out to be even higher.

  Now I have about one hundred filters. The vast majority of them get triggered 
with the skipweight since the email is already at a high spam weight by the 
time it reaches the filters.
  But still every one of these filter files needs to be opened, read and closed 
for every email.
  Again 2 IF statements per filter could avoid opening 100 files. That seems to 
me to be a CPU time saver.
  By the time, email reaches the filters, I think 75% of it is bypassing 
filters by being over the skipweight. At 10,000 emails a day (small to many of 
us). That would mean 750,000 filter files a day would not need to be open, read 
and closed.

  From the programming side, I don't believe the coding changes to be too 
difficult. Weight verification/processing code already exists in the Declude 
program. It would just need to be relocated.

  I'm a pretty small user here, getting about 14,000 spams on a weekday. 
Imagine the potential CPU savings for scaling this up to an ISP with 100,000 
emails per day.

  I don't know if this would have an impact on saving my CPU or not, but it has 
to help even if it is a little.
  Please consider this.

  -
  Scott Fisher
  Director of IT
  Farm Progress Companies
  191 S Gary Ave
  Carol Stream, IL 60188
  630-462-2323

  This email message, including any attachments, is for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
Any unauthorized review, use, disclosure or distribution is prohibited. If you 
are not the intended recipient, please contact the sender by reply email and 
destroy all copies of the original message. Although Farm Progress Companies 
has taken reasonable precautions to ensure no viruses are present in this 
email, the company cannot accept responsibility for any loss or damage arising 
from the use of this email or attachments.



  ---
  This E-mail came from the Declude.JunkMail mailing list. To
  unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
  type unsubscribe Declude.JunkMail. The archives can be found
  at http://www.mail-archive.com. 

---
This E-mail came from the Declude.JunkMail mailing list.  To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type unsubscribe Declude.JunkMail.  The archives can be found
at http://www.mail-archive.com.