RE: [Declude.JunkMail] Declude using 50% cpu

[Colbeck, Andrew]

Title: Message

Ah, I do see why you've moved the ip4r tests to IMail.  I do remember that earlier discussion of ip4r efficiency, but I didn't give it my full attention (I was slogging through the backlog that came in while I was on vacation...) I had concluded that the decision to move ip4r tests from Declude to IMail was moot.

 

Now, however, I do see the value, which is that you have another testing layer above Declude, i.e. that you have IMail drop the message if it triggers 10 ip4r tests.  Thus giving you a second way to get weed out messages before they reach Declude and your heavy text filters (for new readers, the first way was via the kill list).

 

Andrew 8)

-----Original Message-----
From: Kami Razvan [mailto:[EMAIL PROTECTED]
Sent: Friday, July 25, 2003 5:23 AM
To: [EMAIL PROTECTED]
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Hi Andrew:

 

The X-IMAIL that you asked about has to do with the weights that we assign to each test.

 

We are doing the IP4r tests in IMail and adding headers.  Declude filter for these tests in the header has weights assigned to each test.

 

 

1:  To start with:  IMail kill.lst file does not allow the entries to send email to a user.  "unacceptable mail address in MAIL FROM:" - so these emails never make it even to to the IP4r tests.

 

2:  Email is not in the kill list:  IMail does IP4r tests and adds header

3:  Email makes it past IP4r tests in IMail :  Header filter in Declude assigns weights to the header entries

 

There was a lot of discussion about this a while back as to which is more efficient doing it in IMail or Declude.  We did an analysis of the tests and finally concluded that if an email fails 10 of the IP4r tests it is 100% spam.  This was done over 3 months and checking every spam.  So now we have set the IMail 8 to delete any email that fails 13 of the IP4r tests.  We have about 30 of them listed.  This also reduces the load on the server since these emails (& there are a lot of them) never make it to Declude.  On the average all of our tests have a weight of 5 - some are 1 and some are 8,10,14.  If we take 5 as the average then 13 failed tests is equivalent to a weight of 65.  We delete on 60 anyway so we feel safe with this action.

 

As spammers get smarter our filter files keep growing in size and naturally Declude will have more and more to do.  If we know something is getting deleted why bother Declude to check it?

 

I hope that answers your question.

 

Regards,

Kami

 

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew
Sent: Thursday, July 24, 2003 5:16 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Kami, you've mentioned your approach before but never so succinctly; thanks for sharing.  I've been meaning to ask you why you adopted techniques 1) and 2).  Surely this is a longer way to do the same thing as IP4R/RBL tests in JM?

 

3) Does make good sense to me.  I wonder if the same logic applies to 1) because you use "trusted blacklist" types to eliminate messages before they get to declude.exe?  In which case, the X-IMAIL- header is irrelevant because IMail immediately deletes the message.

 

Andrew.

 

p.s. I haven't upgraded to IMail 8 yet; I'm planning on upgrading the hardware, IMail and JunkMail Pro all at the same time.

-----Original Message-----
From: Kami Razvan [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 24, 2003 1:31 PM
To: [EMAIL PROTECTED]
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Hi;

 

Let me share our experience and what we did to help it.

 

1:  We put all the IP4r tests in IMail (of course version 8 if you have it)

 

2:  Created a weight XHeader file and gave the header that appears for each test a weight.  The following is a snapshot of what we have for the file:

 

HEADERS          8          CONTAINS          X-IMAIL-SPAM-DNSBL: (BHOLE-RUSSIA
HEADERS          5          CONTAINS          X-IMAIL-SPAM-DNSBL: (BHOLE-YIPES
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (BLARS
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (COMPU
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (DELINK
HEADERS          6          CONTAINS          X-IMAIL-SPAM-DNSBL: (DSBL
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (DSBLALL

 

3:  With IMail version 8 we also moved the kill list to IMail from Declude.  In previous versions IMail did not handle Kill lists very well but with 8 and the wild card we changed our kill list and moved to IMail.  This helped a lot since emails are simply blocked at IMail level and never make it to Declude.  This reduced a lot of load since Declude will never see an email from a blacklisted address.  We know that Declude finishes everything it has to before taking the action, delete in this case, so if Declude will delete the message anyway why let it see it to start with?

 

Of course these may not matter much to most but we have a lot of filter files so perhaps our imposed load is a tad out of ordinary.

 

All these together - simply taking the best of both worlds - eased the load on the server.

 

I am sure others have other experiences but this is our experience.

 

Regards,

Kami

 

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mark Gordon
Sent: Thursday, July 24, 2003 4:05 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Thanks for that info Andrew, however I am looking at the task manager for my box and even 1 declude process was taking 50% then it popped up another process with 45% granted these go away when the messages are finished, however this server passes a LOT of traffic, so the number of processes for imail is default (30), I hope it wouldnt use that many for it would kill the box. I let declude run for 2 hours and my monitors reported 95-100% cpu usage consistantly.

-----Original Message-----
From: Colbeck, Andrew [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 24, 2003 3:56 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Mark, it may be interesting for you to note that we don't set the number of instances of Declude directly.  Instead, the "max processes" limit in your IMail SMTP advanced settings is what governs the total number of IMail and declude.exe instances.

 

Also, an important infrastructure detail is that IMail calls one declude.exe for each message (not recipient), and declude.exe quits after it is done. Watch Task Manager for a while, sort alphabetically, and watch the number of declude.exe instances come and note that their PIDs are always changing.

 

When I first evaluated Declude JunkMail Pro, I had assumed it ran as a service and IMail passed it data as necessary.  This was entirely wrong, so the previous two paragraphs would have been helpful to me.  I hope they help you.

 

Andrew 8)

-----Original Message-----
From: Mark Gordon [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 24, 2003 12:15 PM
To: '[EMAIL PROTECTED]'
Subject: [Declude.JunkMail] Declude using 50% cpu

We are evaluating declude and have noticed a considerable increase in the cpu cycles associated with mail delivery. Is there anyway have it run in an isolated cpu instance? since there are multiple instances of declude.exe running, I would guess it would be hard to lock it down.