RE: [Declude.JunkMail] Declude using 50% cpu

[Kami Razvan]

Title: Message

Hi Andrew:

 

The X-IMAIL that you asked about has to do with the weights that we assign to each test.

 

We are doing the IP4r tests in IMail and adding headers.  Declude filter for these tests in the header has weights assigned to each test.

 

 

1:  To start with:  IMail kill.lst file does not allow the entries to send email to a user.  "unacceptable mail address in MAIL FROM:" - so these emails never make it even to to the IP4r tests.

 

2:  Email is not in the kill list:  IMail does IP4r tests and adds header

3:  Email makes it past IP4r tests in IMail :  Header filter in Declude assigns weights to the header entries

 

There was a lot of discussion about this a while back as to which is more efficient doing it in IMail or Declude.  We did an analysis of the tests and finally concluded that if an email fails 10 of the IP4r tests it is 100% spam.  This was done over 3 months and checking every spam.  So now we have set the IMail 8 to delete any email that fails 13 of the IP4r tests.  We have about 30 of them listed.  This also reduces the load on the server since these emails (& there are a lot of them) never make it to Declude.  On the average all of our tests have a weight of 5 - some are 1 and some are 8,10,14.  If we take 5 as the average then 13 failed tests is equivalent to a weight of 65.  We delete on 60 anyway so we feel safe with this action.

 

As spammers get smarter our filter files keep growing in size and naturally Declude will have more and more to do.  If we know something is getting deleted why bother Declude to check it?

 

I hope that answers your question.

 

Regards,

Kami

 

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew
Sent: Thursday, July 24, 2003 5:16 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Kami, you've mentioned your approach before but never so succinctly; thanks for sharing.  I've been meaning to ask you why you adopted techniques 1) and 2).  Surely this is a longer way to do the same thing as IP4R/RBL tests in JM?

 

3) Does make good sense to me.  I wonder if the same logic applies to 1) because you use "trusted blacklist" types to eliminate messages before they get to declude.exe?  In which case, the X-IMAIL- header is irrelevant because IMail immediately deletes the message.

 

Andrew.

 

p.s. I haven't upgraded to IMail 8 yet; I'm planning on upgrading the hardware, IMail and JunkMail Pro all at the same time.

-----Original Message-----
From: Kami Razvan [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 24, 2003 1:31 PM
To: [EMAIL PROTECTED]
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Hi;

 

Let me share our experience and what we did to help it.

 

1:  We put all the IP4r tests in IMail (of course version 8 if you have it)

 

2:  Created a weight XHeader file and gave the header that appears for each test a weight.  The following is a snapshot of what we have for the file:

 

HEADERS          8          CONTAINS          X-IMAIL-SPAM-DNSBL: (BHOLE-RUSSIA
HEADERS          5          CONTAINS          X-IMAIL-SPAM-DNSBL: (BHOLE-YIPES
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (BLARS
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (COMPU
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (DELINK
HEADERS          6          CONTAINS          X-IMAIL-SPAM-DNSBL: (DSBL
HEADERS          1          CONTAINS          X-IMAIL-SPAM-DNSBL: (DSBLALL

 

3:  With IMail version 8 we also moved the kill list to IMail from Declude.  In previous versions IMail did not handle Kill lists very well but with 8 and the wild card we changed our kill list and moved to IMail.  This helped a lot since emails are simply blocked at IMail level and never make it to Declude.  This reduced a lot of load since Declude will never see an email from a blacklisted address.  We know that Declude finishes everything it has to before taking the action, delete in this case, so if Declude will delete the message anyway why let it see it to start with?

 

Of course these may not matter much to most but we have a lot of filter files so perhaps our imposed load is a tad out of ordinary.

 

All these together - simply taking the best of both worlds - eased the load on the server.

 

I am sure others have other experiences but this is our experience.

 

Regards,

Kami

 

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mark Gordon
Sent: Thursday, July 24, 2003 4:05 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Thanks for that info Andrew, however I am looking at the task manager for my box and even 1 declude process was taking 50% then it popped up another process with 45% granted these go away when the messages are finished, however this server passes a LOT of traffic, so the number of processes for imail is default (30), I hope it wouldnt use that many for it would kill the box. I let declude run for 2 hours and my monitors reported 95-100% cpu usage consistantly.

-----Original Message-----
From: Colbeck, Andrew [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 24, 2003 3:56 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Declude using 50% cpu

Mark, it may be interesting for you to note that we don't set the number of instances of Declude directly.  Instead, the "max processes" limit in your IMail SMTP advanced settings is what governs the total number of IMail and declude.exe instances.

 

Also, an important infrastructure detail is that IMail calls one declude.exe for each message (not recipient), and declude.exe quits after it is done. Watch Task Manager for a while, sort alphabetically, and watch the number of declude.exe instances come and note that their PIDs are always changing.

 

When I first evaluated Declude JunkMail Pro, I had assumed it ran as a service and IMail passed it data as necessary.  This was entirely wrong, so the previous two paragraphs would have been helpful to me.  I hope they help you.

 

Andrew 8)

-----Original Message-----
From: Mark Gordon [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 24, 2003 12:15 PM
To: '[EMAIL PROTECTED]'
Subject: [Declude.JunkMail] Declude using 50% cpu

We are evaluating declude and have noticed a considerable increase in the cpu cycles associated with mail delivery. Is there anyway have it run in an isolated cpu instance? since there are multiple instances of declude.exe running, I would guess it would be hard to lock it down.