RE: [Declude.JunkMail] TextFilter file, how many entries ?
My question is, How many entries can I put in the file before Declude slow down ? I think this depends on -what HW-ressources you have in use (CPU, storage,...) -how much mail traffic you have during peak times I've had a problem some days ago by adding two filter files having both around 140 kB (one for body one for header filtering) At midnight all worked fine but next morning at around 9:00 AM it was almost not more possible to logon to the server. Ping whas ok, but it has taken over a minute to display the Terminal services login screen. After logging on I've had to wait for over 5 Minutes to see the desktop. Other 2 Minutes to open the taskmanager and see that there was a lot of declude.exe's having CPU-times of over 60 seconds and consuming all disponible CPU-resources. We've running Imail v7.15 on a Intel P4 2,6 GHz and IDE Raid mirroring Declude Junkmail and Virus with 2 scan engines. First I've tried to disable the second AV engine without a result. After removing the two filter files all become normal. We process around 4000 msgs/day. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] TextFilter file, how many entries ?
I posted a question two weeks ago asking if anyone knew a way to calculate the amount of time it takes for individual messages to clear the entire receive/virusscan/junkmailscan/deliver process, and this exactly why I asked. My system doesn't have any filters quite as large as 140kb, or even 70k, but I keep adding steadily to them and it feels like things are somewhat slower. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Markus Gufler Sent: Thursday, 30 October 2003 3:54 AM To: [EMAIL PROTECTED] Subject: RE: [Declude.JunkMail] TextFilter file, how many entries ? My question is, How many entries can I put in the file before Declude slow down ? I think this depends on -what HW-ressources you have in use (CPU, storage,...) -how much mail traffic you have during peak times I've had a problem some days ago by adding two filter files having both around 140 kB (one for body one for header filtering) At midnight all worked fine but next morning at around 9:00 AM it was almost not more possible to logon to the server. Ping whas ok, but it has taken over a minute to display the Terminal services login screen. After logging on I've had to wait for over 5 Minutes to see the desktop. Other 2 Minutes to open the taskmanager and see that there was a lot of declude.exe's having CPU-times of over 60 seconds and consuming all disponible CPU-resources. We've running Imail v7.15 on a Intel P4 2,6 GHz and IDE Raid mirroring Declude Junkmail and Virus with 2 scan engines. First I've tried to disable the second AV engine without a result. After removing the two filter files all become normal. We process around 4000 msgs/day. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] TextFilter file, how many entries ?
I posted a question two weeks ago asking if anyone knew a way to calculate the amount of time it takes for individual messages to clear the entire receive/virusscan/junkmailscan/deliver process, and this exactly why I asked. My system doesn't have any filters quite as large as 140kb, or even 70k, but I keep adding steadily to them and it feels like things are somewhat slower. I suggest to simply try it out. Create a large filter list (definitively larger than you expect to use in future) and assign to all (random) keywords a weight of 0 and no additional action. This should create the same resource usage as with points. Now set up something to send a little bit more mails then your average mail processing rate (for example a Script sending out 20 messages as fast as possible) You can send it all to the same recipient. Imail/Decludes architecture will not process it faster because the messages are all the same. Put some tipical content (1 to 30 kB of text) in the message body. Watch the cpu usage during normal processing and the simulated mail bombardement. If you want you can also set a line PIDDEBUG ON In your global.cfg file This will write a .PID file for every declude process in you spool folder. Note: it's deleted after the process has finished his task so you have to open it during processing (not easy) In this PID file you can read in milliseconds how long any step takes to finish. All your results are something that can be interesting for multiple users on this list but keep in mind to indicate also what CPU, storage system, ... you've in use. Whats the average/peak message processing rate on your server, ... -- About CPU usage: I've had an idea some months ago and still search someone who can help. The problem: certain spam-tests can be very CPU-intensive. This will prevent us to keep filter files and programming logic as simple as possible. (For example long text filter files, regular expressions) The real problem: Any mailserver running a lot of tests before store or forward the message to the final destination is much more vulnerable for peak usage or also simple mailbomb attacks then a MTA configured to simple deliver any message as fast and efficient as possible. The idea: If declude (or our external spamchk test) could determine an average CPU usage value before start all tests it should be possible to dinamicaly exclude certain resource intensive tests if the CPU average is to high. For example: In the global.cfg file a test could be configured like %75 MYFILTER filter d:\imail\declude\large_bodyfilter.txt x 5 0 This test would run only if declude has determined an average cpu usage below 75% Another problem: declude is called as needed for any single message. It's not a service running around the clock and so it's not able to determine and provide a reliable CPU average value. The solution: A small windows service that calculate and serves the 1, 5 or 10 minute CPU average value. Declude could connect over DCOM or a certain TCP/UDP port to this service before run all other tests. If the average is to high this will comment out automatically the big tests. Such a solution will not decrease the detection rate because certain tests will not run sometimes, but will increase the detection rate because this new tests now can run everytime when it's possible. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] TextFilter file, how many entries ?
Could Declude be run as a Service? - Original Message - From: Markus Gufler [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, October 30, 2003 9:59 AM Subject: RE: [Declude.JunkMail] TextFilter file, how many entries ? I posted a question two weeks ago asking if anyone knew a way to calculate the amount of time it takes for individual messages to clear the entire receive/virusscan/junkmailscan/deliver process, and this exactly why I asked. My system doesn't have any filters quite as large as 140kb, or even 70k, but I keep adding steadily to them and it feels like things are somewhat slower. I suggest to simply try it out. Create a large filter list (definitively larger than you expect to use in future) and assign to all (random) keywords a weight of 0 and no additional action. This should create the same resource usage as with points. Now set up something to send a little bit more mails then your average mail processing rate (for example a Script sending out 20 messages as fast as possible) You can send it all to the same recipient. Imail/Decludes architecture will not process it faster because the messages are all the same. Put some tipical content (1 to 30 kB of text) in the message body. Watch the cpu usage during normal processing and the simulated mail bombardement. If you want you can also set a line PIDDEBUG ON In your global.cfg file This will write a .PID file for every declude process in you spool folder. Note: it's deleted after the process has finished his task so you have to open it during processing (not easy) In this PID file you can read in milliseconds how long any step takes to finish. All your results are something that can be interesting for multiple users on this list but keep in mind to indicate also what CPU, storage system, ... you've in use. Whats the average/peak message processing rate on your server, ... -- About CPU usage: I've had an idea some months ago and still search someone who can help. The problem: certain spam-tests can be very CPU-intensive. This will prevent us to keep filter files and programming logic as simple as possible. (For example long text filter files, regular expressions) The real problem: Any mailserver running a lot of tests before store or forward the message to the final destination is much more vulnerable for peak usage or also simple mailbomb attacks then a MTA configured to simple deliver any message as fast and efficient as possible. The idea: If declude (or our external spamchk test) could determine an average CPU usage value before start all tests it should be possible to dinamicaly exclude certain resource intensive tests if the CPU average is to high. For example: In the global.cfg file a test could be configured like %75 MYFILTER filter d:\imail\declude\large_bodyfilter.txt x 5 0 This test would run only if declude has determined an average cpu usage below 75% Another problem: declude is called as needed for any single message. It's not a service running around the clock and so it's not able to determine and provide a reliable CPU average value. The solution: A small windows service that calculate and serves the 1, 5 or 10 minute CPU average value. Declude could connect over DCOM or a certain TCP/UDP port to this service before run all other tests. If the average is to high this will comment out automatically the big tests. Such a solution will not decrease the detection rate because certain tests will not run sometimes, but will increase the detection rate because this new tests now can run everytime when it's possible. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] TextFilter file, how many entries ?
Could Declude be run as a Service? No. Declude.exe is called from Imail'S SMTP-Service for every single incoming message. Declude.exe is called in place of or before Imails original exe-file and terminate after the message is delivered socessfully to the destination. The idea is to run a simple service (completely independent from imail or declude) that calculates the average cpu usage value. This can be done be asking the current CPU-usage every one second. (for example) Sec.Cur%. 10s Avg%. 1 0 0 2 20 10 3 20 13 4 20 15 5 0 12 6 90 25 7 90 34 8 95 41 9 100 48 10 100 54 11 100 64 12 100 72 13 100 82 - don't start more resource intensive tests (RIT) 14 100 91 15 80 96 16 50 91 17 30 85 18 30 78 19 0 68 - begin starting RIT 20 10 59 21 ... Maybe the 10 seconds average are not the definitively right time range and maybe it's also necessary to define a trigger-on and trigger-off treshold as used also in common tecnic regluation processes to guarantee a well balanced regulation of the cpu-usage. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] TextFilter file, how many entries ?
I have two virus scanners with JunkMail Pro fully enabled plus 160 KB of active filters (~10 KB in comments) on a dual 1 Ghz PIII/1 Ghz memory/RAID 5/Win2k server also running a bunch of other Web services, and it takes maybe 5-7 seconds for me to send a very short message to myself and have it come back. If I sent a large executable file though, it would take my server 20 seconds or so to get it back to me, probably mostly because of the virus scanning (Declude only scans the first 32K of the message with text filters). Currently we handle just over 3,000 messages a day, and the processors normally don't go over the 15%-20% range. Without the second scanner and all the custom filters, that was more like 5%-7% as a peak. I certainly expect the server to handle much more than 6 times the current traffic. I'll probably be looking to optimize a bit more at that point though. Matt John Shacklett wrote: I posted a question two weeks ago asking if anyone knew a way to calculate the amount of time it takes for individual messages to clear the entire receive/virusscan/junkmailscan/deliver process, and this exactly why I asked. My system doesn't have any filters quite as large as 140kb, or even 70k, but I keep adding steadily to them and it "feels" like things are somewhat slower. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Markus Gufler Sent: Thursday, 30 October 2003 3:54 AM To: [EMAIL PROTECTED] Subject: RE: [Declude.JunkMail] TextFilter file, how many entries ? My question is, How many entries can I put in the file before Declude slow down ? I think this depends on -what HW-ressources you have in use (CPU, storage,...) -how much mail traffic you have during peak times I've had a problem some days ago by adding two filter files having both around 140 kB (one for body one for header filtering) At midnight all worked fine but next morning at around 9:00 AM it was almost not more possible to logon to the server. Ping whas ok, but it has taken over a minute to display the Terminal services login screen. After logging on I've had to wait for over 5 Minutes to see the desktop. Other 2 Minutes to open the taskmanager and see that there was a lot of declude.exe's having CPU-times of over 60 seconds and consuming all disponible CPU-resources. We've running Imail v7.15 on a Intel P4 2,6 GHz and IDE Raid mirroring Declude Junkmail and Virus with 2 scan engines. First I've tried to disable the second AV engine without a result. After removing the two filter files all become normal. We process around 4000 msgs/day. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
