Matt, is your Delcude gatewayed? Or is it running on the same server as Imail?
-Erik -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Wednesday, May 24, 2006 5:58 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I indicated earlier that it looked like a relative 10% improvement (about the difference between 35% and 32% hourly average CPU utilization). I would think that this comes primarily from not needing to start the old declude executable every time, and the improvement might be more substantial on a system with a overtaxed disk. I don't think the code is any more efficient as far as actual scanning goes. Besides that, it's typically the virus scanners and external tests that eat up most of the CPU on a Declude system and not Declude itself, and those things haven't changed. I'm guessing that it might perform much better when redlined though if you tweak it right. I believed that old Declude when getting rammed with overflow seemed to not perform linearly with normal performance with free CPU, but instead dropped, probably due to having a lot of CPU wait time overhead (there's probably a more accurate term for this). The new Declude can be more effectively controlled so as to not bog down the system as much. So performance here might be 2x or more when redlined under optimal conditions. This effect would be maximized if Declude would tie launching threads to a CPU monitor instead of a fixed number with no real clue as to what is perfect under any particular situation. Matt Nick Hayer wrote: > > Hi Matt, > > So you see any substantive performance improvement over 2x? > > -Nick > > Matt wrote: > >> Jay, >> >> It's not about moving along, it's about limiting the CPU to only >> 100%, or at least not piling it on when it gets there. I could be >> wrong in assuming that 1 thread = 1 message (hopefully I will be >> corrected if so), but 30 average messages being processed at once >> will most definitely peg my processors, and adding more threads when >> you are at 100% will actually slow down performance. >> >> Another note, not all systems are configured equally. A vanilla >> install of Declude would likely handle 4 times the number of messages >> that mine does since I run 4 external filters, two virus scanners, >> and something like 100 Declude filters (though they mostly get >> skipped with SKIPIFWEIGHT and END statements as they are targeted). >> Running a single virus scanner and RBL's is just a fraction of the >> load. With my pre-scanning gateways blocking more than 90% of all >> traffic (about half of that is dictionary attacks and most of the >> rest is done with 'selective greylisting'), I can scale one server to >> handle over 20,000 addresses, possibly as many as 40,000 (doesn't >> host the accounts though), so despite the heavy config, it is optimized. >> >> But back to the real topic...I'm just guessing that 30 >> messages/threads is the limit for my box, but I'm sure that it isn't >> as high as 80, though setting it at 80 would be of no consequence >> outside of a prolonged heavy load caused by something like a backup >> of my spool. It would be a bigger mistake to set it too low. >> >> Matt >> >> >> >> Jay Sudowski - Handy Networks LLC wrote: >> >>> 30 threads seems awfully low. We set ours to 80 on a dual xeon box >>> with a separate drive for spool/logging and we move right along >>> without any issues. >>> >>> Thanks! >>> ----- >>> Jay Sudowski // Handy Networks LLC >>> Director of Technical Operations >>> Providing Shared, Reseller, Semi Managed and Fully Managed Windows >>> 2003 Hosting Solutions >>> Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX >>> www.handynetworks.com >>> ________________________________________ >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Matt >>> Sent: Tuesday, May 23, 2006 3:25 PM >>> To: Declude.JunkMail@declude.com >>> Subject: Re: [Declude.JunkMail] Experience with 4.x >>> >>> Andrew, >>> >>> Thanks for your notes and their history. >>> >>> I'm using the following settings right now: >>> THREADS 30 >>> WAITFORMAIL 500 >>> WAITFORTHREADS 200 >>> WAITBETWEENTHREADS 100 >>> WINSOCKCLEANUP OFF >>> INVITEFIX ON >>> AUTOREVIEW ON >>> There are a few reasons for trying these values. >>> THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID >>> can only handle 30 threads with average messages. In reality, one >>> single message can spike the system to 100%, but these are >>> uncommon. I figure that if I open this up too wide and I am dealing >>> with a backup or something, launching more threads when at 100% CPU >>> utilization will actually slow the system down. This was the same >>> with 2.x and before. There is added overhead to managing threads >>> and you don't want that to happen on top of 100% CPU utilization. I >>> am going to back up my server later tonight to see if I can't find >>> what the magic number is since I don't want to be below that magic >>> number, and it would probably be best to be a little above it. >>> >>> WAITFORMAIL 500 - On my server, this never kicks in, but if it did, >>> it wouldn't make sense to delay for too long because I could build >>> up messages. A half second seems good. >>> >>> WAITFORTHREADS 200 - This apparently kicks in only when I reach my >>> thread limit; sort of like a throttle. I don't want it to be too >>> long because this should only happen when I am hammered, but it is >>> wise not to keep hammering when you are at 100%. Sort of a mixed >>> bag choice here. >>> >>> WAITBETWEENTHREADS 100 - I see this setting as being the biggest >>> issue with sizing a server. Setting it at 100 ms means that I can >>> only handle 10 messages per second, and this establishes an upper >>> limit for what the server can do. I currently average about 5 >>> messages per second coming from my gateways at peak hours, so I >>> figured that to be safe, I should double that value. >>> >>> INVITEFIX ON - I have it on because it comes on by default and I >>> don't know any better. I know nothing about the cause for needing >>> this outside of brief comments. It seems strange that my Declude >>> setup could ruin an invitation unless I was using footers. If this >>> is only triggered by footer use, I would like to know so that I >>> could turn it off. I would imagine that this causes extra load to >>> do the check. >>> >>> AUTOREVIEW ON - I have this on for the same reason that Andrew >>> pointed out. When I restart Decludeproc, messages land in my review >>> folder, and I don't wish to keep manually fishing things out. If >>> there is an issue with looping, it would be wise for Declude to make >>> this only trigger say every 15 minutes instead of more regularly. >>> Feel free to add to this if you want. >>> >>> Matt >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Colbeck, Andrew wrote: I'd second that... on both the observed >>> behaviour and the request for documentation. >>> >>> I'm attaching my highly commented declude.cfg as a reasonable >>> sample. >>> >>> Andrew 8) >>> >>> >>> >>> ________________________________________ >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Matt >>> Sent: Tuesday, May 23, 2006 10:36 AM >>> To: Declude.JunkMail@declude.com >>> Subject: Re: [Declude.JunkMail] Experience with 4.x >>> David, >>> >>> That did the trick. I can't even see any messages in my proc folder >>> any more. I might suggest adding your explanation to the comments >>> in the file just in case others feel the need to turn this on like I >>> did. I recalled the issues from the list and I turned it on because >>> I didn't want the possibility of DNS crapping out and the leakage >>> that this would cause. >>> >>> Here's a screen cap of what my processor graph looks like now: >>> >>> >>> Thanks, >>> >>> Matt >>> >>> >>> >>> David Barker wrote: The purpose of WINSOCKCLEANUP ON is to >>> reset the winsock, what >>> happens when using this setting is that when the \proc directory hit >>> 0 decludeproc will finish processing all the messages in the \work >>> before checking the \proc again. As WINSOCKCLEANUP is to be used >>> only by those who experience DNS issues I would suggest running your >>> tests again with WINSOCKCLEANUP commented out and see how the >>> behavior differs. Also having >>> the WAITFORMAIL to low can cause the CPU to process very high as it is >>> constantly checking the \proc I would suggest a minimum of 500-1000 >>> >>> David B >>> www.declude.com >>> >>> -----Original Message----- >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Matt >>> Sent: Monday, May 22, 2006 8:12 PM >>> To: Declude.JunkMail@declude.com >>> Subject: Re: [Declude.JunkMail] Experience with 4.x >>> >>> Darrell, >>> >>> I put up two Windows Explorer windows side-by-side under normal >>> volume and the pattern was consistent where the proc folder grows >>> while the work folder shrinks until the work folder hits zero at >>> which point the proc folder empties out and everything lands in work >>> and then the pattern repeats with proc growing while work shrinks. >>> >>> My settings are as follows: >>> >>> THREADS 50 >>> WAITFORMAIL 100 >>> WAITFORTHREADS 10 >>> WAITBETWEENTHREADS 50 >>> WINSOCKCLEANUP ON >>> AUTOREVIEW ON >>> INVITEFIX ON >>> >>> Matt >>> >>> >>> >>> >>> Darrell ([EMAIL PROTECTED]) wrote: >>> >>> >>> It's a faulty design that leaves more than half a server's CPU >>> capacity unused due to the mere fact that they wait for all threads >>> to complete before moving in a new batch. >>> I can't speak to what you see on your server, but that is not >>> how it is running on my server. I just double checked again to make >>> sure I am not crazy, but as I watch the thread count on my server >>> (decludeproc) the threads fluctuate between 7 - 30 ( threads >>> currently set to 50). It is not uncommon to see the threads move as >>> follow: 11,8,10,7,15,.... While I was watching it I never seen a >>> case where it went down low enough for the WAITFORMAIL setting to >>> kick in. Watching the proc/work directory you can see files moving >>> in and out, but never really emptying out. Its possible what I am >>> seeing is an anomaly or maybe I am interpreting it wrong. >>> >>> Maybe David can comment on this. >>> >>> Darrell >>> -------------------------------------------------------------------- >>> ---- >>> >>> invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, >>> and ORF. Stop spam at the source the spamvertised domain. More >>> effective than traditional RBL's. Try it today - >>> http://www.invariantsystems.com >>> --- >>> This E-mail came from the Declude.JunkMail mailing list. To >>> unsubscribe, just send an E-mail to [EMAIL PROTECTED], and >>> type "unsubscribe Declude.JunkMail". The archives can be found >>> at http://www.mail-archive.com. >>> >>> >>> --- >>> This E-mail came from the Declude.JunkMail mailing list. To >>> unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type >>> "unsubscribe Declude.JunkMail". The archives can be found at >>> http://www.mail-archive.com. >>> >>> --- >>> This E-mail came from the Declude.JunkMail mailing list. To >>> unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type >>> "unsubscribe Declude.JunkMail". The archives can be found at >>> http://www.mail-archive.com. >>> >>> >>> >>> --- >>> This E-mail came from the Declude.JunkMail mailing list. To >>> unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type >>> "unsubscribe Declude.JunkMail". The archives can be found at >>> http://www.mail-archive.com. >>> >>> >>> >>> >> --- >> This E-mail came from the Declude.JunkMail mailing list. To >> unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type >> "unsubscribe Declude.JunkMail". The archives can be found at >> http://www.mail-archive.com. >> >> > --- > This E-mail came from the Declude.JunkMail mailing list. To > unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type > "unsubscribe Declude.JunkMail". The archives can be found at > http://www.mail-archive.com. > > --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
