Re: [Declude.JunkMail] Experience with 4.x
Sanford Whiteman wrote: I am not denying _you_ shared credit for the concept with the Postfix people, but the idea that your "friend" the vendor can claim it as intellectual property, when you spec'd it out and have documentation of same -- and that that's why you're playing it close to the vest -- is ridiculous. If it's yours or Brian's, it's yours; in reality, it's nobody's. I came up with the idea before I ever heard mention of it anywhere else, and long before 8/2005. I was searching for someone to develop a gateway from scratch with various improvements/adaptations of existing techniques as well as new ones since the first half of 2004. Although ORF saved my arse from dictionary attacks, I realized quickly that it wasn't ever going to become a reliable way of pre-scanning E-mail and I wanted more. With something like 6 billion people in this world, and the concept being as obvious as it seems to me, I definitely wouldn't suggest that I was the only one to have thought of it, and I'm sure that I wasn't the first to ever think of it either. This is not something to waste bytes on. What does matter in the context though is that Brian made many of these things work...and then some. His code is his company's property. He is not the type of guy though to claim rights to such functionality being the good netizen that he is, and I'm sure that we would all agree that much of the software technique patent stuff is damaging to advancements. I don't want to make too much of this one little thing though because it isn't all that earth shattering to think that it could be improved. It would something like suggesting that SpamCop could clean up their false positives by qualifying IP's for listing, or dequalifying them, sort of like CBL does. It's just my opinion that there are a lot of good ideas that don't get developed into mature ideas in this field, and thus far greylisting has been one of them. Besides greylisting, groups of us have talked about or even implemented things before they existed to the public. Keith Anderson gets credit for identifying what a "payload" is and how to extract that information and use it for blocking E-mail. Group discussions started in 9/2003, six months before SURBL.org was even registered as a domain, and clearly the concept pre-existed that day by even longer. We as a group also came up with enhancements such as resolving payload links to IP's, or even MX records, NS records to check against URIBL lists as well as RHSBL lists, and then resolve those again to IP's and look them up in IP4R lists. Keith even created a product that did this called Eradispam before anything else that I know of did (besides the straight URIBL stuff). Not everyone got these ideas from us though, some clearly came up with it on their own as well. There are mounds of other inventive if not unique ideas that power users around here came up with, and some of them we all benefited from. Scott used to design functionality around community development and I'm sure that many of us would love to see a return to that atmosphere. Unfortunately there are more good ideas than there are great programmers that can make the maximum use of this stuff. Brian is not only a great programmer, he also brought something very unique to the table in this framework that made it immensely better than anything else that does this could be. MXRate when combined with greylisting is very powerful and allows for greater accuracy as well as flexibility. MXRate in it's native form isn't just a three zoned IP4R list, it's probability based and the IP4R version is just backwards compatibility. This allows users to choose the level of greylisting and blocking to their own tastes so that they can block more spam and avoid more false positives/delays. Here's the net result from all 4 of my MX records now being bound to Alligate on Tuesday: Incoming message attempts: 708,777 Blocked by gateway pre-scanning: 676,397 *likely to be 3/4 dictionary attacks. Blocked by Declude JunkMail: 9,228 Blocked by Declude Virus: 90 Delivered to customers: 23,062 Landed in Review range: 980 This change in my infrastructure is the single best thing that I have done since starting to use Declude. Even though I stood up for Brian's character when he posted to the list about his product, I didn't even give the idea of using his product a second thought until we started exchanging E-mails and I grew to understand that he was an even better person that I had thought, that he was a good programmer, and he gets "it". I think that we should embrace development like this because it is for the good of everyone, including Declude. Now non-experts can place a address validating and pre-scanning gateway in front of any Declude setup whereas before you nearly had to be an expert at this stuff and be aware of the obscure. The fact that it kills over 99% of zombie spam with near perfect accuracy also flies in the face of frequent
Re: [Declude.JunkMail] Experience with 4.x
Sandy, That link indicates that they actually work in reverse by whitelisting things from greylisting instead this is the other way around where it qualifies messages for greylisting. Considering how successful and accurate this method is, I would suggest that it is better. I can't of course say for sure since I'm not much of a source-code guru or Linux expert. Nevertheless, the open-source and anti-spam communities have made huge contributions, and made it possible for people like me to do what they do, and I'm sure that I wasn't the only one that ever thought about this...but still to my knowledge, it never existed before. Regarding intellectual property, it's definitely the wrong way to say that. I don't have any ownership of anything besides a claim to have had an idea, but I can't lay any claim to the product whatsoever, and there are no expectations or discussions of me giving it a rave review or financial benefit to me outside of what the software itself offers. It could even be considered a detriment to share things that work well in the sense that it makes my system less unique, but I couldn't even come close to quantifying it that way in light of circumstances. Brian deserves a lot of credit for pulling everything together the way that he did. It's no small feat to have this working in a stable and somewhat intuitive manner. This clearly isn't a huge financial venture either, but I doubt that it is going away anytime soon. It's not however proper for general discussion to lay all of the functionality out sort of like how Scott was tight-lipped about some functionality, but also because it is in fact intellectual property in the legal sense and I can't make such calls. Things like tarpitting and standard greylisting are now in some cases being defeated by spammers, and I suppose that it makes sense to not discuss such things in public. That's really what I should have said. I do take pride in having the idea though, even if I wasn't the only one :) What I did do though is map out enough of a picture for all to see if they felt the need, and I think that many of us do. I think that it's valuable to share such things, especially when there have been so few solutions to date that one could run on the same box and accomplish pre-scanning with any degree of accuracy and validate addresses. I described many things about my ORF experiences too when that was the only app in town that fit that bill to some extent. I'm certainly guilty for being overly enthusiastic about this one. With the right tools and not an enormous amount of work, most here could manage pretty close to the theoretical limit in results. Matt Sanford Whiteman wrote: I have never seen anyone else even talk about selective greylisting for instance, and I'm so damn grateful that I now have it. Well -- http://packages.debian.org/unstable/mail/whitelister -- a postfix pd that pre-checks messages against lower-impact tests such as DNSBLs before escalating to postgrey. In other words, selective greylisting. Released 8/2005. If there is still a widespread adversion to this discussion, I would be happy to take it off-list. Gotta tell ya, you almost had me looking the other way until you said you consider the component of the third-party product that you spec'd out to be (your?) "intellectual property." Are we to understand that you do not have a business relationship with the vendor of the product under discussion? That you have not received compensation, profit shares, payment in kind (free copies) for your design input into the commercial product? Hoping you're observing full disclosure in your continuing posts about the product. --Sandy Sanford Whiteman, Chief Technologist Broadleaf Systems, a division of Cypress Integrated Systems, Inc. e-mail: [EMAIL PROTECTED] SpamAssassin plugs into Declude! http://www.imprimia.com/products/software/freeutils/SPAMC32/download/release/ Defuse Dictionary Attacks: Turn Exchange or IMail mailboxes into IMail Aliases! http://www.imprimia.com/products/software/freeutils/exchange2aliases/download/release/ http://www.imprimia.com/products/software/freeutils/ldap2aliases/download/release/ --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Experience with 4.x
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: 26. maj 2006 00:16 Declude could easily plug into Alligate if they wanted to since it supports dropping files into a directory instead of delivering them, and then it will pick them up when the external app is done. I would love to see that happen since I only need IMail as a container for Declude. I would also like to see that. I also only use IMail as a gateway (in front of Exchange), and would like something that is better. I heard some rumors that Declude is looking at gateway solutions - with a release this summer. Anybody got any news on that? Regards, Kaj
RE: [Declude.JunkMail] Experience with 4.x
Hi Matt - I am still somewhat confused because you are painting a rather broad picture about your success with Alligate, and not really getting down to the nitty gritty specifics. I can appreciate that you may not want to disclose your exact configuration. However, I can't even figure out if all of the things you're talking about are available in a stock edition of Alligate with the MXRate subscription, or if you've extended the functionality of Alligate in ways that are specific to your installation, using tools or features that I won't have access to. http://www.alligate.com/help/AdvGreyOptions.htm - this seems to be the case specific greylisting you're talking about ... http://www.alligate.com/help/blocking.htm - has some tarpitting options ... Even so, I am still pretty confused about this. Any clarification would be appreciated. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Thursday, May 25, 2006 6:16 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Pieced below: Jay Sudowski - Handy Networks LLC wrote: I am contemplating implementing Alligate with SmarterMail in some fashion, and would like to pick your mind on the following: 1) SmarterMail does a great, great job at handling a huge number of SMTP threads so dictionary attacks are not really an issue. We have popped up to 2000 SMTP threads without incident. And since addresses being attacked do not exist, the messages get rejected with a 550 error, connection is closed and there's no real strain on the server (at least in terms of I/O) -- do we get any benefit from putting Alligate in front of SmarterMail for dictionary attack prevention? Definitely. Not all dictionary attacks hit completely bad addresses, but detecting and stopping an should be rare if you configure it carefully. 2) Does SMTP AUTH pass-through and recipient verification actually work, all the time? I don't use this since it is not necessary to do so for pre-scanning purposes. This is really designed to take the load off of a hosted mail server by off-loading this traffic. I can't say personally that it works perfectly, but I'm sure that it does. Other pass-through verification stuff that it does definitely works, and works efficiently. 3) I see greylisting as the real area that could benefit us -- we would avoid a lot of the zombie spam by implementing greylisting, and that would save us tons of resources all around. However, I understand that some zombies are now greylist aware and will retry. Greylisting is huge, but it has fundamental problems that kept me from using it in vanilla form with ORF. I know that Nick used it with ORF and backed off. My big issues are two-fold. By arbitrarily applying greylisting to everyone, you will effectively block E-mail from sending software that doesn't spool (and zombie spamware isn't the only thing with that limitation). It's things like blackboxes and other forms of notification programming that lacks such functionality and can fail greylisting, at least for the first message. Some bulk mail (ecommerce and otherwise) will only send once, or may requeue and send through a different IP subsequently causing delivery failures or long delays. Even more important to me is the fact that greylisting causing issues with delaying legitimate E-mail, and I refused to implement it in ORF solely for this reason. It's not acceptable for the class of service that I want to offer. I want E-mail to pass through our system in less than 5 seconds except in extreme circumstances. I fed Brian a framework for doing what I call selective greylisting which resolves these issues. Essentially it only greylists when conditions warrant, and the only drawback is greylisting poorly configured hosts or some that leak spam (but not all of either). It works so well that there is no reason to have it do full greylisting. I won't go into the details of how this is done here because to some extent I consider it to be intellectual property. Also note that Brian isn't just a coding monkey, and he contributed a lot to the end product and filled in many gaps. There are also advanced tarpitting techniques that cause almost as many failures as greylisting does because some spamware detects tarpitting and disconnect, but it is variable according to when and how long it occurs. The last major piece is the MXRate functionality, and it blocks a lot even when limited to just 100% hits (which surely aren't perfect, but they are very near perfect). While I don't feel like running DLAnalyzer, we typically trend about 85% of email is spam that gets
Re: [Declude.JunkMail] Experience with 4.x
Jay, As far as how to configure Alligate, I think that it would be better to discuss this off-list or on the product's own list, but to give a brief answer to your questions, almost all functionality is exposed in the GUI, but there are some undocumented things additionally that may be of use and possibly some things that aren't yet enabled in the released product yet. The interface is a bit confusing since it has it's own language and style of approaching things. In short, you can achieve the virtually same results as I did at the gateway level with what is available now, but you have the ability to tweak the settings whatever way you wish according to your own standards. Matt Jay Sudowski - Handy Networks LLC wrote: Hi Matt - I am still somewhat confused because you are painting a rather broad picture about your success with Alligate, and not really getting down to the nitty gritty specifics. I can appreciate that you may not want to disclose your exact configuration. However, I can't even figure out if all of the things you're talking about are available in a stock edition of Alligate with the MXRate subscription, or if you've extended the functionality of Alligate in ways that are specific to your installation, using tools or features that I won't have access to. http://www.alligate.com/help/AdvGreyOptions.htm - this seems to be the case specific greylisting you're talking about ... http://www.alligate.com/help/blocking.htm - has some tarpitting options ... Even so, I am still pretty confused about this. Any clarification would be appreciated. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Thursday, May 25, 2006 6:16 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Pieced below: Jay Sudowski - Handy Networks LLC wrote: I am contemplating implementing Alligate with SmarterMail in some fashion, and would like to pick your mind on the following: 1) SmarterMail does a great, great job at handling a huge number of SMTP threads so dictionary attacks are not really an issue. We have popped up to 2000 SMTP threads without incident. And since addresses being attacked do not exist, the messages get rejected with a 550 error, connection is closed and there's no real strain on the server (at least in terms of I/O) -- do we get any benefit from putting Alligate in front of SmarterMail for dictionary attack prevention? Definitely. Not all dictionary attacks hit completely bad addresses, but detecting and stopping an should be rare if you configure it carefully. 2) Does SMTP AUTH pass-through and recipient verification actually work, all the time? I don't use this since it is not necessary to do so for pre-scanning purposes. This is really designed to take the load off of a hosted mail server by off-loading this traffic. I can't say personally that it works perfectly, but I'm sure that it does. Other pass-through verification stuff that it does definitely works, and works efficiently. 3) I see greylisting as the real area that could benefit us -- we would avoid a lot of the zombie spam by implementing greylisting, and that would save us tons of resources all around. However, I understand that some zombies are now greylist aware and will retry. Greylisting is huge, but it has fundamental problems that kept me from using it in vanilla form with ORF. I know that Nick used it with ORF and backed off. My big issues are two-fold. By arbitrarily applying greylisting to everyone, you will effectively block E-mail from sending software that doesn't spool (and zombie spamware isn't the only thing with that limitation). It's things like blackboxes and other forms of notification programming that lacks such functionality and can fail greylisting, at least for the first message. Some bulk mail (ecommerce and otherwise) will only send once, or may requeue and send through a different IP subsequently causing delivery failures or long delays. Even more important to me is the fact that greylisting causing issues with delaying legitimate E-mail, and I refused to implement it in ORF solely for this reason. It's not acceptable for the class of service that I want to offer. I want E-mail to pass through our system in less than 5 seconds except in extreme circumstances. I fed Brian a framework for doing what I call selective greylisting which resolves these issues. Essentially it only greylists when conditions warrant, and the only drawback is greylisting poorly configured hosts or some that leak spam (but not all of either). It works so well that there is no reason to have it do full greylisting. I won't go
Re: [Declude.JunkMail] Experience with 4.x
Pieced below: Jay Sudowski - Handy Networks LLC wrote: I am contemplating implementing Alligate with SmarterMail in some fashion, and would like to pick your mind on the following: 1) SmarterMail does a great, great job at handling a huge number of SMTP threads so dictionary attacks are not really an issue. We have popped up to 2000 SMTP threads without incident. And since addresses being attacked do not exist, the messages get rejected with a 550 error, connection is closed and there's no real strain on the server (at least in terms of I/O) -- do we get any benefit from putting Alligate in front of SmarterMail for dictionary attack prevention? Definitely. Not all dictionary attacks hit completely bad addresses, but detecting and stopping an attacker regardless of the good or bad addresses that they may send is wise. Tarpitting takes the bites out of the attacks by slowing them down between recipients, and you can tell Alligate to drop the connection after x number of bad addresses, or even have more tolerance for a higher number of bad addresses if a good address is found. Lastly, SmarterMail and IMail were designed to be mail servers and not pre-scanning gateways. Gateways should be lean, and by also being smart, they can eliminate the vast majority of spam long before your real mail server comes into play. Nothing is blackholed either, so any FP should be noticed by the sender if they should occur, and they should be rare if you configure it carefully. 2) Does SMTP AUTH pass-through and recipient verification actually work, all the time? I don't use this since it is not necessary to do so for pre-scanning purposes. This is really designed to take the load off of a hosted mail server by off-loading this traffic. I can't say personally that it works perfectly, but I'm sure that it does. Other pass-through verification stuff that it does definitely works, and works efficiently. 3) I see greylisting as the real area that could benefit us -- we would avoid a lot of the zombie spam by implementing greylisting, and that would save us tons of resources all around. However, I understand that some zombies are now greylist aware and will retry. Greylisting is huge, but it has fundamental problems that kept me from using it in vanilla form with ORF. I know that Nick used it with ORF and backed off. My big issues are two-fold. By arbitrarily applying greylisting to everyone, you will effectively block E-mail from sending software that doesn't spool (and zombie spamware isn't the only thing with that limitation). It's things like blackboxes and other forms of notification programming that lacks such functionality and can fail greylisting, at least for the first message. Some bulk mail (ecommerce and otherwise) will only send once, or may requeue and send through a different IP subsequently causing delivery failures or long delays. Even more important to me is the fact that greylisting causing issues with delaying legitimate E-mail, and I refused to implement it in ORF solely for this reason. It's not acceptable for the class of service that I want to offer. I want E-mail to pass through our system in less than 5 seconds except in extreme circumstances. I fed Brian a framework for doing what I call "selective greylisting" which resolves these issues. Essentially it only greylists when conditions warrant, and the only drawback is greylisting poorly configured hosts or some that leak spam (but not all of either). It works so well that there is no reason to have it do full greylisting. I won't go into the details of how this is done here because to some extent I consider it to be intellectual property. Also note that Brian isn't just a coding monkey, and he contributed a lot to the end product and filled in many gaps. There are also advanced tarpitting techniques that cause almost as many failures as greylisting does because some spamware detects tarpitting and disconnect, but it is variable according to when and how long it occurs. The last major piece is the MXRate functionality, and it blocks a lot even when limited to just 100% hits (which surely aren't perfect, but they are very near perfect). While I don't feel like running DLAnalyzer, we typically trend about 85% of email is spam that gets deleted before it's delivered, which means we probably had around 40,000 actual deliveries. If we can eliminate 50% of the spam using MXRate and greylisting, that is a huge burden off of declude and the server in general. I block 99.9% of all spam and our false positive rate is less than 1 in 10,000, with most of that being some form of advertising sent from hosts that also send for dirty lists, and most don't care about such things. Most of our issues with personal E-mail are the result of foreign traffic, especially with China. Alligate certainly helps since it eliminates almost all zombies and viruses, but it also stops some static spammers as well, but Declude is still of course
RE: [Declude.JunkMail] Experience with 4.x
In light of this reference to WINSOCKCLEANUP being an iMail only issue, do I even need to enable this on our SmarterMail servers? -Jay -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 6:30 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x The only way that we have detected this was with Imail and mail being stuck in the spool. ...network stack causing loss of functionality for basic network operations is generic but if I remember correctly when this happened the admin was not even able to ping an outside server, which would suggest to me other IP communications fail as well. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail
RE: [Declude.JunkMail] Experience with 4.x
30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS 30 WAITFORMAIL 500 WAITFORTHREADS 200 WAITBETWEENTHREADS 100 WINSOCKCLEANUP OFF INVITEFIX ON AUTOREVIEW ON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above it. WAITFORMAIL 500 - On my server, this never kicks in, but if it did, it wouldn't make sense to delay for too long because I could build up messages. A half second seems good. WAITFORTHREADS 200 - This apparently kicks in only when I reach my thread limit; sort of like a throttle. I don't want it to be too long because this should only happen when I am hammered, but it is wise not to keep hammering when you are at 100%. Sort of a mixed bag choice here. WAITBETWEENTHREADS 100 - I see this setting as being the biggest issue with sizing a server. Setting it at 100 ms means that I can only handle 10 messages per second, and this establishes an upper limit for what the server can do. I currently average about 5 messages per second coming from my gateways at peak hours, so I figured that to be safe, I should double that value. INVITEFIX ON - I have it on because it comes on by default and I don't know any better. I know nothing about the cause for needing this outside of brief comments. It seems strange that my Declude setup could ruin an invitation unless I was using footers. If this is only triggered by footer use, I would like to know so that I could turn it off. I would imagine that this causes extra load to do the check. AUTOREVIEW ON - I have this on for the same reason that Andrew pointed out. When I restart Decludeproc, messages land in my review folder, and I don't wish to keep manually fishing things out. If there is an issue with looping, it would be wise for Declude to make this only trigger say every 15 minutes instead of more regularly. Feel free to add to this if you want. Matt Colbeck, Andrew wrote: I'd second that... on both the observed behaviour and the request for documentation. I'm attaching my highly commented declude.cfg as a reasonable sample. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 10:36 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x David, That did the trick. I can't even see any messages in my proc folder any more. I might suggest adding your explanation to the comments in the file just in case others feel the need to turn this on like I did. I recalled the issues from the list and I turned it on because I didn't want the possibility of DNS crapping out and the leakage that this would cause. Here's a screen cap of what my processor graph looks like now: Thanks, Matt David Barker wrote: The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re
Re: [Declude.JunkMail] Experience with 4.x
Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: 30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS30 WAITFORMAIL500 WAITFORTHREADS200 WAITBETWEENTHREADS100 WINSOCKCLEANUPOFF INVITEFIXON AUTOREVIEWON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above it. WAITFORMAIL 500 - On my server, this never kicks in, but if it did, it wouldn't make sense to delay for too long because I could build up messages. A half second seems good. WAITFORTHREADS 200 - This apparently kicks in only when I reach my thread limit; sort of like a throttle. I don't want it to be too long because this should only happen when I am hammered, but it is wise not to keep hammering when you are at 100%. Sort of a mixed bag choice here. WAITBETWEENTHREADS 100 - I see this setting as being the biggest issue with sizing a server. Setting it at 100 ms means that I can only handle 10 messages per second, and this establishes an upper limit for what the server can do. I currently average about 5 messages per second coming from my gateways at peak hours, so I figured that to be safe, I should double that value. INVITEFIX ON - I have it on because it comes on by default and I don't know any better. I know nothing about the cause for needing this outside of brief comments. It seems strange that my Declude setup could ruin an invitation unless I was using footers. If this is only triggered by footer use, I would like to know so that I could turn it off. I would imagine that this causes extra load to do the check. AUTOREVIEW ON - I have this on for the same reason that Andrew pointed out. When I restart Decludeproc, messages land in my review folder, and I don't wish to keep manually fishing things out. If there is an issue with looping, it would be wise for Declude to make this only trigger say every 15 minutes instead of more regularly. Feel free to add to this if you want. Matt Colbeck, Andrew wrote: I'd second that... on both the observed behaviour and the request for documentation. I'm attaching my highly commented
RE: [Declude.JunkMail] Experience with 4.x
We have not seen this issue with SmarterMail. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jay Sudowski - Handy Networks LLC Sent: Wednesday, May 24, 2006 4:00 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x In light of this reference to WINSOCKCLEANUP being an iMail only issue, do I even need to enable this on our SmarterMail servers? -Jay -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 6:30 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x The only way that we have detected this was with Imail and mail being stuck in the spool. ...network stack causing loss of functionality for basic network operations is generic but if I remember correctly when this happened the admin was not even able to ping an outside server, which would suggest to me other IP communications fail as well. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe
Re: [Declude.JunkMail] Experience with 4.x
Hi Matt, So you see any substantive performance improvement over 2x? -Nick Matt wrote: Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: 30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS30 WAITFORMAIL500 WAITFORTHREADS200 WAITBETWEENTHREADS100 WINSOCKCLEANUPOFF INVITEFIXON AUTOREVIEWON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above it. WAITFORMAIL 500 - On my server, this never kicks in, but if it did, it wouldn't make sense to delay for too long because I could build up messages. A half second seems good. WAITFORTHREADS 200 - This apparently kicks in only when I reach my thread limit; sort of like a throttle. I don't want it to be too long because this should only happen when I am hammered, but it is wise not to keep hammering when you are at 100%. Sort of a mixed bag choice here. WAITBETWEENTHREADS 100 - I see this setting as being the biggest issue with sizing a server. Setting it at 100 ms means that I can only handle 10 messages per second, and this establishes an upper limit for what the server can do. I currently average about 5 messages per second coming from my gateways at peak hours, so I figured that to be safe, I should double that value. INVITEFIX ON - I have it on because it comes on by default and I don't know any better. I know nothing about the cause for needing this outside of brief comments. It seems strange that my Declude setup could ruin an invitation unless I was using footers. If this is only triggered by footer use, I would like to know so that I could turn it off. I would imagine that this causes extra load to do the check. AUTOREVIEW ON - I have this on for the same reason that Andrew pointed out. When I restart Decludeproc, messages land in my review folder, and I don't wish to keep manually fishing things out. If there is an issue with looping, it would be wise for Declude to make this only trigger say every 15 minutes instead of more regularly. Feel free to add to this if you want. Matt Colbeck, Andrew wrote: I'd second
Re: [Declude.JunkMail] Experience with 4.x
I indicated earlier that it looked like a relative 10% improvement (about the difference between 35% and 32% hourly average CPU utilization). I would think that this comes primarily from not needing to start the old declude executable every time, and the improvement might be more substantial on a system with a overtaxed disk. I don't think the code is any more efficient as far as actual scanning goes. Besides that, it's typically the virus scanners and external tests that eat up most of the CPU on a Declude system and not Declude itself, and those things haven't changed. I'm guessing that it might perform much better when redlined though if you tweak it right. I believed that old Declude when getting rammed with overflow seemed to not perform linearly with normal performance with free CPU, but instead dropped, probably due to having a lot of CPU wait time overhead (there's probably a more accurate term for this). The new Declude can be more effectively controlled so as to not bog down the system as much. So performance here might be 2x or more when redlined under optimal conditions. This effect would be maximized if Declude would tie launching threads to a CPU monitor instead of a fixed number with no real clue as to what is perfect under any particular situation. Matt Nick Hayer wrote: Hi Matt, So you see any substantive performance improvement over 2x? -Nick Matt wrote: Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: 30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS30 WAITFORMAIL500 WAITFORTHREADS200 WAITBETWEENTHREADS100 WINSOCKCLEANUPOFF INVITEFIXON AUTOREVIEWON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above it. WAITFORMAIL 500 - On my server, this never kicks in, but if it did, it wouldn't make sense to delay for too long because I could build up messages. A half second seems good. WAITFORTHREADS 200 - This apparently kicks in only when I reach my thread limit; sort of like a throttle. I don't want it to be too long because this should only happen when I am
RE: [Declude.JunkMail] Experience with 4.x
Matt, is your Delcude gatewayed? Or is it running on the same server as Imail? -Erik -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Wednesday, May 24, 2006 5:58 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I indicated earlier that it looked like a relative 10% improvement (about the difference between 35% and 32% hourly average CPU utilization). I would think that this comes primarily from not needing to start the old declude executable every time, and the improvement might be more substantial on a system with a overtaxed disk. I don't think the code is any more efficient as far as actual scanning goes. Besides that, it's typically the virus scanners and external tests that eat up most of the CPU on a Declude system and not Declude itself, and those things haven't changed. I'm guessing that it might perform much better when redlined though if you tweak it right. I believed that old Declude when getting rammed with overflow seemed to not perform linearly with normal performance with free CPU, but instead dropped, probably due to having a lot of CPU wait time overhead (there's probably a more accurate term for this). The new Declude can be more effectively controlled so as to not bog down the system as much. So performance here might be 2x or more when redlined under optimal conditions. This effect would be maximized if Declude would tie launching threads to a CPU monitor instead of a fixed number with no real clue as to what is perfect under any particular situation. Matt Nick Hayer wrote: Hi Matt, So you see any substantive performance improvement over 2x? -Nick Matt wrote: Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: 30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS30 WAITFORMAIL500 WAITFORTHREADS200 WAITBETWEENTHREADS100 WINSOCKCLEANUPOFF INVITEFIXON AUTOREVIEWON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above
Re: [Declude.JunkMail] Experience with 4.x
Erik, I have the primary Alligate Gateway installed on the same box as IMail/Declude, but on different ports. My IMail doesn't actually host any real users, it's just a container for Declude and storage for the lowest scoring spam. The idea is to dump as much illegitimate stuff as possible during the SMTP envelope through the use of the 'pre-scanning' gateway, and then do the heavy lifting with Declude. The one-box solution saves money, and the gateway also saves money since it reduces the burden of heavy lifting by reducing the volume. Matt Erik wrote: Matt, is your Delcude gatewayed? Or is it running on the same server as Imail? -Erik -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Wednesday, May 24, 2006 5:58 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I indicated earlier that it looked like a relative 10% improvement (about the difference between 35% and 32% hourly average CPU utilization). I would think that this comes primarily from not needing to start the old declude executable every time, and the improvement might be more substantial on a system with a overtaxed disk. I don't think the code is any more efficient as far as actual scanning goes. Besides that, it's typically the virus scanners and external tests that eat up most of the CPU on a Declude system and not Declude itself, and those things haven't changed. I'm guessing that it might perform much better when redlined though if you tweak it right. I believed that old Declude when getting rammed with overflow seemed to not perform linearly with normal performance with free CPU, but instead dropped, probably due to having a lot of CPU wait time overhead (there's probably a more accurate term for this). The new Declude can be more effectively controlled so as to not bog down the system as much. So performance here might be 2x or more when redlined under optimal conditions. This effect would be maximized if Declude would tie launching threads to a CPU monitor instead of a fixed number with no real clue as to what is perfect under any particular situation. Matt Nick Hayer wrote: Hi Matt, So you see any substantive performance improvement over 2x? -Nick Matt wrote: Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: 30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS30 WAITFORMAIL500 WAITFORTHREADS200 WAITBETWEENTHREADS100 WINSOCKCLEANUPOFF INVITEFIXON AUTOREVIEWON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100
RE: [Declude.JunkMail] Experience with 4.x
Sounds like you have a very intensive setup. We run minimal filter tests, one virus scanner and Sniffer. When we have experienced spool backups, I've tinkered with the number of threads and found 80 seems to result in the best message throughput for our particular configuration. Any lower and we were not using the available resources, and any higher the stress on the system resulted in message processing slow downs. If we're facing a particularly bad queue backup, I will disable Sniffer and can further increase the number of threads without any impact on the overall time to process a single message. For our customers, a small amount of spam leakage is far better than delayed email. We process about 200,000 - 250,000 messages per day, and the majority of those are during normal business hours. As you can imagine, any small disruption to normal queue processing can result in a fairly large queue backup - a 30 minute issue during normal business hour can result in 10,000 queued messages. -Jay -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Wednesday, May 24, 2006 9:34 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: 30 threads seems awfully low. We set ours to 80 on a dual xeon box with a separate drive for spool/logging and we move right along without any issues. Thanks! - Jay Sudowski // Handy Networks LLC Director of Technical Operations Providing Shared, Reseller, Semi Managed and Fully Managed Windows 2003 Hosting Solutions Tel: 877-70 HANDY x882 | Fax: 888-300-2FAX www.handynetworks.com From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Tuesday, May 23, 2006 3:25 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS30 WAITFORMAIL500 WAITFORTHREADS200 WAITBETWEENTHREADS100 WINSOCKCLEANUPOFF INVITEFIXON AUTOREVIEWON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above it. WAITFORMAIL 500 - On my server, this never kicks in, but if it did, it wouldn't make sense to delay for too long because I could build up messages. A half second seems good. WAITFORTHREADS 200 - This apparently kicks in only when I reach my thread limit; sort of like a throttle. I don't want it to be too long because this should only happen when I am hammered, but it is wise not to keep hammering when you are at 100%. Sort of a mixed bag choice here. WAITBETWEENTHREADS 100 - I see this setting as being the biggest issue with sizing a server. Setting it at 100 ms means that I can
RE: [Declude.JunkMail] Experience with 4.x
The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
Matt, That would explain why the behavior on my server is different than yours - I do not use the WINSOCKCLEANUP and thus see the alternate behavior. Darrell --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. David Barker writes: The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Experience with 4.x
Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
Andrew has always been the King of Comments. Nick Hayer wrote: Very nice! It looks like Matt has taught you well on how to comment a file :) -Nick Colbeck, Andrew wrote: I'd second that... on both the observed behaviour and the request for documentation. I'mattaching my highlycommented declude.cfg as a reasonable sample. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Tuesday, May 23, 2006 10:36 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x David, That did the trick. I can't even see any messages in my proc folder any more. I might suggest adding your explanation to the comments in the file just in case others feel the need to turn this on like I did. I recalled the issues from the list and I turned it on because I didn't want the possibility of DNS crapping out and the leakage that this would cause. Here's a screen cap of what my processor graph looks like now: Thanks, Matt David Barker wrote: The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
Andrew, Thanks for your notes and their history. I'm using the following settings right now: THREADS 30 WAITFORMAIL 500 WAITFORTHREADS 200 WAITBETWEENTHREADS 100 WINSOCKCLEANUP OFF INVITEFIX ON AUTOREVIEW ON There are a few reasons for trying these values. THREADS 30 - I'm pretty confident that dual 3.2 Ghz Xeons and RAID can only handle 30 threads with average messages. In reality, one single message can spike the system to 100%, but these are uncommon. I figure that if I open this up too wide and I am dealing with a backup or something, launching more threads when at 100% CPU utilization will actually slow the system down. This was the same with 2.x and before. There is added overhead to managing threads and you don't want that to happen on top of 100% CPU utilization. I am going to back up my server later tonight to see if I can't find what the magic number is since I don't want to be below that magic number, and it would probably be best to be a little above it. WAITFORMAIL 500 - On my server, this never kicks in, but if it did, it wouldn't make sense to delay for too long because I could build up messages. A half second seems good. WAITFORTHREADS 200 - This apparently kicks in only when I reach my thread limit; sort of like a throttle. I don't want it to be too long because this should only happen when I am hammered, but it is wise not to keep hammering when you are at 100%. Sort of a mixed bag choice here. WAITBETWEENTHREADS 100 - I see this setting as being the biggest issue with sizing a server. Setting it at 100 ms means that I can only handle 10 messages per second, and this establishes an upper limit for what the server can do. I currently average about 5 messages per second coming from my gateways at peak hours, so I figured that to be safe, I should double that value. INVITEFIX ON - I have it on because it comes on by default and I don't know any better. I know nothing about the cause for needing this outside of brief comments. It seems strange that my Declude setup could ruin an invitation unless I was using footers. If this is only triggered by footer use, I would like to know so that I could turn it off. I would imagine that this causes extra load to do the check. AUTOREVIEW ON - I have this on for the same reason that Andrew pointed out. When I restart Decludeproc, messages land in my review folder, and I don't wish to keep manually fishing things out. If there is an issue with looping, it would be wise for Declude to make this only trigger say every 15 minutes instead of more regularly. Feel free to add to this if you want. Matt Colbeck, Andrew wrote: I'd second that... on both the observed behaviour and the request for documentation. I'mattaching my highlycommented declude.cfg as a reasonable sample. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Tuesday, May 23, 2006 10:36 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x David, That did the trick. I can't even see any messages in my proc folder any more. I might suggest adding your explanation to the comments in the file just in case others feel the need to turn this on like I did. I recalled the issues from the list and I turned it on because I didn't want the possibility of DNS crapping out and the leakage that this would cause. Here's a screen cap of what my processor graph looks like now: Thanks, Matt David Barker wrote: The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote
RE: [Declude.JunkMail] Experience with 4.x
Thanks, Nick. It's a defensivemechanism I've used for years: keep the documentation with the settings. I often do the same with registry keys by adding a text string and blathering away. Adding dates and initials is also a good idea. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Nick HayerSent: Tuesday, May 23, 2006 11:33 AMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Experience with 4.x Very nice!It looks like Matt has taught you well on how to comment a file :)-NickColbeck, Andrew wrote: I'd second that... on both the observed behaviour and the request for documentation. I'mattaching my highlycommented declude.cfg as a reasonable sample. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of MattSent: Tuesday, May 23, 2006 10:36 AMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Experience with 4.xDavid,That did the trick. I can't even see any messages in my proc folder any more. I might suggest adding your explanation to the comments in the file just in case others feel the need to turn this on like I did. I recalled the issues from the list and I turned it on because I didn't want the possibility of DNS crapping out and the leakage that this would cause.Here's a screen cap of what my processor graph looks like now: Thanks,MattDavid Barker wrote: The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Experience with 4.x
"Those who cannot remember their mistakes are doomed to repeat them!" Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Tuesday, May 23, 2006 12:26 PMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Experience with 4.x Andrew has always been the King of Comments.Nick Hayer wrote: Very nice!It looks like Matt has taught you well on how to comment a file :)-NickColbeck, Andrew wrote: I'd second that... on both the observed behaviour and the request for documentation. I'mattaching my highlycommented declude.cfg as a reasonable sample. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of MattSent: Tuesday, May 23, 2006 10:36 AMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Experience with 4.xDavid,That did the trick. I can't even see any messages in my proc folder any more. I might suggest adding your explanation to the comments in the file just in case others feel the need to turn this on like I did. I recalled the issues from the list and I turned it on because I didn't want the possibility of DNS crapping out and the leakage that this would cause.Here's a screen cap of what my processor graph looks like now: Thanks,MattDavid Barker wrote: The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. As WINSOCKCLEANUP is to be used only by those who experience DNS issues I would suggest running your tests again with WINSOCKCLEANUP commented out and see how the behavior differs. Also having the WAITFORMAIL to low can cause the CPU to process very high as it is constantly checking the \proc I would suggest a minimum of 500-1000 David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Monday, May 22, 2006 8:12 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Experience with 4.x
David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
I have an idea. Maybe this should be triggered automatically if every DNS lookup times out on a single message. That way we wouldn't have to set it, and it would only be called when conditions warrant. Matt Colbeck, Andrew wrote: David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: "David Barker" [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Experience with 4.x
David, that sounds like the case I saw that noted that his firewall wasn't allowing outbound DNS and also noted that implementing WINSOCKCLEANUP ON worked for him. I wasn't at all sure that the winsock fix was relevant for him! I'll keep watching my folders. Perhaps I'll get lucky enough to need to the fix and may offer some further insight here. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:30 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x The only way that we have detected this was with Imail and mail being stuck in the spool. ...network stack causing loss of functionality for basic network operations is generic but if I remember correctly when this happened the admin was not even able to ping an outside server, which would suggest to me other IP communications fail as well. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again
RE: [Declude.JunkMail] Experience with 4.x
Thanks for the help. Any feedback appreciated. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:36 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, that sounds like the case I saw that noted that his firewall wasn't allowing outbound DNS and also noted that implementing WINSOCKCLEANUP ON worked for him. I wasn't at all sure that the winsock fix was relevant for him! I'll keep watching my folders. Perhaps I'll get lucky enough to need to the fix and may offer some further insight here. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:30 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x The only way that we have detected this was with Imail and mail being stuck in the spool. ...network stack causing loss of functionality for basic network operations is generic but if I remember correctly when this happened the admin was not even able to ping an outside server, which would suggest to me other IP communications fail as well. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience
RE: [Declude.JunkMail] Experience with 4.x
Andrew I had this problme last year, decludeproc will suffer memory creep, slowly building over time. You can see if it's happening by opening taskmanager and checking the memory used by decludeproc. For us it would run for about 4 hours before a problem arose. I can't remember what finally fixed the problem for us. I have used the dnsoverride function to direct it to a less used dns server. But I can't recall if that was the fix or not. We might have had a firewall port issue. I'd turn the winsockcleanup off and monitor your memory usage. If it keeps creeping up turn it back on. John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review subfolder which require manual processing. - Original Message - From: David Barker [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, May 23, 2006 7:34 AM Subject: RE: [Declude.JunkMail] Experience with 4.x The purpose of WINSOCKCLEANUPON is to reset the winsock, what happens when using this setting is that when the \proc directory hit 0 decludeproc will finish processing all the messages in the \work before checking the \proc again. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL
RE: [Declude.JunkMail] Experience with 4.x
This sounds like me. I found some notes from my issue back in December. When upgrading to v 3 from v 2x we would loose connectivity every 3 or 4 hours. We finally figured out Decludeproc needs to get out on port 53 for some dns function. We allowed outbound port 53 for dns and the problem went away. This apparently is for the initial authorization of the software. I think this is a one time event. Upgrading from V3 to V4 we had the same problem. It was resolved with a phone call to Declude and the discovery that that the initial authorization had been moved from port 53 to port 25. After making that change to the firewall V4 runs fine. Good luck John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:36 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, that sounds like the case I saw that noted that his firewall wasn't allowing outbound DNS and also noted that implementing WINSOCKCLEANUP ON worked for him. I wasn't at all sure that the winsock fix was relevant for him! I'll keep watching my folders. Perhaps I'll get lucky enough to need to the fix and may offer some further insight here. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:30 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x The only way that we have detected this was with Imail and mail being stuck in the spool. ...network stack causing loss of functionality for basic network operations is generic but if I remember correctly when this happened the admin was not even able to ping an outside server, which would suggest to me other IP communications fail as well. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED
RE: [Declude.JunkMail] Experience with 4.x
Thanks for ringing in, John. Wow, this new-fangled support forum thingy does have it's uses! Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Doyle Sent: Tuesday, May 23, 2006 4:13 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x This sounds like me. I found some notes from my issue back in December. When upgrading to v 3 from v 2x we would loose connectivity every 3 or 4 hours. We finally figured out Decludeproc needs to get out on port 53 for some dns function. We allowed outbound port 53 for dns and the problem went away. This apparently is for the initial authorization of the software. I think this is a one time event. Upgrading from V3 to V4 we had the same problem. It was resolved with a phone call to Declude and the discovery that that the initial authorization had been moved from port 53 to port 25. After making that change to the firewall V4 runs fine. Good luck John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:36 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, that sounds like the case I saw that noted that his firewall wasn't allowing outbound DNS and also noted that implementing WINSOCKCLEANUP ON worked for him. I wasn't at all sure that the winsock fix was relevant for him! I'll keep watching my folders. Perhaps I'll get lucky enough to need to the fix and may offer some further insight here. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:30 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x The only way that we have detected this was with Imail and mail being stuck in the spool. ...network stack causing loss of functionality for basic network operations is generic but if I remember correctly when this happened the admin was not even able to ping an outside server, which would suggest to me other IP communications fail as well. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 6:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUP
RE: [Declude.JunkMail] Experience with 4.x
John, I've been monitoring DecludeProc.exe with Performance Monitor, specifically, the process counters for threads, cpu, handles and working set memory and have not seen any leaks. It was through monitoring the threads that I was able to comment on the sawtooth pattern that Matt and I were both seeing. I'm also using an in-house script to count the number of message files in the proc folder and email the results if the count is more than 100 to an internal mailserver's IP address thanks to a 3rd party SMTP tool (postie.exe)... That whole bandaid mechanism should survive any DNS relates WinSock outage. I think I've got my belt and suspenders on. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Doyle Sent: Tuesday, May 23, 2006 3:43 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew I had this problme last year, decludeproc will suffer memory creep, slowly building over time. You can see if it's happening by opening taskmanager and checking the memory used by decludeproc. For us it would run for about 4 hours before a problem arose. I can't remember what finally fixed the problem for us. I have used the dnsoverride function to direct it to a less used dns server. But I can't recall if that was the fix or not. We might have had a firewall port issue. I'd turn the winsockcleanup off and monitor your memory usage. If it keeps creeping up turn it back on. John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:26 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Thanks, David. I've read all of the support forum emails that have been posted on the WINSOCKCLEANUP and even reviewed them again via the mail archive website before my own implementation. What I haven't been able to tell is whether I can diagnose this issue if I have it before it becomes an outage. Can it only be detected by it's side-effect of filling up the proc folder? If I have a mechanism on my IMail server that does DNS queries... Will they fail when the WinSock needs being cleaned up? I think not, as at least one posting specifically mentioned that IMail IP4R tests worked when DecludeProc IP4R tests timed out. Your official description for WINSOCKCLEANUP ON says ...network stack causing loss of functionality for basic network operations; is this deliberately generic so that you don't have to explain what a DNS test is, or does it imply that other IP communications will also fail, e.g. SMTP and (critically for me) RDP? Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 3:12 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Andrew, In certain cases we found that Imail would stop resolving, it seemed that stop/starting the decludeproc or smtp service fixed the problem by resetting the winsock. So we added WINSOCKCLEANUP to deal with this specific Imail issue. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, May 23, 2006 3:45 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x David, is there a proactive way to detect if an installation would benefit from the WINSOCKCLEANUP ON directive in declude.cfg? I would rather be able to detect this while it's happening than to react when I find that spam is leaking or that the proc folder is continually growing. Andrew. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Barker Sent: Tuesday, May 23, 2006 7:48 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Experience with 4.x Mike, 1. The WINSOCKCLEANUPON activates when the \Proc reaches 0 2. If Decludeproc stops unexpectedly files it is busy with are move to the \review 3. You can use AUTOREVIEW ON to have these move back to the \proc 4. Be aware though if there is a real problem message you may find that the message gets looped 5. Make sure you have the latest version of decludeproc ... There should be a release later today or tommorow. David B www.declude.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike N Sent: Tuesday, May 23, 2006 10:23 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x I found that WINSOCKCLEANUP ON would force a reset if the \proc directory never hits 0. In this case, files build up in the \review
Re: [Declude.JunkMail] Experience with 4.x
The problem with this architecture is that when it moves a batch of messages into the work folder for processing, it quickly pegs the processor at 100% as it launches all of the threads, but most messages go through all of the steps quickly so the processors sit almost idle while it is waiting for DNS lookups and virus scanning (which I do after JM). Here's a sample of my combined CPU graph showing the pattern that this creates: I commented along time ago about this. Declude does give you options to work with this. Essentially changing your WAITBETWEENTHREADS setting works around this fairly decent. I use it to give just a bit of time between each of the threads to prevent (50+ external programs from being called at once. Now as you tweak that setting and others you will give yourself a more predictable cpu curve line. By tweaking the settings you can keep decludeproc processing messages and hopefully never entering the WAITFORMAIL cycle. Darrell --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
Darrell, I've tweaked the settings, knowing the issues before I tried 4.x. The problem is that because they are batch processing and because there is significant latency in scanning some messages to factors such as messages size, virus scanning and DNS timeouts, the server does nothing for long periods while it waits for one task to finish regardless of whether there is one message in proc or there are 1 million messages in proc. It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. Matt Darrell ([EMAIL PROTECTED]) wrote: The problem with this architecture is that when it moves a batch of messages into the work folder for processing, it quickly pegs the processor at 100% as it launches all of the threads, but most messages go through all of the steps quickly so the processors sit almost idle while it is waiting for DNS lookups and virus scanning (which I do after JM). Here's a sample of my combined CPU graph showing the pattern that this creates: I commented along time ago about this. Declude does give you options to work with this. Essentially changing your WAITBETWEENTHREADS setting works around this fairly decent. I use it to give just a bit of time between each of the threads to prevent (50+ external programs from being called at once. Now as you tweak that setting and others you will give yourself a more predictable cpu curve line. By tweaking the settings you can keep decludeproc processing messages and hopefully never entering the WAITFORMAIL cycle. Darrell --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
I have a few more things to add now with a little more testing. I let my gateways backup on E-mail so that I could slam Declude, and here's what happens. When Declude is not hitting it's THREADS setting, it waits until the work folder is empty before moving in a new batch. When Declude is hitting the THREADS setting, it seems to move in messages just as quickly as the threads are freed up. As soon as the proc folder no longer contains more messages than your THREADS setting, Declude again starts waiting until the work directory is empty before moving in another batch. Here's a graph of one of these periods: So while hitting your thread limit, it seems to be able to reach virtually full server capacity, but it performs much differently when you are under your THREADS capacity. This suggests that it is a bad idea to have your THREADS setting too high, though this is only because of the strange way that Declude acts. It should be able to do almost 100% under these circumstances, but it's not optimal behavior. Matt Matt wrote: I figured that i would just start a new thread for this instead of adding to the old one. This was the first time that I have used a version of Declude that runs as a service. I'm unfortunately surprised and disappointed at how it handles things, but a lot more makes sense now. In a nutshell, what happens is now Declude batch processes messages, and waits for every message from a batch to complete processing before it grabs a new batch. The time that it takes for a batch to process is variable, and mostly depends on virus scanning and DNS timeouts/slowness. Some batches complete in 10 seconds, but some take more than a minute. The one that I noticed that took over a minute was the result of 3/4 of that time spent on the last message which needed to be virus scanned, and the message was large so it took a while to be virus scanned. The problem with this architecture is that when it moves a batch of messages into the work folder for processing, it quickly pegs the processor at 100% as it launches all of the threads, but most messages go through all of the steps quickly so the processors sit almost idle while it is waiting for DNS lookups and virus scanning (which I do after JM). Here's a sample of my combined CPU graph showing the pattern that this creates: This is actually the most optimal pattern, but in one where there is a significant delay in one message can go on for over a minute at lower than 5% CPU utilization while it waits for one message that requires extra work. During peak times on my server, I am seeing between 10 and 40 messages being moved to "work" at one time. The net result of this is that my server will only handle less than half the number of the number of messages that it could if I could actually ride 100% and launch threads as messages came in instead of in threads. Under this architecture, there is nothing that I can do about this. I also don't like the idea of hitting 100% so frequently as it does since running at 100% causes a much increased chance of instability. This architecture also explains why so many posted here about E-mail backups. Their default settings would move in less messages than they were collecting in the proc folder while waiting for things to be processed. I'm sure that this graph doesn't look nearly as bad with a vanilla install of Declude, but with 4 external tests, one of which does multiple serial DNS lookups, and then the latency of having two virus scanners run in serial means that finishing up a batch takes more time, which is less time that the CPU is actually doing much. Others have reported that Declude 3+ is faster than 2-. I run SNMP monitoring of my server's CPU that takes one minute averages, and when you average those out over an hour, my peaks are maybe 10% lower relative to before (i.e. 32% instead of 35%). So it does apparently work more efficiently to some extent, but there is no way that my server could ever surpass 50% hourly averages when using the batch processing, so my server's capacity is actually less than half of what it was with 2.0.6.16. I am absolutely astonished that I haven't seen anyone comment about this before. What Declude needs to do is not wait for an entire batch to complete. They should set a maximum thread limit, and then allow us to configure the number of free threads to wait for before picking up new messages to process. So instead of waiting 30 seconds for a virus scan to complete on one message with 40 waiting in the proc folder, it should start picking up messages as soon as it sees free threads. I also immediately came across a bug with whitelisting. For some reason WHITELIST IP is now being applied before IPBYPASS, and to complicate matters, it won't log for JunkMail at LOGLEVEL MID when this happens. In 2.0.6.16 and before, IPBYPASS was applied before any processing based on the IP was done. I'm guessing that when they fixed
Re: [Declude.JunkMail] Experience with 4.x
Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Experience with 4.x
I'd sure like to see some Declude comments on this discussion. Ben BC Web - Original Message - From: Matt [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Monday, May 22, 2006 5:12 PM Subject: Re: [Declude.JunkMail] Experience with 4.x Darrell, I put up two Windows Explorer windows side-by-side under normal volume and the pattern was consistent where the proc folder grows while the work folder shrinks until the work folder hits zero at which point the proc folder empties out and everything lands in work and then the pattern repeats with proc growing while work shrinks. My settings are as follows: THREADS50 WAITFORMAIL100 WAITFORTHREADS10 WAITBETWEENTHREADS50 WINSOCKCLEANUPON AUTOREVIEWON INVITEFIXON Matt Darrell ([EMAIL PROTECTED]) wrote: It's a faulty design that leaves more than half a server's CPU capacity unused due to the mere fact that they wait for all threads to complete before moving in a new batch. I can't speak to what you see on your server, but that is not how it is running on my server. I just double checked again to make sure I am not crazy, but as I watch the thread count on my server (decludeproc) the threads fluctuate between 7 - 30 ( threads currently set to 50). It is not uncommon to see the threads move as follow: 11,8,10,7,15, While I was watching it I never seen a case where it went down low enough for the WAITFORMAIL setting to kick in. Watching the proc/work directory you can see files moving in and out, but never really emptying out. Its possible what I am seeing is an anomaly or maybe I am interpreting it wrong. Maybe David can comment on this. Darrell invURIBL - Intelligent URI filtering plug-in for Declude, mxGuard, and ORF. Stop spam at the source the spamvertised domain. More effective than traditional RBL's. Try it today - http://www.invariantsystems.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
