Re: [Declude.JunkMail] Experience with 4.x

[Matt]

Pieced below: Jay Sudowski - Handy Networks LLC wrote: I am contemplating implementing Alligate with SmarterMail in some fashion, and would like to pick your mind on the following: 1) SmarterMail does a great, great job at handling a huge number of SMTP threads so dictionary attacks are not really an issue. We have popped up to 2000 SMTP threads without incident. And since addresses being attacked do not exist, the messages get rejected with a 550 error, connection is closed and there's no real strain on the server (at least in terms of I/O) -- do we get any benefit from putting Alligate in front of SmarterMail for dictionary attack prevention? Definitely.  Not all dictionary attacks hit completely bad addresses, but detecting and stopping an attacker regardless of the good or bad addresses that they may send is wise.  Tarpitting takes the bites out of the attacks by slowing them down between recipients, and you can tell Alligate to drop the connection after x number of bad addresses, or even have more tolerance for a higher number of bad addresses if a good address is found.  Lastly, SmarterMail and IMail were designed to be mail servers and not pre-scanning gateways.  Gateways should be lean, and by also being smart, they can eliminate the vast majority of spam long before your real mail server comes into play.  Nothing is blackholed either, so any FP should be noticed by the sender if they should occur, and they should be rare if you configure it carefully. 2) Does SMTP AUTH pass-through and recipient verification actually work, all the time? I don't use this since it is not necessary to do so for pre-scanning purposes.  This is really designed to take the load off of a hosted mail server by off-loading this traffic.  I can't say personally that it works perfectly, but I'm sure that it does.  Other pass-through verification stuff that it does definitely works, and works efficiently. 3) I see greylisting as the real area that could benefit us -- we would avoid a lot of the zombie spam by implementing greylisting, and that would save us tons of resources all around. However, I understand that some zombies are now greylist aware and will retry. Greylisting is huge, but it has fundamental problems that kept me from using it in vanilla form with ORF.  I know that Nick used it with ORF and backed off.  My big issues are two-fold.  By arbitrarily applying greylisting to everyone, you will effectively block E-mail from sending software that doesn't spool (and zombie spamware isn't the only thing with that limitation).  It's things like blackboxes and other forms of notification programming that lacks such functionality and can fail greylisting, at least for the first message.  Some bulk mail (ecommerce and otherwise) will only send once, or may requeue and send through a different IP subsequently causing delivery failures or long delays.  Even more important to me is the fact that greylisting causing issues with delaying legitimate E-mail, and I refused to implement it in ORF solely for this reason.  It's not acceptable for the class of service that I want to offer.  I want E-mail to pass through our system in less than 5 seconds except in extreme circumstances. I fed Brian a framework for doing what I call "selective greylisting" which resolves these issues.  Essentially it only greylists when conditions warrant, and the only drawback is greylisting poorly configured hosts or some that leak spam (but not all of either).  It works so well that there is no reason to have it do full greylisting.  I won't go into the details of how this is done here because to some extent I consider it to be intellectual property.  Also note that Brian isn't just a coding monkey, and he contributed a lot to the end product and filled in many gaps.  There are also advanced tarpitting techniques that cause almost as many failures as greylisting does because some spamware detects tarpitting and disconnect, but it is variable according to when and how long it occurs.  The last major piece is the MXRate functionality, and it blocks a lot even when limited to just 100% hits (which surely aren't perfect, but they are very near perfect). While I don't feel like running DLAnalyzer, we typically trend about 85% of email is spam that gets deleted before it's delivered, which means we probably had around 40,000 actual deliveries. If we can eliminate 50% of the spam using MXRate and greylisting, that is a huge burden off of declude and the server in general. I block 99.9% of all spam and our false positive rate is less than 1 in 10,000, with most of that being some form of advertising sent from hosts that also send for dirty lists, and most don't care about such things.  Most of our issues with personal E-mail are the result of foreign traffic, especially with China.  Alligate certainly helps since it eliminates almost all zombies and viruses, but it also stops some static spammers as well, but Declude is still of course critical, and so is Sniffer and Eradispam, as well as some of my own programming.  My stats are accurate since they are based on manual review in repeated circumstances.  We also hold less than 1% of blocked E-mail for review making the task easy, and therefore false positives are more readily found. Out of the net 288,842 messages that our gateway blocked, only 40,137 of >them received permanent 550 errors not related directly to bad recipients. >All of the others were blocked by methods that are more passive in nature >and designed to exploit weaknesses of spamware. Is this a reference to greylisting or some other feature in Alligate I'm not seeing? I described the main techniques above.  It is a combination of address validation, tarpitting, greylisting and MXRate, but not all address validation, tarpitting, and greylisting is created equally.  I have never seen anyone else even talk about selective greylisting for instance, and I'm so damn grateful that I now have it.  The devil is in the details though, and even not all Alligate setups will achieve such results with such accuracy just like Declude setups. Please also note that Brian once was bashed here for promoting his product while in beta, and I'm sure that I might be pissing a few off by talking about it, but I have tried to steer away from talk of Alligate outside of mentions except for where people ask since I believe it is valuable information and that is why we are here.  Declude could easily plug into Alligate if they wanted to since it supports dropping files into a directory instead of delivering them, and then it will pick them up when the external app is done.  I would love to see that happen since I only need IMail as a container for Declude.  If there is still a widespread adversion to this discussion, I would be happy to take it off-list. Matt