Re: [sa-list] RE: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
On Wed, 10 Oct 2007, Bret Miller wrote: sa-update does NOT feed a local blocklist generated by *my* particular corpus of spam emails. Think of it as the RBL equivalent of sitewide-bayes. Or think of it as a way of SA saying when I get twelve spams of score 10+ from ip 208.23.118.172...I will feed the auto-expiring RBL, which *SENDMAIL* works off of, thus keeping my *SPAMASSASSIN* load lower. How do you call SpamAssassin? If whatever calls SpamAssassin in your setup knows what IP the connecting relay has, it can hopefully also do what you describe above. SpamAssassin doesn't really need to support this (through plugins or anything else) for it to be possible (and feasible). And I did something very similar as well. The problem I found is that you need a very large white list to avoid blocking big ISPs for a sudden flood of spam. I ended up rejecting legitimate email far too often from the temporary block. I still like the idea and would do it in a second if I could change the 5xx reject to a 4xx try later type of block. But I can't' without switching to a different MTA. milter-greylist lets me do this (reject 4XX based on a DNSBL). I've found it to be highly customizable, if not a bit of a memory pig. On the other hand, if there is a big ISP who is sending me spam...should they not be blocked, anyway? -Dan -- Long live little fat girls! -Recent Taco Bell Ad Slogan, Literally Translated. (Viva Gorditas) Dan Mahoney Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---
Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,
On Mon, 8 Oct 2007, Rob McEwen wrote: Therefore, I recommend that you re-think your choices here! Don't let your quest for guaranteed long-term perfection keep you from making **substantial** progress today! Rob, Then help rally the SA team to include those RBLs that you mentioned in the stock config. Also, rally them to update the documentation on the wiki on how to configure SA for third-party DNSBL's, because it blows (and refers to years-old versions of SA). Yes, I know the point of a wiki is that ANYONE can update it, but I'm not about to update it with information I don't understand for certain. ((Q: This documentation doesn't seem to cover how to configure dns-blocklists. It says Support for these is built-in but I can't believe that all free BL's is called each time a mail is beeing checked. There must be a way to configure which to use. A: You're right. You might look at the [WWW] Mail::SpamAssassin::Conf documentation page which I admit doesn't really say how to configure which DNSBL to use, or the rules file [WWW] 20_dnsbl_tests.cf, for internal details, but no clear examples of how to configure the inclusion of various DNSBLs either. For the latest list of DNSBLs you want to be using SpamAssassin version 2.63 or 3.0.0-pre2, for the same reason that you wouldn't use an out-of-date virus scanner, but that also doesn't really have anything to do with the question.)) Finally, rally them to pay attention to the topic I'm proposing here, which is: allow users to run their own RBL + feeder so that they can auto-rbl and floodgate themselves (and yes, it allows me to combine your corpus, plus my corpus, plus HIS corpus) in a scoring config, which is FUN...or it lets you say, quite simply SA said you sent too much spam, now sendmail won't listen for X hours per spam run. soapbox While I've had a long history of getting decent responses from the developers on this list some of the time -- nobody has managed to answer the questions I've asked in the previous thread: * can we do something with the ironport headers * can we do something with the SPF softfail which my MTA registered but SA didn't (and why didn't it?) * can we do something with the X-Originating-IP: 127:1 (is it a legit header, or is it there to evade filters?) * can we fix something about the DKIM_POLICY_SIGNSOME, * and after I changed the topic: Can we get a plugin that lets us feed our own blocklists, currently I get dictionary floods that are enough to overload SA (even right now). and many is the time I've just sent an email out to this list on a given topic, seen a lack of useful answer, and shrugged it off. /soapbox -- Check it out, it's just like Christmas. Except it sucks. -Jason Seguerra, 3/2/05 Dan Mahoney Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---
Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,
Dan Mahoney, System Admin wrote: On Mon, 8 Oct 2007, Rob McEwen wrote: Therefore, I recommend that you re-think your choices here! Don't let your quest for guaranteed long-term perfection keep you from making **substantial** progress today! Rob, Then help rally the SA team to include those RBLs that you mentioned in the stock config. Also, rally them to update the documentation on the wiki on how to configure SA for third-party DNSBL's, because it blows (and refers to years-old versions of SA). Yes, I know the point of a wiki is that ANYONE can update it, but I'm not about to update it with information I don't understand for certain. ((Q: This documentation doesn't seem to cover how to configure dns-blocklists. It says Support for these is built-in but I can't believe that all free BL's is called each time a mail is beeing checked. There must be a way to configure which to use. A: You're right. You might look at the [WWW] Mail::SpamAssassin::Conf documentation page which I admit doesn't really say how to configure which DNSBL to use, or the rules file [WWW] 20_dnsbl_tests.cf, for internal details, but no clear examples of how to configure the inclusion of various DNSBLs either. For the latest list of DNSBLs you want to be using SpamAssassin version 2.63 or 3.0.0-pre2, for the same reason that you wouldn't use an out-of-date virus scanner, but that also doesn't really have anything to do with the question.)) Finally, rally them to pay attention to the topic I'm proposing here, which is: allow users to run their own RBL + feeder so that they can auto-rbl and floodgate themselves (and yes, it allows me to combine your corpus, plus my corpus, plus HIS corpus) in a scoring config, which is FUN...or it lets you say, quite simply SA said you sent too much spam, now sendmail won't listen for X hours per spam run. soapbox While I've had a long history of getting decent responses from the developers on this list some of the time -- nobody has managed to answer the questions I've asked in the previous thread: * can we do something with the ironport headers * can we do something with the SPF softfail which my MTA registered but SA didn't (and why didn't it?) * can we do something with the X-Originating-IP: 127:1 (is it a legit header, or is it there to evade filters?) * can we fix something about the DKIM_POLICY_SIGNSOME, * and after I changed the topic: Can we get a plugin that lets us feed our own blocklists, currently I get dictionary floods that are enough to overload SA (even right now). Why would you be accepting messages to non-existent users? If you reject these at the MTA, then SA would never see them and your MTA would not have to deal with bounces to forged sender addresses (backscatter). Bill and many is the time I've just sent an email out to this list on a given topic, seen a lack of useful answer, and shrugged it off. /soapbox -- Check it out, it's just like Christmas. Except it sucks. -Jason Seguerra, 3/2/05 Dan Mahoney Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---
Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,
Dan, Then help rally the SA team to include those RBLs that you mentioned in the stock config. My RBL (ivmSIP.com) wouldn't work as a default value in SA because it is only available via RSYNC or Zone Transfer to subscribers (or... currently... testers who have specifically requested access). The other weird thing is that I use SA as a helper app in my spam filtering and I've custom written my own spam filter. Mostly, I still include SA in the mix for SARE rules ( other rules), as well as checksum filtering like RAZAR, etc. But I've turned off all RBL URIBL filtering in SA because I do those on my own and, most of the time, SA isn't even needed. As a result, I pay very little attention to many of the implementation details of RBLs in SA since I don't personally use them in SA. I have enough to worry about without these extra details. However, I'll be happy to share some tips that might help others or the SA folks with possible improvements in future versions. First, one thing that I did years ago (and continue to do) is that I'm always carefully reviewing lists that I might potentially use and/or am already using. For example, if I notice that a particular dnsbl is hitting on more and more messages which ultimately score under the spam threshold and, upon examination, I verify that most or all of these really are legit, them I'm at least going to lower the points assigned to hits on that dnsbl... and I might even remove that dnsbl from my spam filtering altogether. If, on the other hand, I find that ALL such messages really were spam, I might start increasing the points given to that particular list, assuming that I'm not also seeing some FPs from that list. Next, if I see a spam (that wasn't sent from a legit ISPs mailserver) and it scored rather low, I'll then take that IP and run it against a spam blacklist checker (dnsstuff, robtex, etc) to see if there are any RBLs that would have caught it, but that I'm not using yet. (Of course, I ignore various FP-ridden lists like APEWS in that search.) If I see a pattern whereby a particular list consistently hits on IPs that scored too low in my spam filtering, I might then add that dnsbl to my filtering... starting off with a low score... then double-checking for FPs... then bumping the score up depending on how little FPs there are. (in this case, I'm calling any hits on legit messages a FP, but, at this stage, these will generate too low a score to outright block and this FP really did get delivered to the inbox.) Doing this, over the years, I've added a good mix of RBLs with very fine tuned scoring (in my own spam filtering program, not referring to SA). At one point, I noticed that many of the more aggressive dnsbls are really really good at catching new IPs, but have too many FPs. As a result, I have to keep their score low. But it seemed such a shame because these IPs were taking too long to get on the FP-safe dnsbls. Then I noticed that, many times, three or four of the more aggressive RBLs would quickly hit on the same spammer's IP, where that IP that wasn't yet on SpamHaus, etc... then... if a few lists hit on that new spammer' IP, chances were, it was worthy of blocking in comparison to if just one list hit on it... so much so that the score really needed to be higher than merely the sum of the FP-risky dnsbl's scores. As a result, I changed my formula so that I took into account the number of dnsbls that hit on that IP as well as the score. (it was something like.. for every added dnsbld hit the overall RBL score would get increased by an additional 10% or 20%)... next, I adjusted down some of the raw scores so as to not allow the RBL scoring to get out of control. IOW... the whole really was worth more than the sum of its parts! Get it? Of course, even then, I have extensive whitelisting of IPs that I have placed in front of this... both my own (that I've put literally thousands of hours into!) and third parties. Currently, my own IP blacklist isn't (yet) on dnsstuff or robtex... but if it something like it were there and produced by someone else... I would have spotted it in that systematic checking that I described and I'd have been thrilled at its results... IOW... I created a product that I myself would have greatly desired to have if it had been created and distributed by someone else. I probably would have been one the first subscribers.. had this been someone else's product. (Why? Because my RBL provides that same fast reacting aggressiveness... just without the FPs!) Still, besides my own RBL's subscription barrier to inclusion... other lists which also require RSYNC access would not be able to come preinstalled in SA since they too need a little TLC to get up and running in one's spam filtering environment. These couldn't be used out of the box without some configuring of various programs on one's server. Something else to ponder. I hope this is
RE: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,
-Original Message- From: Dan Mahoney, System Admin [mailto:[EMAIL PROTECTED] Sent: Tuesday, 9 October 2007 7:14 AM To: Rob McEwen Cc: users@spamassassin.apache.org Subject: Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, On Mon, 8 Oct 2007, Rob McEwen wrote: Therefore, I recommend that you re-think your choices here! Don't let your quest for guaranteed long-term perfection keep you from making **substantial** progress today! Rob, Then help rally the SA team to include those RBLs that you mentioned in the stock config. Also, rally them to update the documentation on the wiki on how to configure SA for third-party DNSBL's, because it blows (and refers to years-old versions of SA). Yes, I know the point of a wiki is that ANYONE can update it, but I'm not about to update it with information I don't understand for certain. You should update the Wiki nevertheless and append a disclaimer of sorts! Choosing not to update in fear of appearing clueless is just lame! If you believe that what you are posting is halfway valid, then someone else can update. This is the sole function of a Wiki as otherwise there'd be no need for an UPDATE function!!! . . .
