Re: [Declude.JunkMail] Negative weighting filters to reduce false positives

[Darin Cox]

Hi Kami,   I like the way you've named your tests.  It's also good to know that others have already done similar Negative weighting for the larger ISPs while keeping some of these marginal tests.   I had even thought about turning it into an RHSBL.  Anyone have any interest in pooling lists to create one? Darin.     ----- Original Message ----- From: Kami Razvan To: Declude.JunkMail@declude.com Sent: Friday, April 15, 2005 5:58 AM Subject: RE: [Declude.JunkMail] Negative weighting filters to reduce false positives Hi;   Just to add my 2 cents to this discussion.   My experience is similar to others with these two but I again think they have their place as a factor in combo filters.   This is how we use these two.   A filter for the RFC Ignorant   File name: Combo_RFC_Ignorant.txt   TESTSFAILED   END   CONTAINS   [EXCEPTION.RFC-IGNORANT]   TESTSFAILED   0   CONTAINS   [RFC-IGNORANT.rhsbl.NOABUSE] TESTSFAILED   0   CONTAINS   [RFC-IGNORANT.rhsbl.NOPOSTMASTER] TESTSFAILED   0   CONTAINS   [RFC-IGNORANT.rhsbl.DSN] TESTSFAILED   0   CONTAINS   [RFC-IGNORANT.ip4r.BOGUSMX]   Then we have a filter that defines the exceptsions to the rule and a filter for large ISP's.   File name: Exception_RFC_Ignorant.txt   STOPATFIRSTHIT   TESTSFAILED    0 CONTAINS [EXCEPTION.LARGEISP]   REVDNS 0 ENDSWITH .cheetahmail.com REVDNS 0 ENDSWITH .cv.net REVDNS 0 ENDSWITH .fcps.edu ....   File name: Exception_LargeISP.txt   STOPATFIRSTHIT   REVDNS 0 ENDSWITH .abac.net REVDNS 0 ENDSWITH .adelphia.net REVDNS 0 ENDSWITH .aol.com REVDNS 0 ENDSWITH .aplus.net REVDNS 0 ENDSWITH .att.net REVDNS 0 ENDSWITH .cavtel.net ...   I use weight 0 for both of the RFC Ignorant lists but use them in my elevate filters as one factor in making the decision to add weight.  Naturally if the RFC Ignorant filter is not triggered because of exception filters then those tests will not have effect on the emails originating from large ISP's.    Regards, - Kami   From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Darin Cox Sent: Thursday, April 14, 2005 8:29 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Negative weighting filters to reduce false positives Well you make a good point on the value of NOABUSE and NOPOSTMASTER.  NOABUSE hits on about 18% of incoming mail, while NOPOSTMASTER hits on about 10%.  Most of the false positives we see from them can be eliminated with these filters, extending their usefulness....so there's still value in them for us.   Certainly REVDNS would avoid forging issues... we just went with MAILFROM initially since many of the false positives we saw were coming from various mailservers, some in the domain, some not....but the filter files could mix MAILFROM and REVDNS as desired, depending on the patterns seen. Darin.     ----- Original Message ----- From: Scott Fisher To: Declude.JunkMail@declude.com Sent: Thursday, April 14, 2005 6:03 PM Subject: Re: [Declude.JunkMail] Negative weighting filters to reduce false positives I myself have pondered why I am even running the RFCI-noabuse and RFCI-nopostmaster test. The NoAbuse misfired 23.6% of the time. The NoPostmaster misfired 12.7% of the time. Due to the underperformance, I weight each test 5 (hold at 200).   The test failures are who's who of ISP / webmail providers. The vast majority of the results are from e-mails with faked mailfrom addresses... Zombie spammers.   I wonder if you'd be better of using the REVDNS instead of the Mailfrom. ----- Original Message ----- From: Darin Cox To: Declude.JunkMail@declude.com Sent: Thursday, April 14, 2005 4:41 PM Subject: [Declude.JunkMail] Negative weighting filters to reduce false positives We just started something I've been thinking about for a while:  Negative weight tests to offset specific test failures for well-known domains.  For example, a large number of false positives we see are from Earthlink, Mindspring, Sprint, Verizon, etc.   Now you may be thinking, of course, these are large providers with dial-up user bases, so you would expect a large percentage of false positives to be from them...but hold on a minute.  Many of these large domains are being penalized in our system for routing or not having abuse@ or postmaster@ addresses.  Almost all of these would not have ended up in the hold queue if they had not been so penalized...thus the idea to figure out a manageable way to NOT penalize them for these technical RFC violations.   So, what we've done is to start filters to counteract the weights for major tests that a few of these domains fail.  By doing it specifically for a particular domain, we reduce false positives but avoid losing the effectiveness of the test on other domains.   Anyway, attached zip are the filter files.  As I mentioned, they have just been started, so there are just a few domains in them at present.  At the top of the filter file are suggested guidelines on how to use them.  There are probably better ways to handle this, so I welcome comments/feedback. Darin.