-----Original Message-----
From:
[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matthew Bramble
Sent: Wednesday, September 17,
2003 3:54 PM
To: [EMAIL PROTECTED]
Subject: [Declude.JunkMail]
DYNAMIC - 09/17/2003 - A new filter to detect IP'd reverse DNS entries
Ok, I've been testing this one for about a week with
very positive results. It's still a work in progress as far as exclusions
go (candidates welcome), but I have been using it with a good deal of success
as is for the past week. The filter is called DYNAMIC and it can be
downloaded at the following location:
http://www.mailpure.com/decludefilters/dynamic/Dynamic_09-17-2003.txt
(Links to the most recent versions of the filters that I have been testing are
located at the bottom of this message. I will put up some HTML soon to
help enable the process since I have noted a few people downloading older
versions from older postings to this group)
What the DYNAMIC filter
does is detect E-mail from a sender with a reverse DNS lookup that has the
tell-tale marks of being used for dial-up, DSL or cable broadband access.
I have found it to be very useful in scoring spam and it has a good impact on
messages that don't fail many tests without being responsible for rejecting
messages due to false positives. As an extra added bonus, the use of the
WHITELIST AUTH functionality that Scott announced yesterday is beneficial to
this filter's use (explained in the file).
The method is a little controversial because it doesn't look for direct signs
of spam such as OBFUSCATION, GIBBERISH or GIBBERISHSUB, but instead looks at
where the message is coming from, knowing that dial-up, DSL and cable broadband
address space is becoming increasingly problematic for spam origination, maybe
due to recent virus outbreaks installing SMTP servers or backdoors on always-on
connected machines. There are plenty of examples where such space though
hosts legitimate mail servers without customized reverse DNS, typically being
business users. Declude's own servers should trip this test if not
whitelisted. Therefore the scoring is low, however in a recent thorough
test of over 1,000 filter hits (excluding Declude of course), the false
positive rate was still only 2.0% of filter hits and nothing failed because of
this test alone. Unlike the other filters that I have recently been
testing, this one doesn't tend to catch opt-in advertising, just small-busuness
false positives that have mostly properly configured machines that score very
low, so adding a few points to some of them is of no real harm.
This test also often crosses over into DUL territory, especially the less than
pure EASYNET-DYNA blocklist. Because of that, one should be careful to
adjust the scores so that a double hit won't fail a message alone. I also
use SORBS-DUL which seems remarkably pure to the idea of being dynamic
addresses where mail servers aren't allowed to be hosted on, so I don't feel
there is any danger in having that test as a part of the mix. Please see
the detailed comments in the filter file for more information on
configuration. For those statistically inclined, I did a painstaking
review on 2 days of traffic in order to get an impression on exactly what the
impact was:
DYNAMIC FILTER STATISTICS
==================================================================
5,530 - Unique Incoming Messages
4,183 - Messages Rejected as Spam from All Filters (75.6% of Unique Incoming
Messages, approximate)
1,053 - Filter Hits (19% of Unique Incoming Messages)
==================================================================
1,032 - Positives (98.0% of Filter Hits)
21 - False Positives (2.0% of Filter Hits)
=================================================================
70 - Hits That Made a Difference* (6.6% of Filter Hits)
23 - Spams Failed or at Least Scored Because of Filter (2.2% of
Filter Hits)
0 - False Positives Failed Because of the Addition of This
Filter (0.0% of Filter Hits)
OTHER NOTABLES
==================================================================
604 - EASYNET-DYNA & DYNAMIC Hits (57.4% of DYNAMIC Filter Hits)
86 - SORBS-DUL & DYNAMIC Hits (8.2% of DYNAMIC Filter Hits)
6 - Number of Spammers That Spoofed Local User (0.1% of
Unique Messages)
*I define "Hits That Made a Difference" as spams that would have
scored at or below 150% of fail weight without test. My scoring has
improved immensly with many new filters added, so default configurations should
benefit much more in this area.
APPROXIMATE EASYNET-DYNA COMPARATIVE STATISTICS*
===================================================================
873 - Filter Hits (15.8% of Unique Incoming Messages)
===================================================================
604 - EASYNET-DYNA Filter Hits in Common with DYNAMIC Filter (69.2% of
Filter Hits)
369 - EASYNET-DYNA Filter Hits Not in Common with DYNAMIC FILTER (30.8%
of Filter Hits)
449 - DYNAMIC Filter Hits Not in Common with EASYNET-DYNA (42.6% of
Filter Hits)
*Approximated because I wasn't capturing and instead assumed a similar
percentage of hits out of the total on Unique Incoming Mail as seen with the
DYNAMIC filter, and checked against all individually logged messaged.
Links to the most recent versions of all of the recent filters that I've
shared:
DYNAMIC
http://www.mailpure.com/decludefilters/dynamic/Dynamic_09-17-2003.txt
GIBBERISH and ANTIGIBBERISH (use in combination)
http://www.mailpure.com/decludefilters/gibberish/Gibberish_09-16-2003.txt
http://www.mailpure.com/decludefilters/gibberish/AntiGibberish_09-16-2003.txt
GIBBERISHSUB and ANTIGIBBERISHSUB (use in combination)
http://www.mailpure.com/decludefilters/gibberishsub/GibberishSub_09-15-2003.txt
http://www.mailpure.com/decludefilters/gibberishsub/AntiGibberishSub_09-15-2003.txt
OBFUSCATION
http://www.mailpure.com/decludefilters/obfuscation/Obfuscation_09-14-2003c.txt
Feedback is important, so please feel free to post a comment or send me an
E-mail even if you aren't sure about your conclusion.
Thanks,
Matt
