Len, I don't think that anyone is doubting that the statistics you are presenting are good enough at stopping false positives FOR YOU. I think that you are missing the real point though. Every ISP, hosting provider, mail server admin, etc. has their own standards for what constitutes an acceptable false/positive ratio. In certain situations, for certain clients, NO amount of false positives are acceptable. Period. With others, higher numbers are acceptable, but only to a certain point. It is up to each client or mail admin to set that point for themselves.
The bottom line is, what is acceptable to you is not acceptable to everyone. Your "proof" that your system "works" provides just as much evidence for someone who is more conservative about false positives that your system does NOT "work". In other words, it's strictly a matter of personal preference. There is no right and wrong here. Neither you, nor Scott, are right or wrong, and neither of you should have to "shut up". I think that presenting the numbers is useful, but please draw conclusions about what the data should mean for yourself, not for every mail admin on the mailing list. Personally, I find .3% to be too high a ratio, but that's just me, and it reflects the type of clientele that I personally cater to. That is, customers with low volume, but highly important e-mail traffic. I am speaking about attorneys, law enforcement, accountants, Hospitals, etc. These people do not care about percentages or internet standards. They care about that ONE message that never got through to them. In my business, the customer is always right, and I am always striving to improve upon our reliability at reducing spam, while doing my best to get as close to 0% false positives as possible. I know that you have a different definition of what false positives are, and what makes e-mail legit or not. Again, your methodology works for you, and that is great for you and your customers. Everyone else has their own standards and opinions though, and I don't think that just because someone disagrees with your personal standards that they are "wrong". We all have our own opinions. I wish that everyone on this list would just realize that, and agree to disagree on some issues, because in this case there is no right or wrong answer. William Van Hefner System Administrator TheDigest.Com/TelCompare.Com > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Behalf Of Len Conrad > Sent: Thursday, September 18, 2003 7:40 AM > To: [EMAIL PROTECTED] > Subject: Re: [IMail Forum] Certain ISP's failure/IMail > > > For the experiment with dynablock running first, then my > subscriber filter, > for all 24 hours of Wednesday, the total rejects: > > 1 SMTP Exceeded Hard Error Limit after CONNECT > 1 RBL spamdomains.blackholes.easynet.nl > 3 SMTP Exceeded Hard Error Limit after END-OF-MESSAGE > 3 RBL opm.blitzed.org > 4 DNS no A/MX for @recipient.domain > 9 ACL helo_hostnames > 16 SMTP invalid [EMAIL PROTECTED] > 17 SMTP Exceeded Hard Error Limit after ETRN > 24 SMTP helo hostname invalid > 28 ACL header checks > 30 ACL mta_clients_bw > 32 ACL unauthorized relay > 39 ETRN Mail theft attempt > 49 ACL bogon network header > 88 ACL HTML obfuscation > 89 SMTP unauthorized pipelining > 95 SMTP invalid [EMAIL PROTECTED] > 109 ACL PTR hostname does not match hostname (forged HELO) > 113 RBL list.dsbl.org > 186 RBL proxies.relays.monkeys.com > 190 RBL relays.ordb.org > 256 SMTP Exceeded Hard Error Limit after MAIL > 353 DNS nxdomain for MTA PTR hostname (forged @sender.domain) > 414 RBL korea.services.net > 425 DNS timeout for MTA PTR hostname (forged @sender.domain) > 632 RBL dnsbl.njabl.org > 634 ACL forged @sender.domain not from sender PTR domain > 676 DNS no A/MX for @sender.domain > 919 ACL from_senders_bw > 1990 SMTP helo hostname is an IP > 2323 SMTP helo hostname not fully qualified > 2697 ACL from_senders_imgfx > 3126 RBL sbl.spamhaus.org > 4978 RBL blackholes.easynet.nl > 13887 ACL from_senders_slet > 18091 ACL mta_clients_dict > 28076 SMTP Exceeded Hard Error Limit after DATA > 69412 ACL subscriber network <<<<<<<<<< > 73298 SMTP Exceeded Hard Error Limit after RCPT > 96529 RBL dynablock.easynet.nl <<<<<<<<<< > 111142 ACL to_relay_recipients unknown recipient > ====================== > 430984 TOTAL rejects > > So the dynablock rejected 96.5 K msgs (16k are oranges, 80k are apples), > and then the subscriber filter rejected an additional 69K msgs (apples), > passed over by dynablock. > > The subscriber rejects contain 6087 unique IPs. > > The dynablock rejects contain 11556 unique IPs, 1811 of > which have no PTR > (oranges), meaning they would not be blocked by my subscriber > filter which > filters by PTR domain name. Note that dynablock has the disadvantage, > when it works, and a vulnerability when it doesn't, of having made, at > least, 11556 DNS queries to the dynablock remote server. My subscriber > filter is a local ACL file, much faster (scaleable) and more reliable (no > killing your mail server with timeouts for unreachable/slow/DDoSed RBL > servers). > > I scanned through the PTR hostnames of the dynablock rejects, and > there is > of course a huge amount of overlap between what dynablock rejects > and what > my subscriber filter rejects. Whatever, the outcome is that > dynablock let > pass 70K pieces of subscriber junk. (But I'm pretty sure I know > what's so > inferior about the dynablock concept approach to my subscriber filter > approach.). > > I'd bet good money that if I had run the experiment backwards, with my > subscriber filter first, and dynablock second (and disqualifying the > dynablock "oranges" of 16k rejects of 1811 PTR-less IPS since they are > illegit by definition and not targeted by the subscriber "apples" > filter), > of the remaining total of > > 69K + (97K - 16K PTR-less) = 150 K PTR-full rejects > > .... would have been very probably 150K rejected by subscriber and "round > off error" rejected by dynablock. :) > > The basic question is, for the subscriber filter, how many of the total > 6097 blocked IPs are legit mail servers? > > 1% would be 61 IPs, 0.1% (1 in 1000) would be 6 IPs. > > I found 4 IPs that were legit (all whitelisted now), with intelligible > [EMAIL PROTECTED] and the helo domain fully qualified agreeing, and a > credible > website, but in all these blocked_legit cases, the PTR hostname was the > network operator's domain (mismtaching the server's HELO > hostname), exactly > the same problem that Scott has with charter for mail.declude.com's IP. > > I am willing to extrapolate from the 4 IPs I found to 20, allowing that I > could have missed 10 to 15 legit IPs that sent only 1 or 2 msgs > each. There was certainly no way, looking at the crap, that 100s of IPs > were legit, or even the 60 IPs for a 1% rate. There is > ABSOLUTELY no way > the 70K subscriber msgs that were accepted as ok by dynablock > were all legit. > > So I estimate that 20/6087 = 0.3% would be a rough, max rate for > legit IPs > blocked by the subscriber filter, which would be reduced to well > below that > by running subscriber filter in warn_if_reject mode for a week to > catch all > the habitual legit senders screwed up on subscriber PTR hostnames. > > For your amusement, some observations on the crap: > > The top end of the rejects per IP: > > 8301 209.219.48.148 > 3840 209.219.48.136 > 2660 64.105.160.171 > 2225 64.105.160.174 > 1620 209.219.48.144 > 1252 199.222.161.177 > 1145 209.219.48.147 > 798 64.105.160.172 > 459 216.239.225.50 > 294 65.30.207.98 > 200 24.164.14.148 > 174 66.235.1.59 > 174 24.74.164.110 > 166 210.63.226.136 > 162 66.41.74.152 > 154 24.71.64.93 > 152 66.190.98.202 > 148 142.179.159.114 > 145 24.95.229.31 > 145 24.168.217.243 > 143 24.81.201.121 > 143 24.24.202.31 > 138 24.160.72.229 > 138 24.128.216.220 > 136 217.58.148.138 > 135 66.255.179.178 > 124 66.7.18.133 > 122 24.174.216.72 > 121 24.85.86.250 > 119 67.119.169.178 > 117 24.194.171.40 > 113 194.144.54.55 > 112 69.132.32.170 > 111 24.87.230.37 > 111 24.195.60.215 > etc .... for 6087 IPs > > 13674 subscriber rejects also had a not-fully qualified helo hostname, a > single criteria that also merits blocking. > > For the combined total of dyna + subs rejects: > > 151322 rejected msgs said HELO and only 14642 said EHLO. What good-faith > mail servers send HELO? postfix defaults to EHLO, and I suppose IMail, > Exchange, exim, sendmail, qmail all send EHLO unless programmed not > to. EHLO is preferred because it gets the ExtendedSMTP service > announcements like SIZE and MIME support and other useful features that > legit servers like to work with. otoh, spammers prefer HELO since the > don't want the delay of ESMTP announcements coming back. Subscriber IPs > sending over 10:1 HELO:EHLO is another mark against their legitimacy. > > In flagrant delicto, an entire ClassC (only one IP shown below) at > transedge exploiting the Veriscam DNS validation hole: > > 136.48.219.209.transedge.com[209.219.48.136]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bgcemx.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bgdcul.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bgdcul.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bgdplq.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bgfhzz.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bglxnt.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bglxnt.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bglxnt.com> > 148.48.219.209.transedge.com[209.219.48.148]: > from=<[EMAIL PROTECTED]> proto=SMTP helo=<bglxnt.com> > > Note on the above: sender@ is fixed length garbage added to forged > hotmail.com, and the helo domain is fully qualified junk hostname under > Veriscam .com. You can block MTAs in 209.219.48/24 or with PTR > transedge.com in complete tranquility. :) > > @sender.domain forgeries of the usual frequently forged domains: > > @aol 4702 > @hotmail 21677 > @msn 1173 > @earthlink 1214 > @yahoo 7324 > @bigfoot 298 > @concentric 123 > @compuserve 168 > @cs. 260 > @email 101 > @erols 110 > @free 208 > @gmx 891 > @juno 180 > @lycos 95 > @nycmail 1676 > @one of our domains 1411 > @prodigy 141 > > For the subscriber rejects, I have a file containing the PTR[ip], > [EMAIL PROTECTED], SMTP PROTOCOL, and HELO hostname fields if anybody > wants to see it, about 1 MB zipped. > > So for this experiment, the dynablock filter let pass about 70K msgs that > were blocked with my subscriber filter, where 99+% of those 70K were true > crap. The subscriber filter had a credible legit_IP/abuse_IP > ratio max of > 20/6087. Let's say those 20 legit IPs sent an avg of 5 legit msgs/day > (about only about 2000 of the 6100 IPs sent had 5 or more rejects). 100 > legit msg in 70K spam is roughly .2%. Good enough. > > If you guys want to nitpick and hairsplit to 4 decimal places, have at > it. I don't find that kind of accuracy helpful in running a mail > operation. > > These results totally refute Scott's unsupported, self-serving FUD of > "(subscriber filter) can have very, very high false positive > rate". I put > up, you shut up. > > This analysis also refutes Scott's claim for the dynablock filter being > "better" than my subscriber filter. If it's so "better", why did it let > pass 70k pieces of subscriber junk, a failure rate of approaching 50%? > > The above analysis of a sufficiently large sample supports my claim that > msgs originating from subscriber networks are 99+% crap, merit blanket > blocking, single-criteria rejection, and are finally just illegitmate > networks at this time. When the network operators stop the > effluent, we'll > stop the blanket blocking. (Is anybody holding their breath for > the stench > to clear?) > > Len > > > To Unsubscribe: http://www.ipswitch.com/support/mailing-lists.html > List Archive: http://www.mail-archive.com/imail_forum%40list.ipswitch.com/ > Knowledge Base/FAQ: http://www.ipswitch.com/support/IMail/ > --- > [This E-mail scanned for viruses by Declude Virus] > > --- [This E-mail scanned for viruses by Declude Virus] To Unsubscribe: http://www.ipswitch.com/support/mailing-lists.html List Archive: http://www.mail-archive.com/imail_forum%40list.ipswitch.com/ Knowledge Base/FAQ: http://www.ipswitch.com/support/IMail/
