Warning: This is a very long email about DNSBLs. On Thu, Mar 04, 2004 at 02:47:21AM +0100, Steven Glogger wrote: > hi andy > > > Unless you are implying "too unstable FOR ME", then I disagree: > > there are plenty of well-managed, sensible DNSBLs in existence. > > Sorting them from the insane ones can be hard, it requires research, > > and they don't all work the same way or suit the same needs. > > There's no such thing as a magic bullet. > > i didnt wanted to offend somebody.
You didn't offend *me* at least. :) I was just trying to point out that it's not really possible to say "these DNSBLs are sane, these ones aren't", because it depends too much on too many variables. For example, few of us would probably disagree with a person who said "a DNSBL that shuts down overnight and starts blocking all email is insane, and I will never trust another DNSBL run by that person" (you may recall this happened recently). On the other hand, you said you are willing to block at SMTP time based on bl.spamcop.net, and I know a lot of people who would call that insane. :) It also depends on the stated goals of the DNSBLs. Some DNSBLs try to list compromised machines, they list based on tests they do, and so if you find an IP on those lists then you can be fairly confident that it is a waste of time talking to that IP. Examples of these lists: CBL, NJABL proxy, SORBS proxy, OPM, ... Some DNSBLs attempt to map dynamic IPs (for various interpretations of "dynamic"). Most of these just try to list anything which is domestic, dynamic, and not supposed to run mail servers. If you're willing to reject a bunch of real email from people who "shouldn't be directly sending email" then these are great too, because they block vast amounts of spam. NJABL DUL, SORBS DUL, PDL, ... Some DNSBLs are geographically-based or ISP-based. They simply list IPs that are believed to be delegated to networks in a particular geographic region, or from a particular ISP. They can be useful for scoring, or for small-scale use where for example, no email from China or from Rackspace is expected. The various http://blackholes.us/ lists would be one example. Some DNSBLs attempt to list spam sources. Unlike all previously mentioned categories, this one is subjective and is based on the opinions of the people who report the spam. Sometimes humans make mistakes. Sometimes legitimate email is reported as spam. SBL, spamcop, SPEWS, ... Some DNSBLs are completely silly. They are invented either to LART people who configure any DNSBL they find, or else they are only meant for individual use. I have seen DNSBLs which reject email if the IP address as a 32-bit integer is a prime number, or if it is divisible by 2. > what i ment with unstable was (and of course from my personal point of > view), that we've seen enough DNSBLs closing down in the last some months > (due attack or whatever). This is absolutely true, but a DNSBL going down is not really a problem provided that the DNSBL is run sensibly. Almost all popular DNSBLs offer an rsync feed so that anyone who wishes to use them can take regular copies and host the zones locally. Tools like spfilter allow a set of DNSBLs to be aggregated into one file so that multiple zones can be looked up in one query if desired. Running local copies of zones tends to be the only option anyway once the amount of queries per day builds up to levels that large ISPs represented here would experience. Assuming a local copy, then, unreachability ceases to be a problem. The only problem left is what happens when the DNSBL voluntarily shuts down. History has shown several different methods chosen by DNSBL operators: > e.g. easynet.nl list (doesnt exists), The maintainer gave weeks and weeks of notice of these zones going away, and then either the host went away or else it was emptied (I forget which), either way, not really a problem for its users. > monkeys (closed down), Gave weeks of notice. Finally started returning everything as positive so that people still using it got legitimate email rejected. I won't defend that choice, but I've seen worse... > sorbs (seems to be back, was under heavy attack last year > afaik), SORBS comprises many lists, some of which have policies which I personally cannot agree with, but the DUL part (taking over from Easynet NL) and the proxy part are good examples of their type. You're right that SORBS has been recently unreachable for long periods of time due to DDoS but I do suggest that instead of blaming the guy for being down when his not-for-profit project is attacked by criminal spammers, people could help him out by providing mirrors. And of course rsync is available to create own private local copy which means that if he gets DDoS'd then it doesn't affect you. > orbz.org (which is now a porn site??) and so on. Yes. The point is that you're right to be wary because it is giving away some control to essentially unaccountable people who could do something unpredictable one day like return positive for any query. But at the same time, it is not as bad as you suggest. > i know, there are many other (and for sure) well managed other blacklists > (and we use some of them) like ordb.org, spamhaus, visi.com, etc.and some > blacklists are (for me) just not quite useful (like blacklists which are > collectiong a whole bunch of dialup ip's etc. - i've made a bad experience > after having 3-5 calls from customers each day complaining about not getting > all their mails from abroad [and i dont want to start here a discussion > about running a public mail server in a dialup/dynamic enviroment, because > they can use the ISP's mailserver]). Sure, it's a matter of whether your userbase will accept the types of false positives that each DNSBL will generate. As you say, DUL-style DNSBLs generate false positives where people are sending legitimate email direct from their mail servers on dynamic IPs (domestic cable, dsl, etc.) without using their ISP's smarthost. It does not matter that a good DUL-style DNSBL will block vast amounts of virus and proxy spam if it also blocks too much email that your users want. Even incredibly conservative proxy DNSBLs like OPM or CBL will block legitimate email because some people with insecure proxies or proxy-dropping trojans are on dynamic IPs. They get listed, they disconnect, a new user gets the IP, that user might send email directly, and find they are blocked. The numbers get worse for proxy DNSBLs that don't care much about dynamic IPs (NJABL, SORBS). > i think it's quite hard for each of us deciding which RBL to use and which > not. some have a very high level (which could have a too hight amount of > false positives) and some not. It is hard. There is also a thread going on about this on nanog at the moment. It is better to be conservative, and to remember that you do not need to always block at SMTP time. Personally I would recommend use of cbl.abuseat.org and opm.blitzed.org to anyone at the SMTP level. Any IP in those DNSBLs is very very likely to be an active, owned machine, and thus not worth talking to. The false positive rates are vanishingly low. If your users will accept it, one of the DUL-style lists (SORBS DUL or NJABL DUL) is good here too. I would not use the biggest proxy DNSBLs at SMTP time unless I was also using a DUL-style one, since there is a lot of overlap. Other lists that are good for scoring purposes (accepting the mail but marking in headers, or spamassassin-style use, etc.) are sbl.spamhaus.org and bl.spamcop.net. One of the several much bigger proxy lists (SORBS proxy, NJABL proxy) are also worth using in scoring mode as well. That's just my opinion though, based on my own actual stats and that of others. Some people prefer to be more conservative (to the point of using no DNSBLs at all), others less so. Some people reading will probably take the opinion that they just pass bits around and it is up to the customer to decide if they want those bits or not. The postmaster for a customer network can use a lot of DNSBLs and be very fascist if *his* only "customers" are employees of the same company, since he only has to keep business-related mail flowing! > > It should also be pointed out that _although you personally may not > > have noticed_, bl.spamcop.net is a DNSBL based on the opinions of > > its human contributors and has been known to make mistakes. If that > > level of false positive is okay with you then that's fine, but it > > probably won't be for some other people, which is why everyone > > should research what they use based on their own needs. > > in the last 1-2 years (which i'm using spamcop) i never had a single > complaint from a customer. so i thought, that this level was not bad. It's not bad, on the whole. I know that Julian and his spamcop deputies do a really good job, but they do make mistakes, and the teeming masses of spamcop reporters make mistakes too. I've seen first hand the complaints from legitimate marketers (yes, real, legitimate marketers who have never spammed) who get blacklisted for no good reason. It's not because "spamcop sucks" -- it doesn't suck. It's just that spamcop is an opinion-based blacklist and people are notorious for reporting things as spam that aren't spam, but really just "mail I don't want". Sometimes they even report mail they do want, like emailed bank statements! A really common scenario for spamcop goes like this: - Huge company, ISP, or service provider decides it wants to send a mail out to its customers about some new aspect of its service. The company might be SBC, AOL, Red Hat, Microsoft, or whoever. We will call this $COMPANY. - $COMPANY is not in the business of doing massive mail runs, so it decides to look for an email marketing company to mail this out. $COMPANY is well aware of spam though, it doesn't want to spam, nor to use an email marketing company that is known for spam. $COMPANY knows that if it picks a spammy marketing company then that company will likely be heavily backlisted already, and it will be counter-productive. - After much money spent on research, $COMPANY finds an email marketing company who are squeaky clean about spam, have a very high reputation, and because of that they charge lots of money for their services. $COMPANY is glad to pay $MARKETER the high fee though, because it is still less money than it would cost to set up the infrastructure for the mail run itself. - The deal is done, $COMPANY passes on the copy of the email to $MARKETER, $MARKETER begins the mail run. Don't forget that this is email coming from a totally clean network, and going to customers of $COMPANY, who have a pre-existing relationship and who have requested this type of email. It may even be important account information, required by law to be sent to notify about change in terms and conditions etc. - Some idiot customer at the start of the mail run reports the email as spam. This is known to happen even with emails that contain important account info. This is known to happen even with emails that users requested minutes earlier. Spamcop blacklists $MARKETER based on a single report. This has happened multiple times in the past. - Rest of email run proceeds and huge amounts of it are blocked because $MARKETER is now in spamcop. At the same time, other email runs currently taking place for other customers of $MARKETER are not failing. There is a publically visible page at spamcop.net which now says that $MARKETER is a spammer. So what a shambles this has become. $COMPANY went to an extreme amount of effort to find a whitehat marketing company and still got blocked by antispammers, they probably wonder why the hell they should work with antispammers at all when even the best efforts fail in this way. $MARKETER has had its reputation damaged, its core business of ethical email marketing damaged, and is now experiencing an increase rate of delivery failures and having to explain to their clients why mail runs are being blocked because of "spam" when it isn't spam. It is easy to go a very long time without ever seeing this, unless you happen to be one of these non-spamming email marketing companies (yes! there are some), or unless you happen to have a very vocal customer who missed out on mail he wanted. So, that is a typical type of problem with spamcop, which you can go a very long time without seeing until it suddenly becomes a big deal for someone... This does not mean that bl.spamcop.net is worthless for everyone. Hope that was informative for someone at least. ---------------------------------------------- [EMAIL PROTECTED] Maillist-Archive: http://www.mail-archive.com/swinog%40swinog.ch/
