Hi all, First, sorry for the (possibly - probably - undue) length of this note!
This thought has been bugging me for a while now, and I think it might be the time to let you ponder over it too, what with my needing sleep sooner or later and so on. I especially want to know if the idea is a bad one simply because it might be a little bit dishonest; it relies on SMTP client behaviour being almost always predictably simple. The extension needed to implement the idea will be a little different than most, even if most clients won't notice a thing while they are being treated to its compulsory effects. So here goes: Greylisting ( http://www.greylisting.org/ ) would be a neat trick but for one essential problem that is overlooked too often: it makes the assumption that all mail transactions, having been issued at least once, are always issued again and again until the delivery is successful from the same host that made the initial attempt. That is, the IP address of the connecting client is a part of the identification of that transaction, and changing it is impossible without triggering a new greylisting. Without the IP address as part of the greylisting process, of course, you lose most of the defense offered by it. There are even Greylisting implementations which rely exclusively on the IP address, with quite a good success rating. However, in environments where clustering, proxying, load balancing or gatewaying are used to share the load of distributing mail from an identical source, greylisting will delay mail for longer than is necessary if multiple attempts happen to be made by different hosts in a cluster. If there are enough hosts sending mail, and depending on the greylisting timeouts and the queueing timeouts set at recipient and sender of the mail respectively, and supposing every host that tries is uniquely chosen, it might even be possible that mail will fail to be delivered in the time the sender decides mail can wait for delivery in a queue. The mail will then be returned to the sender as undeliverable for a transient reason, which most greylisting implementations obfuscate as general system faults. (Although it seems that this problem is already known well enough, no-one seems to have noticed any such returned mail and plenty of people are using greylisting now.) There is also the fact that the delay is applied to a *transaction*, not a *connection*; spammers are not punished for not trying again, they simply don't get their mail delivered in some cases when they fail to come back and the few times when the mail does get delivered they have simply been lucky enough to make the attempt with all the right circumstances at a later time when greylisting wasn't being imposed rather than because they were conscientiously trying to deliver again. Ideally, there would be a way to make the spammer suffer until greylisting were no longer needed for a time. My idea is to replace greylisting with a connection-delaying technique that will make the SMTP client wait until we're certain it is genuine. We differ from Greylisting only in how we determine that an SMTP client is genuine; greylisting uses the fact that the client will come back, and we'll use the fact that the client knows what continuation lines are and that it must stay silent while we send periodic lines in the HELO/EHLO response for a given time. This isn't unlike teergrubing, except that we don't need dedicated hosts for this technique and we slow the connection down as soon as it's possible rather than while mail is being delivered. We must not allow the client to pipeline any commands before EHLO or HELO, and we must not allow the client to initiate a MAIL transaction until issuing EHLO or HELO and completing the challenge either (unless, at the server's discretion, the client has already "Proven" itself). A client that waits ("Proves itself") for a specified duration in minutes (five, say), during which it is receiving five-secondly (for instance) "Stay on the line" notices, shall eventually be allowed to see service extension lines and continue the mail transaction, while its IP address is stored in a whitelist. The client must not say anything (or have pipelined anything in advance) beyond the HELO/EHLO line, for otherwise it shall be banished in an administratively-defined fashion (probably just a 421 response followed by a connection closure, but there's nothing to stop you blacklisting, firewalling, or what have you; people sensitive about read buffers can just empty the buffer with nonblocking reads, issue a final "250 XTARPIT Sorry, matey, you've blown it!" and then reject every following line except QUIT with 503). Whitelisted entries run down to zero in seconds, starting with some administratively-defined time (a month, say). How I came up with the idea: For a time, I used to rely on the fact that most genuine clients could wait up to the RFC 2821-defined five minute read timeout. Unfortunately (of course, there has to be someone, doesn't there?) not all hosts would wait even a minute. Sendmail's greet_pause rule gave me the idea of trying long timeouts before the 220 greeting in the hope that spammers would drop connections in disgust, not for just avoiding HTTP proxies identified with pre-greeting traffic. They did. I was on Windows at the time with Mercury/32, so I wrote a proxy (in Tcl, naturally) to do the delay work, drop connections with pre-greeting traffic and log local or remote closures and then experimented with various timeouts. I soon noticed that spammers would be gradually (surprise, surprise) willing to wait longer and longer, until I had one waiting the full 150 seconds (2.5 minutes) without closing. From the addresses I saw connect, I'd assume they were zombie boxes. Soon thereafter, it was what I saw most of the time. By this time, I was losing connections from such prominently uninteresting services as Gmail.com, and so gave up the hope. With this scheme, I hope to take away any final excuse any self-respecting SMTP client might have for giving up too early, by essentially not letting them time out. I think it can work, and that we can set our timeouts as high as we like and still manage to exhaust spammer patience without severely impacting our mail deliveries or, should we fail to impact the spammers when many people are implementing this and spammers are willing to wait, to at least increase the risk that their mass-mailing tools will break under them on however many machines they are deployed upon. Each waiting session to a mailer is a thread or process wasted, but only the spammers lose by not being patient - the rest are tied up only for one occasion in a set period. Spammers are, as we all know, lazy, and look for excuses to move on. The one thing they really do have, though, is potentially limitless botpower; we have to put ourselves in a position where we can really challenge it. Now, to the extension. This is where we feel a bit dirty. I hope you can see so far that there are no opportunities for delaying the client before he greets us (no, we can't tamper with the TCP send window, nor can we add continuation lines to the 220 greeting line without definitely stepping on very shaky ground [RFCs 821 and 2821 don't clearly provision for a multiline response that isn't sent in response to a command]), and there's no point in delaying the client once he's greeted us since our PIPELINING extension is there for a reason and we want every good mailer to use it. (We might do this tarpit at the MAIL command if we didn't send PIPELINING to a new client and did to a known-good one, but then the first transaction of a new client would always have to be non-pipelined and if it happened to be, say, the output of a mailing list job for a given host, we'd have a lot of waste. However, if the client is using HELO he can never use pipelining anyway, so we can delay it until MAIL if anyone here thinks it would help at all.) The extension must be repeated multiple times in the EHLO response. This is the important bit (and the point of this long message). The below conditions are found to be true with Sendmail: This is all for nothing if SMTP clients will not ignore unknown extensions by simply forgetting about them and expect all of the lines sent to be in one write! Personally, I'm pretty sure this is true for *most* truely robust clients, but it's also true that this extension makes novel use of the response by outputting lines on a timer. OpenBSD's spamd stutters characters on a line; however, I am aware of at least one modern implementation which will actually choke if the entire line is not sent in one write with CRLF, never mind the group of lines. It may be that the extension used can convey additional data to the client, such as how long it'll have to wait, but it could just as easily (and this is probably better for security reasons) convey a human-readable string (see below for an example). There's no way for me to experiment with this idea easily without writing my own MTA (something I was thinking of doing anyway, also mostly in Tcl, as it happens), as Sendmail, my current choice, can't possibly be extended without pretty heavy patching. I might try to whip up a proxy, but it'll be tricky if I still intend to keep all the other service extensions and if I can get other people's ideas first that would go a long way. Using the time-honoured convention of "C: " marking lines sent by an SMTP client and "S: " marking those sent by the SMTP service, here's an example of the whole thing in action, as I imagine it. The client is connecting from the same address on two occasions. Not all the possibilities are shown here, but that can wait. Remember that the HELO command works in the same way as EHLO except that the extensions aren't listed (or, as stated above, it could work in the usual way and the tarpit could be delayed until MAIL if it helps since there can be no pipelining anyway). Also, a server might (but it isn't shown here) be enterprising enough to note the hostname in EHLO/HELO in the whitelisting along with the IP address, so that it can react differently when the same host issues another name on a reconnection. I don't know if this is necessarily a good idea, although it shouldn't affect most clients. Here's the example: S: [listening on SMTP port] C: [Connects to server's listening port] S: 220 bloodstone.yamta.org ESMTP YAMTA (MEOW!) C: ehlo goodmail.example.net S: 250-bloodstone.yamta.org Okay, I'll run off and hide. You wait. S: 250-XTARPITTING Be quiet! S: [Hesitates 5 sec] S: 250-XTARPITTING Be quiet! S: [Hesitates 5 sec] S: 250-XTARPITTING Be quiet! S: [... and so on until five minutes have passed] S: 250-XTARPITTING No, you're nice! (PURR) Feed me! I support: S: 250-PIPELINING S: 250-8BITMIME S: 250-BINARYMIME S: 250-CHUNKING S: 250-CHECKPOINT S: 250-SIZE S: 250-DSN S: 250-AUTH CRAM-MD5 DIGEST-MD5 PLAIN S: 250 HELP C: [Goes on to complete any number of MAIL/RCPT/DATA|BDAT transactions] C: quit S: 221 Nice to meet you! (RUB) S: [Closure of channel, back to listening] C: [After arbitrary amount of time prior to whitelist record expiry, connects again] S: 220 bloodstone.yamta.org ESMTP YAMTA (MEOW!) C: ehlo goodmail.example.net S: 250-bloodstone.yamta.org You again? (PURR) Feed me! I support: S: [ ... service extensions as before] C: [As before, transactions] C: quit S: 221 Nice to meet you! (RUB) S: [Closure of channel, back to listening] C: [closure of channel] I think that's everything. Please let me know what you think. Is it feasible? What would the implementers say? Are the assumptions I've made bad or violating anything? Even if it worked in practice (I'm quite sure it will), should it be discarded for being possibly a little too dependent on common implementation strategy (I.E. the robustness principle)? Cheers, Sabahattin -- Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com> Address harvesters, snag this: [EMAIL PROTECTED] Phone: +44 20 88008915 Mobile: +44 7986 053399
