Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/12 14:46:25, David F. Skoll wrote: We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. We use grey listing on our low volume server, and as others have noted, it works well because a high percentage of spam bots do not bother to retry. But as others have mentioned, it can be painful waiting for the delayed confirmation on a registration to a web site to come in an hour/two later, or email from a new client who is waiting on a response. Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Assume that either some to-be-implemented SA filter, or some mail gateway front-end (like MIMEDefang), adds a new tag/two, for example: SENDER_FIRST_RCPT, SENDER_LOW_FREQ, SENDER_HI_FREQ, or SENDER_HI_AVE_SA_SCORE? All these tags might be based upon some look back period (say: 90 days). Theoretically, these new tags could be calculated after the fact when passing through a spam corpus. And since many/most grey listing systems differentiate by some form of (sender, recipient) pairing this analysis can be reliably/repeatably performed by an SA plug-in at the point of delivery to the user, if needed. It would need to be shown that these new tags improve the ability to discriminate spam from ham. If the scheme worked well, there might be no need for grey listing at all.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. We use grey listing on our low volume server, and as others have noted, it works well because a high percentage of spam bots do not bother to retry. But as others have mentioned, it can be painful waiting for the delayed confirmation on a registration to a web site to come in an hour/two later, or email from a new client who is waiting on a response. Using dnswl.org to whitelist against greylisting might help some. Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Assume that either some to-be-implemented SA filter, or some mail gateway front-end (like MIMEDefang), adds a new tag/two, for example: SENDER_FIRST_RCPT, SENDER_LOW_FREQ, SENDER_HI_FREQ, or SENDER_HI_AVE_SA_SCORE? All these tags might be based upon some look back period (say: 90 days). Theoretically, these new tags could be calculated after the fact when passing through a spam corpus. And since many/most grey listing systems differentiate by some form of (sender, recipient) pairing this analysis can be reliably/repeatably performed by an SA plug-in at the point of delivery to the user, if needed. It would need to be shown that these new tags improve the ability to discriminate spam from ham. If the scheme worked well, there might be no need for grey listing at all.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Mon, 2012-12-03 at 07:23 -0800, Gary Funck wrote: Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Yes. If you keep a list of the recipients of outgoing mail its easy to whitelist any mail you receive from them. This approach does what you want: a sender is treated as suspicious until you've sent mail to them and recipient list maintenance is easy to automate. I use a mail archive system as my recipients list because it has a record of everybody I've sent mail to. I use an SA plugin to access the archive. The combination of it and an associated rule will whitelist anybody who is recorded in the archive as having received mail from me. However, the database archives messages at 4-6 /sec, so this and/or the storage requirements (4.3 GB to store 143,000 messages) may mean that, if you're a high volume site and/or don't need an archive, you'd be better off just keeping a list of the recipient(s) of outgoing messages. I wrote my archive for personal use because I can find an old e-mail with the archive search tool faster than I can by ferreting though a set of mail folders: it was never designed as a high volume solution, but should manage small business volumes quite easily with both it and SA running on a typical desktop PC. Up to early this year I was using an 866 MHz P3 with 512MB RAM that easily kept up while PostgreSQL,the archive, Postfix and SA. That is all now running on a 3GHz dual Athlon with 4 GB RAM but not going any faster - an upgrade to Fedora 16 forced the change because its installer wouldn't run in less than 1GB RAM. If you think my SA plugin or the mail archive would be of use to you, contact me off-list. Martin
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Mon, 3 Dec 2012 07:23:59 -0800 Gary Funck wrote: Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Personally I wouldn't want to do it that way round - with a positive score for unknown rather than a negative score for known. YMMV but almost all of the FPs I've had in the last ten years have been that sort of mail because it's less likely to be recognised by Bayes.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 08:46 PM, David F. Skoll wrote: [...] Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Do you treat yahoo like spam sources in the same way?
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 12:27, Andrzej A. Filip wrote: On 11/29/2012 08:46 PM, David F. Skoll wrote: [...] Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Do you treat yahoo like spam sources in the same way? There's almost no point in greylisting an IP that you know will retry properly anyway, so why wouldn't you allow that IP to bypass greylisting in the future? -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
Am 29.11.2012 20:46, schrieb David F. Skoll: On Thu, 29 Nov 2012 14:36:45 -0500 vec...@vectro.org wrote: I've never had any complaints about delivery speed, but some senders have broken mail servers that don't retry on receiving a temporary failure. Many such servers use broken SMTP implementations that can't handle a 4xx code in response to RCPT properly. We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Regards, David. greylisting isnt state of art, however it might helpfull in some domains ( everyone has its own spam), using postscreen with postfix before selective greylisting is a good choice Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 09:31 PM, Dave Warren wrote: On 11/29/2012 12:27, Andrzej A. Filip wrote: On 11/29/2012 08:46 PM, David F. Skoll wrote: [...] Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Do you treat yahoo like spam sources in the same way? There's almost no point in greylisting an IP that you know will retry properly anyway, so why wouldn't you allow that IP to bypass greylisting in the future? I assume that greylisting of yahoo like spam sources increases chances of bulk detectors detecting spam. Is not it trues? [based on real data]
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 21:27:19 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Do you treat yahoo like spam sources in the same way? With respect to greylisting, of course. If a machine passes greylisting once, it's extremely likely to pass it in future and it's an utter waste of time to greylist it. Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 09:53 PM, David F. Skoll wrote: On Thu, 29 Nov 2012 21:27:19 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Do you treat yahoo like spam sources in the same way? With respect to greylisting, of course. If a machine passes greylisting once, it's extremely likely to pass it in future and it's an utter waste of time to greylist it. Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? [ based on your experience ]
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 21:59:45 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? [ based on your experience ] I suppose it might, but I don't use razor, pyzor, dcc or anything similar so I have no personal experience. Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
I've never had any complaints about delivery speed, but some senders have broken mail servers that don't retry on receiving a temporary failure. Many such servers use broken SMTP implementations that can't handle a 4xx code in response to RCPT properly. We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Every 60 seconds we look at all messages that arrived in last 60 seconds. If there Spamassassin score is less the 1 we add that server to a whitelist for 6 months. If its already on whitelist we update the last message time. If a message scores over 5 we remove it from whitelist if its on it. We do not greylist servers on the whitelist. Works very well. Even though we use greylisting our users very rarely notice if at all due to this.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
Just wondering how many boxes: rcpt domains: rcpt users: you guys are sending through greylisting. Axb
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012, David F. Skoll wrote: On Thu, 29 Nov 2012 21:27:19 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Do you treat yahoo like spam sources in the same way? With respect to greylisting, of course. If a machine passes greylisting once, it's extremely likely to pass it in future and it's an utter waste of time to greylist it. Modulo spamvertised URIs and spam checksums sent via such hosts, particularly if they are freemail. Filtering out the spambots who don't retry (and as trivial as that is to defeat, a large amount still gets blocked by this in my experience) is not the _only_ reason to greylist. Giving the URIBLs a chance to list a new URI and the checksum services a chance to recognize a new body are also benefits of greylisting. (But, as you said, you don't take advantage of those tools.) Also, greylisting generally keys on host+sender, not just host. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Bother, said Pooh as he struggled with /etc/sendmail.cf, it never does quite what I want. I wish Christopher Robin was here. -- Peter da Silva in a.s.r --- 26 days until Christmas
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 22:47:45 +0100 Axb axb.li...@gmail.com wrote: boxes: About 50 000 rcpt domains: About 2000 rcpt users: Lots. I don't have an exact figure. you guys are sending through greylisting. This is on our machines. Our larger customers have significantly higher numbers. Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? No. A remarkable fraction of ratware still doesn't bother to retry, so the most simple minded greylister will deter them. That's why it's useful. I've never seen any support for the theory that greylisting delays make it more likely that the host will be blacklisted when it retries. I haven't seen many legit senders that don't retry as David says he has, but I don't have his volume of mail, either.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 30 Nov 2012, John Levine wrote: Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? No. A remarkable fraction of ratware still doesn't bother to retry, so the most simple minded greylister will deter them. That's why it's useful. I've never seen any support for the theory that greylisting delays make it more likely that the host will be blacklisted when it retries. It's not so much the host being blacklisted, as a checksum of the spam being published by pyzor et. al., or for spamvertised websites in the spam being published by URIBLs, so that when the sender tries again the score for that message will be higher than it would the first time around, hopefully high enough to classify it as spam rather than a FN. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Bother, said Pooh as he struggled with /etc/sendmail.cf, it never does quite what I want. I wish Christopher Robin was here. -- Peter da Silva in a.s.r --- 26 days until Christmas
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 18:01:38 -0800 (PST) John Hardin jhar...@impsec.org wrote: It's not so much the host being blacklisted, as a checksum of the spam being published by pyzor et. al., or for spamvertised websites in the spam being published by URIBLs, so that when the sender tries again the score for that message will be higher than it would the first time around, hopefully high enough to classify it as spam rather than a FN. I would love to gather some hard data on this. Maybe a research project for the future... since we do our greylisting post-DATA, we could in principle run all the content-filtering and URIBL lookups and check if the score changes between the first attempt and the final attempt after greylisting. Or those who use SA without greylisting could reprocess messages after an hour or two and see if the score goes up. [My gut instinct says that a reasonable greylisting interval is too short for most DNSBLs to react. Pyzor/Razor/DCC may be somewhat more adept at reacting quickly.] Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 17:37, John Levine wrote: Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? No. A remarkable fraction of ratware still doesn't bother to retry, so the most simple minded greylister will deter them. That's why it's useful. I've never seen any support for the theory that greylisting delays make it more likely that the host will be blacklisted when it retries. If I run my accepted-and-quarantined spam corpus through a filter to test against DNSBL effectiveness, I always see higher effectiveness ratings than what was shown during the SMTP phase. I haven't done so in recent enough memory to have any actual numbers, but when I last did a comparison, slow moving DNSBLs showed little/no change at all, fast-acting trap-driven ones show more of a difference. Now I've not studied the exactly amount of time it takes for hosts to start getting listed, but since I only greylist questionable stuff already and since I whitelist aggressively, I've been able to set my greylisting in the 30-60 minute range without too many seizures from users and with higher rejection counts -- Since greylisting doesn't cause higher reject counts, I assume (yes, just assume) that it's due to higher hit rates. I admit that it would make sense to do further testing, but for fast-acting DNSBLs, and body-hash based systems, it makes sense that the longer one defers a message, the greater the odds of a hit against a new zombie or a new spam-run. -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 18:54, David F. Skoll wrote: [My gut instinct says that a reasonable greylisting interval is too short for most DNSBLs to react. Pyzor/Razor/DCC may be somewhat more adept at reacting quickly.] Something trap-driven like NIX is a candidate. No, it's not safe enough to reject based on it's output, but it was worth use in a scoring system. Invalument too responds reasonably quickly, enough that it sometimes tripped during the greylist period. The other trick is how you define reasonable. A reasonable greylist period for greylisting all mail is about 3 seconds, otherwise you'll have users screaming. However, if you only greylist questionable stuff to start with (rDNS failures, mismatches, etc, SPF fails, borderline-spammy stuff, DUL hits), you can get away with much longer times since most of it is crap anyway but a greylist period can help let the odd gem through. -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
