Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Robert Schetterer]

Am 04.08.2016 um 22:30 schrieb Chris:
> Greylisting is just one of several tools available to a system
> administrator for filtering out spam

as multiple described it does not


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Chris]

I do not use postfix but I do greylist so I thought I would chime in
with my opinion.

Greylisting is just one of several tools available to a system
administrator for filtering out spam, like any of the other tools if
used incorrectly it will be problematic.

I do much cheaper filtering first before the server even considers
greylisting, filtering which does not require calling external tools,
does not require dns lookups.  That kind of filtering should be done
on messages first as it is cheaper.  Next I do not greylist every
single email, that would be excessive, I know some sysadmins do this,
but it would cause too much delays causing complaints and increase
server load with more retries.  Instead I greylist email's that come
from server's that fail basic 101 configuration checks, such as a
mismatched/missing rdns record, or failed spf check.  Whilst running
in this configuration for a number of years I have had zilch
complaints of missing emails, only the occasional moan about delayed
emails.  I also configure my server's so that end users who decide to
opt out can opt out, I have a whitelist file with target domain's that
will allow these failed rdns/spf emails to be delivered immediately
although they will still be subject to other checks unless whitelisted
in other checks also.

Regarding RBL lists, this one is perhaps not so simple, I do outright
block email's from certian lists I consider to be very reliable, aware
that occasionally the likes of gmail may find themselves on such a
list I exclude the major email providers from RBL checks, this of
course also reduces queries sent to those providers.  Plus as with the
greylisting, customers of mine can opt out of these checks.

List providers with history of false positives I tend to not use or I
may use them when they have a record of expiring senders quickly but
only using defer instead of deny, which should make the sender
reattempt delivery later.  I do not yet have a internal scoring
system, the only scoring system I use is spamassassin which is ok but
I found over the years it is definitely becoming less effective.

My plan is to combine RBL providers alongside some other spam
networking communities and use a scoring system, so I can do away with
the outright blocking, as although I do not get complaints, I respect
there is always the possibility of false positives.

There is also the option of delaying the incoming server for a few
seconds before allowing it to proceed, this can weed out spammers as
well who dont like been slowed down so may skip over it.

Chris

On 2 August 2016 at 22:18, John Hardin <jhar...@impsec.org> wrote:
> On Tue, 2 Aug 2016, Benny Pedersen wrote:
>
>> On 2016-08-02 20:00, John Hardin wrote:
>>
>>>  Is there any way to use postscreen as a frontend filter for a sendmail
>>>  MTA?
>>
>>
>> content-filter works nicely in postfix, but that postscreen will not use
>> content-filter to help on its problem
>>
>> postfix can use sendmail as a content-filter
>
>
> Guffaw.
>
>> what goal ?
>
>
> To get the benefits of postscreen without replacing my working sendmail
> install.
>
>
> --
>  John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
>  jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
>  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> ---
>   Vista "security improvements" consist of attempting to shift blame
>   onto the user when things go wrong.
>
> ---
>  3 days until the 281st anniversary of John Peter Zenger's acquittal

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[John Hardin]

On Tue, 2 Aug 2016, Benny Pedersen wrote:


On 2016-08-02 20:00, John Hardin wrote:


 Is there any way to use postscreen as a frontend filter for a sendmail
 MTA?


content-filter works nicely in postfix, but that postscreen will not use 
content-filter to help on its problem


postfix can use sendmail as a content-filter


Guffaw.


what goal ?


To get the benefits of postscreen without replacing my working sendmail 
install.



--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Vista "security improvements" consist of attempting to shift blame
  onto the user when things go wrong.
---
 3 days until the 281st anniversary of John Peter Zenger's acquittal

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 02.08.2016 um 23:02 schrieb Benny Pedersen:

On 2016-08-02 20:00, John Hardin wrote:


Is there any way to use postscreen as a frontend filter for a sendmail
MTA?


content-filter works nicely in postfix


which is not the topic


but that postscreen will not use
content-filter to help on its problem


that's why it's not the topic


postfix can use sendmail as a content-filter
what goal?


*postscreen as it is* in front of something else
there is nothing complex in the question you quoted above



signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Benny Pedersen]

On 2016-08-02 21:27, Robert Schetterer wrote:


you may use a complete postfix server including postscreen etc "before"
sendmailbut then it might better to simply change to postfix in
total, but such setups are often use with MS exchange


if that can serve as a content-filter it could be used in postfix 
standard smtp is nice imho :=)

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Benny Pedersen]

On 2016-08-02 20:00, John Hardin wrote:

Is there any way to use postscreen as a frontend filter for a sendmail 
MTA?


content-filter works nicely in postfix, but that postscreen will not use 
content-filter to help on its problem


postfix can use sendmail as a content-filter

what goal ?

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Bill Cole]

On 2 Aug 2016, at 14:00, John Hardin wrote:


On Tue, 2 Aug 2016, Bill Cole wrote:


What's special about the postscreen delay is:

1. It delays only the last line of a multi-line greeting, so it 
catches MANY more bots than a simple delay.


2. It caches PASS results so even the very short (6s by default) 
delay that it imposes only hits the first encounter with a client 
that connects frequently. This is critically important in high-volume 
situations where the difference between mean session lengths of 0.5s 
and 6s is the difference between 2 and 12 MX boxes in a cluster.


Combined, this is why Sendmail and other MTA greeting delays are less 
spectacularly effective than they used to be and less effective than 
postscreen. The resource cost of prolonging every session to 6s is 
untenable for busy machines, so bots that have adapted can get 
through. Back in the early days of Sendmail's GreetPause a value of 
3s would catch most bots but over the years some bots have adapted by 
doing their own hard delays and others have learned to wait for 
anything from the server. Few (if any) have adapted by actually 
parsing the greeting and making sure that they've seen the end of a 
multi-line greeting before talking.


That all sounds great.

Is there any way to use postscreen as a frontend filter for a sendmail 
MTA?


Not currently.

Architecturally, Postfix and Sendmail are so different that it isn't 
immediately obvious how one would hook postscreen to a sendmail process. 
A postscreen process accepts connections over the net and hands off the 
file descriptor for a "passing" connection to a smtpd process via a unix 
domain socket, after which it is not involved in that session. It's not 
a proxy or a pass-through filter, it's a connection handler.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Robert Schetterer]

Am 02.08.2016 um 20:04 schrieb Reindl Harald:
> 
> 
> Am 02.08.2016 um 20:00 schrieb John Hardin:
>> On Tue, 2 Aug 2016, Bill Cole wrote:
>>
>>> What's special about the postscreen delay is:
>>>
>>> 1. It delays only the last line of a multi-line greeting, so it
>>> catches MANY more bots than a simple delay.
>>>
>>> 2. It caches PASS results so even the very short (6s by default) delay
>>> that it imposes only hits the first encounter with a client that
>>> connects frequently. This is critically important in high-volume
>>> situations where the difference between mean session lengths of 0.5s
>>> and 6s is the difference between 2 and 12 MX boxes in a cluster.
>>>
>>> Combined, this is why Sendmail and other MTA greeting delays are less
>>> spectacularly effective than they used to be and less effective than
>>> postscreen. The resource cost of prolonging every session to 6s is
>>> untenable for busy machines, so bots that have adapted can get
>>> through. Back in the early days of Sendmail's GreetPause a value of 3s
>>> would catch most bots but over the years some bots have adapted by
>>> doing their own hard delays and others have learned to wait for
>>> anything from the server. Few (if any) have adapted by actually
>>> parsing the greeting and making sure that they've seen the end of a
>>> multi-line greeting before talking.
>>
>> That all sounds great.
>>
>> Is there any way to use postscreen as a frontend filter for a sendmail
>> MTA?
> 
> no - postscreen is not a smtp proxy
> 
> in fact the connection is handed over from postcreen to the smtpd
> process after a client has passed the tests
> 

you may use a complete postfix server including postscreen etc "before"
sendmailbut then it might better to simply change to postfix in
total, but such setups are often use with MS exchange


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 02.08.2016 um 20:00 schrieb John Hardin:

On Tue, 2 Aug 2016, Bill Cole wrote:


What's special about the postscreen delay is:

1. It delays only the last line of a multi-line greeting, so it
catches MANY more bots than a simple delay.

2. It caches PASS results so even the very short (6s by default) delay
that it imposes only hits the first encounter with a client that
connects frequently. This is critically important in high-volume
situations where the difference between mean session lengths of 0.5s
and 6s is the difference between 2 and 12 MX boxes in a cluster.

Combined, this is why Sendmail and other MTA greeting delays are less
spectacularly effective than they used to be and less effective than
postscreen. The resource cost of prolonging every session to 6s is
untenable for busy machines, so bots that have adapted can get
through. Back in the early days of Sendmail's GreetPause a value of 3s
would catch most bots but over the years some bots have adapted by
doing their own hard delays and others have learned to wait for
anything from the server. Few (if any) have adapted by actually
parsing the greeting and making sure that they've seen the end of a
multi-line greeting before talking.


That all sounds great.

Is there any way to use postscreen as a frontend filter for a sendmail MTA?


no - postscreen is not a smtp proxy

in fact the connection is handed over from postcreen to the smtpd 
process after a client has passed the tests




signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[John Hardin]

On Tue, 2 Aug 2016, Bill Cole wrote:


What's special about the postscreen delay is:

1. It delays only the last line of a multi-line greeting, so it catches MANY 
more bots than a simple delay.


2. It caches PASS results so even the very short (6s by default) delay that 
it imposes only hits the first encounter with a client that connects 
frequently. This is critically important in high-volume situations where the 
difference between mean session lengths of 0.5s and 6s is the difference 
between 2 and 12 MX boxes in a cluster.


Combined, this is why Sendmail and other MTA greeting delays are less 
spectacularly effective than they used to be and less effective than 
postscreen. The resource cost of prolonging every session to 6s is untenable 
for busy machines, so bots that have adapted can get through. Back in the 
early days of Sendmail's GreetPause a value of 3s would catch most bots but 
over the years some bots have adapted by doing their own hard delays and 
others have learned to wait for anything from the server. Few (if any) have 
adapted by actually parsing the greeting and making sure that they've seen 
the end of a multi-line greeting before talking.


That all sounds great.

Is there any way to use postscreen as a frontend filter for a sendmail 
MTA?



--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  From the Liberty perspective, it doesn't matter if it's a
  jackboot or a Birkenstock smashing your face. -- Robb Allen
---
 3 days until the 281st anniversary of John Peter Zenger's acquittal

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 02.08.2016 um 18:55 schrieb Bill Cole:

Combined, this is why Sendmail and other MTA greeting delays are less
spectacularly effective than they used to be and less effective than
postscreen. The resource cost of prolonging every session to 6s is
untenable for busy machines, so bots that have adapted can get through.
Back in the early days of Sendmail's GreetPause a value of 3s would
catch most bots but over the years some bots have adapted by doing their
own hard delays and others have learned to wait for anything from the
server. Few (if any) have adapted by actually parsing the greeting and
making sure that they've seen the end of a multi-line greeting before
talking


in fact most bots have a timeout around 10 seconds
postscreen_greet_wait = ${stress?3}${stress:11}s

and you will see a massive drop down of rejected mails in mailgraph 
because they all hang up after 10 seconds and since this result in 
postscreen is cached a legit client must only pass it one time while the 
bot not passing the tests hang forever there


well, and that greet_wait is also a good thing for all the scored 
dnsbl/dnswl because if there are some slow repsonding they don't get 
skipped and so the bot sits there useless waiting 11 seconds to get a 
"blocked using highest scored DNSBl"


postscreen other than smtpd is a single process, dnsbl/dnswl requests 
are done by extra, shared processes and so postscreen is unbeatable for 
years now if it comes to a inbound MX


after that you have 200-800 rejected mails in your contentfilters and 
smtpd-restricitions compared ot many thousands without




signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Bill Cole]

On 1 Aug 2016, at 15:53, Axb wrote:


On 01.08.2016 21:30, Vincent Fox wrote:
I keep seeing people say "well if you have postscreen, greylisting is 
just dumb".


I think that's a bit too strong. Robust greylisting that accommodates 
the reality of mail systems that share one spool across many outbound 
machines is not "dumb" but it is substantially less useful if you have 
any spambot protection before it. In the case of the original question, 
the context was specific to Postfix, where postscreen is a highly 
effective optional tool operating well ahead of any full greylisting 
implementation and taking out most of the bots that greylisting would 
with less risk and resource cost.



Well what is the equivalent for other MTA?


There is no complete equivalent for any other MTA that I am aware of.

Postscreen has a unique (as far as I know) greeting delay implementation 
(see below) as well as scored DNSBL checking, which makes it more 
feasible to use some hair-trigger DNSBLs safely than it is if you use 
them as hard tests (by requiring multiple hits on iffy lists.) There are 
also optional after-greeting behavioral checks which amount to 
simplistic zero-minimum-time greylisting because a "passing" client gets 
a 4xx from postscreen itself at RCPT, after which it must retry within 
the cache lifetime (30d by default) to get normal handling. Most sites 
don't enable the after-greeting tests because it has that greylist-like 
effect without the protections of a good greylist implementation.



google for "Greet pause" and "Early talker"

afaik there's implementations for Sendmail


Which had GreetPause well before postscreen or even the weaker 
pre-postscreen 'sleep' trick that predated postscreen.


and Haraka. There may be something similar for EXIM , QMAIL may have 
some obscure patch..


CGP also implements a greeting delay.

What's special about the postscreen delay is:

1. It delays only the last line of a multi-line greeting, so it catches 
MANY more bots than a simple delay.


2. It caches PASS results so even the very short (6s by default) delay 
that it imposes only hits the first encounter with a client that 
connects frequently. This is critically important in high-volume 
situations where the difference between mean session lengths of 0.5s and 
6s is the difference between 2 and 12 MX boxes in a cluster.


Combined, this is why Sendmail and other MTA greeting delays are less 
spectacularly effective than they used to be and less effective than 
postscreen. The resource cost of prolonging every session to 6s is 
untenable for busy machines, so bots that have adapted can get through. 
Back in the early days of Sendmail's GreetPause a value of 3s would 
catch most bots but over the years some bots have adapted by doing their 
own hard delays and others have learned to wait for anything from the 
server. Few (if any) have adapted by actually parsing the greeting and 
making sure that they've seen the end of a multi-line greeting before 
talking.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[@lbutlr]

On 01 Aug 2016, at 11:02, Matus UHLAR - fantomas  wrote:
>> On 31 Jul 2016, at 22:12, Benny Pedersen  wrote:
>>> i bet greylist is cough invalid mailservers at the doorstep, it could be 
>>> that postscreen is bad aswell ?
> 
> On 01.08.16 07:46, @lbutlr wrote:
>> Sure, if by “invalid” you mean Amazon, most banks, several airlines, large
>> mail services, and many many others.
>> 
>> Nearly any company with multiple mail servers will send mail from any of
>> their servers, and may retry from a different server than the initial
>> attempt, thus resetting the greylist.
> 
> while we're at it, I really don't understand why they do it like this.
> what's the point behind changing IP address after each delivery attempt?

It’s not necessarily intentional.

I have 100 mail servers. I send a mail to someone. It goes to one of the 100 
mail servers to get sent out. It has a temp fail. I queue the mail up send 
again in 15 minutes. It goes to one of the 100 mail server to get sent out.

Lather, rinse, repeat.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 02.08.2016 um 00:05 schrieb Benny Pedersen:

On 2016-08-01 19:02, Matus UHLAR - fantomas wrote:


while we're at it, I really don't understand why they do it like this.
what's the point behind changing IP address after each delivery attempt?


goal is to expose more networks ips to be blocked at the recipient
server for abuse, ironical :)


fascinating how a single human can talk that much nonsense from the 
viewpoint of his micro-world of a simple inbound server for himself and 
his brother


go ahadea and block the wolrd excpect your worlds whitelist

that will stop to scale frm the moment you are responsible for others 
mails where the RCPT don't care about yoor narrow-sighted viewpoint and 
hire some other mailadmin which touch of the reality




signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Benny Pedersen]

On 2016-08-01 19:18, Larry Rosenman wrote:

Shared outbound spool, and the next available host sends it.


next host start a new greylist time frame to delay again


It's not nefarious, just load balancing.


yes misunderstanding what not to do

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Benny Pedersen]

On 2016-08-01 19:02, Matus UHLAR - fantomas wrote:


while we're at it, I really don't understand why they do it like this.
what's the point behind changing IP address after each delivery 
attempt?


goal is to expose more networks ips to be blocked at the recipient 
server for abuse, ironical :)

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 01.08.2016 um 23:36 schrieb sha...@shanew.net:

Others could probably add to that list, but that's just off the top of
my head.  But, even if a spam source retries and successfully makes it
past the greylisting, the greylisting still provides potential
benefits, like:

- While it was waiting to retry, its IP has been added to BLs, which
  my other filters will score appropriately
- While it was waiting to retry, the phishing URL in it has been
  reported and taken down (or the URL shortener link it used has been
  removed)
- While it was waiting to retry, the virus it carries has been
  identified and pushed out to my virus definitions
- While it was waiting to retry, its registered domain has been
  removed
- While it was waiting to retry, others who received the spam have
  reported it to services like Razor and DCC, which other filters will
  act on


excatly that's the point
whe had last month 1212 greylistings
a majority was blacklisted later, bet it RBL or URIBL

well, 1212 is not much of the overall mailflow, the reason is just 
"knowing what you are doing" and have greylisting only as last resort 
before contentfllters and skip it if the sender matches SPF or is on any 
known DSNWL


finally that means zero bad impact for regular mail
if only *one* phishing mail which would have trapped a user was rejected 
with *zero* costs on a wise setup it has done it's job


again: the point is not to delay everybody, it's about to delay pure 
junk senders wihouth touching anything which has a single reputation 
sign and that goal won't change at any point of time - just becasue of 
the zero-cost and zero-impact for any regular mailflow




signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[shanew]

On Sun, 31 Jul 2016, Robert Schetterer wrote:


Greylisting was invented as an idea against bots. Its based on the idea
that bots "fire and forget" when they see a tmp error and dont get back.

But thats historic, bots are recoded, better antibot tecs were invented.
The only problem now is people still believe in historic stuff.


This argument ignores two important facts.

First, even if 98% of bots and viruses (and that number is pure
conjecture on my part) are now smart enough to retry, that doesn't
change that greylisting is a just about the lowest "cost" way of
preventing the ones that aren't smart enough (or aren't designed to
retry because they want to push the most amount of junk at the
lowest-hanging fruit).

Second, the ability of a bot, virus, server or any other spam source
to retry delivery after a temp failure is not the only "weakness"
greylisting takes advantage of.  A spam source might not get past my
greylist for any number of reasons, including the classic case of poor
coding/design, but also:

- It is detected and blocked (or taken offline) by the source network
  before its greylist period is up
- It make use of a compromised account, and that account is disabled
  or secured before its greylist period is up
- It is part of a distributed botnet, so subsequent attempts come from
  a different IP/network
- It sends a high volume of spam, so it doesn't come back around to
  retry again until after its entry has been removed, requiring
  a whole new greylisting period

Others could probably add to that list, but that's just off the top of
my head.  But, even if a spam source retries and successfully makes it
past the greylisting, the greylisting still provides potential
benefits, like:

- While it was waiting to retry, its IP has been added to BLs, which
  my other filters will score appropriately
- While it was waiting to retry, the phishing URL in it has been
  reported and taken down (or the URL shortener link it used has been
  removed)
- While it was waiting to retry, the virus it carries has been
  identified and pushed out to my virus definitions
- While it was waiting to retry, its registered domain has been
  removed
- While it was waiting to retry, others who received the spam have
  reported it to services like Razor and DCC, which other filters will
  act on
- If it has to keep retrying addresses to my server, I'm consuming
  resources (however minimally) that could be used to send their junk
  to others

Again, I'm sure others could add more based on their experiences.

I'm not saying greylisting is without problems, that it just works out
of the box (initial and ongoing configuration is critical), or that
everyone should be using it, but there's a lot more going on here than
just outwitting poorly written bots.

--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/|  System Admin - UT CompSci
=--+---
All syllogisms contain three lines |  sha...@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Vincent Fox]

I have greet_pause long ago enabled in sendmail.


Out of 1,112,871 messages yesterday only 1096 tripped greet_pause.

If it occasionally trips up a miscoded client I tell them to fix it or stop 
using

and am happy that this tiny effort made the internet a slightly saner place.

But it's not in the same league at all.


Between milter-greylist, and several measures, it covers quite a bit

at the first layer, before we have to get to the expensive SpamAssassin stuff.


The hate on greylisting seems a bit off to me.  I realize it's not the most

effective tool in my toolbox, but it helps.  There have been this year malware

blast campaigns, where I traced that the IP involved were blocked by the

MX servers with greylisting, and examples that got through came through

department MTA that did not.


On the whole I'd rather have postscreen, but I don't make those choices.

Thus we patch together a simulacrum.



From: Axb <axb.li...@gmail.com>
Sent: Monday, August 1, 2016 12:53:27 PM
To: users@spamassassin.apache.org
Subject: Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - 
not scanning after hold)

On 01.08.2016 21:30, Vincent Fox wrote:
> I keep seeing people say "well if you have postscreen, greylisting is just 
> dumb".
>
> Well what is the equivalent for other MTA?

google for "Greet pause" and "Early talker"

afaik there's implementations for Sendmail and Haraka. There may be
something similar for EXIM , QMAIL may have some obscure patch..

Axb

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Axb]

On 01.08.2016 21:30, Vincent Fox wrote:

I keep seeing people say "well if you have postscreen, greylisting is just 
dumb".

Well what is the equivalent for other MTA?


google for "Greet pause" and "Early talker"

afaik there's implementations for Sendmail and Haraka. There may be 
something similar for EXIM , QMAIL may have some obscure patch..


Axb

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Vincent Fox]

I keep seeing people say "well if you have postscreen, greylisting is just 
dumb".

Well what is the equivalent for other MTA?


I still see a lot of spambots on PBL hosts, that never contact again.  So the

blanket statement "bots are recoded" just doesn't jibe with what I see.

Maybe you could many bots, or newer bots, but not all of them in current

usage recognize the 4xx, wait and retry.



From: @lbutlr <krem...@kreme.com>
Sent: Sunday, July 31, 2016 8:55 PM
To: users@spamassassin.apache.org
Subject: Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - 
not scanning after hold)

On 31 Jul 2016, at 01:06, Robert Schetterer <r...@sys4.de> wrote:
> But thats historic, bots are recoded, better antibot tecs were invented.
> The only problem now is people still believe in historic stuff.

Yeah, that about sums it up. Greylisting never worked well, always caused 
problems with lost email, and in 2016 is simply a bad idea. Not just a not good 
idea, but a bad idea.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 01.08.2016 um 19:02 schrieb Matus UHLAR - fantomas:

On 31 Jul 2016, at 22:12, Benny Pedersen  wrote:

i bet greylist is cough invalid mailservers at the doorstep, it could
be that postscreen is bad aswell ?


On 01.08.16 07:46, @lbutlr wrote:

Sure, if by “invalid” you mean Amazon, most banks, several airlines,
large
mail services, and many many others.

Nearly any company with multiple mail servers will send mail from any of
their servers, and may retry from a different server than the initial
attempt, thus resetting the greylist.


while we're at it, I really don't understand why they do it like this.
what's the point behind changing IP address after each delivery attempt?


load balancing of the outgoing mailqueue
these are not single instance servers



signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Larry Rosenman]

On 2016-08-01 12:02, Matus UHLAR - fantomas wrote:

On 31 Jul 2016, at 22:12, Benny Pedersen  wrote:
i bet greylist is cough invalid mailservers at the doorstep, it could 
be that postscreen is bad aswell ?


On 01.08.16 07:46, @lbutlr wrote:
Sure, if by “invalid” you mean Amazon, most banks, several airlines, 
large

mail services, and many many others.

Nearly any company with multiple mail servers will send mail from any 
of

their servers, and may retry from a different server than the initial
attempt, thus resetting the greylist.


while we're at it, I really don't understand why they do it like this.
what's the point behind changing IP address after each delivery 
attempt?

Shared outbound spool, and the next available host sends it.

It's not nefarious, just load balancing.

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@lerctr.org
US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Matus UHLAR - fantomas]

On 31 Jul 2016, at 22:12, Benny Pedersen  wrote:

i bet greylist is cough invalid mailservers at the doorstep, it could be that 
postscreen is bad aswell ?


On 01.08.16 07:46, @lbutlr wrote:

Sure, if by “invalid” you mean Amazon, most banks, several airlines, large
mail services, and many many others.

Nearly any company with multiple mail servers will send mail from any of
their servers, and may retry from a different server than the initial
attempt, thus resetting the greylist.


while we're at it, I really don't understand why they do it like this.
what's the point behind changing IP address after each delivery attempt?

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
A day without sunshine is like, night.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Ryan Coleman]

> On Aug 1, 2016, at 10:15 AM, Benny Pedersen  wrote:
> 
>>> 
>>> i bet greylist is cough invalid mailservers at the doorstep, it could be 
>>> that postscreen is bad aswell ?
>> Sure, if by “invalid” you mean Amazon, most banks, several airlines,
>> large mail services, and many many others.
> 
> basicly badly marketing
> 
>> Nearly any company with multiple mail servers will send mail from any
>> of their servers, and may retry from a different server than the
>> initial attempt, thus resetting the greylist.
> 
> marketing trys to fuck how smtp works have always being a fail with greylist


What on earth are you trying to say here? Because after my 20 years in the 
industry I haven’t a clue what you’re saying.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Benny Pedersen]

On 2016-08-01 15:46, @lbutlr wrote:


Where did you get the idea that postfix will not deliver later?


i did not say that

i bet greylist is cough invalid mailservers at the doorstep, it could 
be that postscreen is bad aswell ?


Sure, if by “invalid” you mean Amazon, most banks, several airlines,
large mail services, and many many others.


basicly badly marketing


Nearly any company with multiple mail servers will send mail from any
of their servers, and may retry from a different server than the
initial attempt, thus resetting the greylist.


marketing trys to fuck how smtp works have always being a fail with 
greylist



There is a reason that greylist software comes with default
exclusions, because greylisting is known to cause missed email if used
as designed.


postscreen dont care at all

take that

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[@lbutlr]

On 31 Jul 2016, at 22:12, Benny Pedersen <m...@junc.eu> wrote:
> On 2016-08-01 05:55, @lbutlr wrote:
>> On 31 Jul 2016, at 01:06, Robert Schetterer <r...@sys4.de> wrote:
>>> But thats historic, bots are recoded, better antibot tecs were invented.
>>> The only problem now is people still believe in historic stuff.
>> Yeah, that about sums it up. Greylisting never worked well, always
>> caused problems with lost email, and in 2016 is simply a bad idea. Not
>> just a not good idea, but a bad idea.
> 
> back to basic then, why would a mta like postfix not deliver later when it 
> get a tempfail ?

Where did you get the idea that postfix will not deliver later?

> i bet greylist is cough invalid mailservers at the doorstep, it could be that 
> postscreen is bad aswell ?

Sure, if by “invalid” you mean Amazon, most banks, several airlines, large mail 
services, and many many others.

Nearly any company with multiple mail servers will send mail from any of their 
servers, and may retry from a different server than the initial attempt, thus 
resetting the greylist.

There is a reason that greylist software comes with default exclusions, because 
greylisting is known to cause missed email if used as designed.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Benny Pedersen]

On 2016-08-01 05:55, @lbutlr wrote:

On 31 Jul 2016, at 01:06, Robert Schetterer <r...@sys4.de> wrote:
But thats historic, bots are recoded, better antibot tecs were 
invented.

The only problem now is people still believe in historic stuff.


Yeah, that about sums it up. Greylisting never worked well, always
caused problems with lost email, and in 2016 is simply a bad idea. Not
just a not good idea, but a bad idea.


back to basic then, why would a mta like postfix not deliver later when 
it get a tempfail ?


i bet greylist is cough invalid mailservers at the doorstep, it could be 
that postscreen is bad aswell ?


see in sqlgrey whitelist how many old ips that do not retry, shurg

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[@lbutlr]

On 31 Jul 2016, at 01:06, Robert Schetterer <r...@sys4.de> wrote:
> But thats historic, bots are recoded, better antibot tecs were invented.
> The only problem now is people still believe in historic stuff.

Yeah, that about sums it up. Greylisting never worked well, always caused 
problems with lost email, and in 2016 is simply a bad idea. Not just a not good 
idea, but a bad idea.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Robert Schetterer]

Am 30.07.2016 um 13:10 schrieb Kim Roar Foldøy Hauge:
> On Sat, 30 Jul 2016, Robert Schetterer wrote:
> 
>> Am 30.07.2016 um 03:34 schrieb Reindl Harald:
>>>
>>>
>>> Am 29.07.2016 um 22:48 schrieb Dianne Skoll:
>>>> On Fri, 29 Jul 2016 22:39:15 +0200
>>>> Robert Schetterer <r...@sys4.de> wrote:
>>>>
>>>>>> I don't use postfix or postscreen.
>>>>> hm.. that does not fit the subject..why did you involved yourself ?
>>>>
>>>> I am sorry.  I should have changed the thread subject.
>>>>
>>>>> you may get that quite better, i see
>>>>> a lot of server greylisting useless ,only filling up others queues
>>>>> waiting for a second slot ,so it may only cheap for you but not for
>>>>> your partners
>>>>> Dont slow down communication if you dont need to
>>>>
>>>> So what I didn't mention is that in our implementation, once an IP
>>>> address successully passes greylisting, we no longer greylist it for
>>>> the next 45 days.  (It would probably be pointless... if an IP passes
>>>> greylisting once, it probably will keep passing it.)
>>>
>>> that's nothing special and postgrey does the same, the whole point of
>>> greylisting is that badly written bots don't try again (the same happens
>>> if they connect to a backup-MX responding with 4xx)
>>>
>>> also it don't help for clients which *do not* pass like large senders
>>> with outbound clusters coming each time from a different IP
>>>
>>> hence you skip greylisting based on DNSWL and spf-policyd because that
>>> big legit senders hit DNSWL or have a proper SPF while random bots of
>>> infected machines don't and this ones are your target for greylisting
>>>
>>>
>>>
>>
>> Harald is right, the goal has to be "reject" spam asap, not to tell
>> "come again later", i.e i had 4 bot cons per second, this will run out
>> the system of smtp slots rapidly which means any good sender isnt able
>> to sent mail too, greylisting makes such situations more worst.
>>
> 
> I'm no expert here, but postgrey is usually a purely local test. It
> should terminate with a "currently busy, try again later" message very
> quickly. SPF checks and white listing require dns lookups that can
> potentially take much longer. Several orders of magnitude longer.
> 
> Efficient handling of spam is all about doing the least expensive tests
> first in terms of cpu/time. Caching DNS can probably help a bit, but it
> will still require the occasional lookup now and then that take a lot
> longer than a good greylisting implementation should ever do.
> 
> Doing an expensive test on every mail when it's not needed is badly
> designed setup.
> 
> Many of the dns based lists also limit the amount of checks per day.
> Worst case scenario, you stop getting results from lists due to over
> use. If you use google's 8.8.8.8 servers for dns lookups one can quickly
> run into that problem, I did. A high volume of dns checks could force
> you into having to pay for the amount of traffic you cause.
> 
> Many expensive network (takes a long time) checks will probably make you
> run out of slots a lot faster than the reconnects due to greylisting
> will do due to the time spent waiting for the lookups to finish.
> 
> If speed of delivery is important, you could lower the amount of time
> mail stays greylisted. Ideally you'd like the mail delivered the first
> time a server tries to send it again. If a server tries to resend once,
> it will most likely try more than once anyway. Having a minimum time of
> 300 seconds, the default of postgrey, is probably a bit excessive.
> 

Greylisting was invented as an idea against bots. Its based on the idea
that bots "fire and forget" when they see a tmp error and dont get back.

This idea was criticized for design failures since it exists ( Harald
and me explained it in detail ), but was acceptable in lack of better
ideas that time.

But thats historic, bots are recoded, better antibot tecs were invented.
The only problem now is people still believe in historic stuff.


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 30.07.2016 um 23:10 schrieb Bill Cole:

On 30 Jul 2016, at 7:10, Kim Roar Foldøy Hauge wrote:


I'm no expert here, but postgrey is usually a purely local test. It
should terminate with a "currently busy, try again later" message very
quickly.


Unless your database is very large, yes.


SPF checks and white listing require dns lookups that can potentially
take much longer. Several orders of magnitude longer.


Occasionally, yes, no matter how you do DNS. However, Postfix smtpd will
do multiple DNS lookups that have a strong chance of being slow before
getting to any policy daemon like Postgrey. Alternatively, postscreen is
a much simpler process than smtpd and in its usual config does nearly no
input processing while doing DNSBL lookups in parallel, time-limited to
10s. This is usually much more efficient for dealing with spambots than
going through everything that smtpd does before calling any policy
daemon. Most DNSBLs worth using have robust authoritative DNS, so
(unlike SPF or IP->Hostname->IP checking, both of which can require many
queries) it is rarely slow to get DNSBL results


and it don't eat any smtpd process, responses shoudl be cached by a 
local, rescursing resolver anyways and finally postfix-smtpd/postrey 
have only to deal with a very low volume and the very expensive content 
filter sees mostly ham


*if* a message makes it through postscreen and up to greylisting the 
DNSWL and SPF results are cached anyways and re-used by spamasssassin 
which can even re-use the spf header of the python policyd


our ould MX had peaks with 3000 MHz on the ESXi cluster, the current one 
after switch to postscreen and configure things proper including 
shortcircuit is running most of the time with 60-250 Mhz by a amount of 
15000 destination addresses spammers try to deliver crap




signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Bill Cole]

On 30 Jul 2016, at 7:10, Kim Roar Foldøy Hauge wrote:

I'm no expert here, but postgrey is usually a purely local test. It 
should terminate with a "currently busy, try again later" message very 
quickly.


Unless your database is very large, yes.

SPF checks and white listing require dns lookups that can potentially 
take much longer. Several orders of magnitude longer.


Occasionally, yes, no matter how you do DNS. However, Postfix smtpd will 
do multiple DNS lookups that have a strong chance of being slow before 
getting to any policy daemon like Postgrey. Alternatively, postscreen is 
a much simpler process than smtpd and in its usual config does nearly no 
input processing while doing DNSBL lookups in parallel, time-limited to 
10s. This is usually much more efficient for dealing with spambots than 
going through everything that smtpd does before calling any policy 
daemon. Most DNSBLs worth using have robust authoritative DNS, so 
(unlike SPF or IP->Hostname->IP checking, both of which can require many 
queries) it is rarely slow to get DNSBL results.


Efficient handling of spam is all about doing the least expensive 
tests first in terms of cpu/time. Caching DNS can probably help a bit, 
but it will still require the occasional lookup now and then that take 
a lot longer than a good greylisting implementation should ever do.


Actually, having a local (same machine or same LAN) caching DNS resolver 
is extremely helpful relative to using a resolver on the other side of a 
router (i.e. an ISP's resolver) or worse, somewhere on the other side of 
2-20 routers (i.e. OpenDNS, Google Public DNS, etc.) A lookup from a 
local cache typically takes <10ms, while a lookup from a public DNS 
server routinely will have 30-100ms of network latency built in and even 
one's connectivity provider's DNS will probably have at least 10ms. In 
any case, a DNS lookup that isn't in a resolver's cache is likely to 
take on the order of 100ms (but potentially multiple seconds to timeout 
on lookups that lead nowhere, indirectly) on top of the network latency. 
This leads to another reason to have a local resolver dedicated to mail 
servers: you can tune reply timeouts to assure that you never suffer 
those really long waits.


Doing an expensive test on every mail when it's not needed is badly 
designed setup.


Which is not really a strong argument for Postgrey or any other 
greylisting tool that works as as a Postfix policy daemon. The only way 
to spare a Postfix server from the potentially very slow client hostname 
DNS queries is to have postscreen in front of smtpd, at least to do 
pre-greet enforcement.


Many of the dns based lists also limit the amount of checks per day. 
Worst case scenario, you stop getting results from lists due to over 
use.


Yes, but those query limits are designed to make sure that commercial 
entities that get value from those DNSBLs and are large enough to afford 
participation in keeping them in operation pay for that value they 
receive.


If you use google's 8.8.8.8 servers for dns lookups one can quickly 
run into that problem, I did. A high volume of dns checks could force 
you into having to pay for the amount of traffic you cause.


Using Google's public DNS or anything like it assures that a mail system 
is low-performing and unreliable. There's nothing wrong with running an 
amateur mail system but one should understand that using free distant 
DNS run by unaccountable entities relegates a mail system to "hobbyist" 
class.


Many expensive network (takes a long time) checks will probably make 
you run out of slots a lot faster than the reconnects due to 
greylisting will do due to the time spent waiting for the lookups to 
finish.


I can only imagine that being a problem, and only on a very busy and 
badly misconfigured mail server. I've run systems using multiple 
machines each handling over a million SMTP connections per day and never 
had a substantial problem with doing multiple DNSBL lookups on every 
connection that passes a short greeting delay and SPF checks on 
everything that gets to the point of body filtering (i.e. SPF within 
SpamAssassin.)


If speed of delivery is important, you could lower the amount of time 
mail stays greylisted. Ideally you'd like the mail delivered the first 
time a server tries to send it again. If a server tries to resend 
once, it will most likely try more than once anyway. Having a minimum 
time of 300 seconds, the default of postgrey, is probably a bit 
excessive.


Reducing that is unlikely to reduce delays, because a lot of systems use 
300s as a minimum delay between retries; for a long time the standard 
queue run time for Sendmail was 300s and that has been the default for 
Postfix since v2.4. The main exceptions using shorter retry times would 
be high-volume mailing lists systems (legit or otherwise) that cannot 
afford to let their deferral queues grow large.

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 30.07.2016 um 13:10 schrieb Kim Roar Foldøy Hauge:

On Sat, 30 Jul 2016, Robert Schetterer wrote:


Am 30.07.2016 um 03:34 schrieb Reindl Harald:



Am 29.07.2016 um 22:48 schrieb Dianne Skoll:

On Fri, 29 Jul 2016 22:39:15 +0200
Robert Schetterer <r...@sys4.de> wrote:


I don't use postfix or postscreen.

hm.. that does not fit the subject..why did you involved yourself ?


I am sorry.  I should have changed the thread subject.


you may get that quite better, i see
a lot of server greylisting useless ,only filling up others queues
waiting for a second slot ,so it may only cheap for you but not for
your partners
Dont slow down communication if you dont need to


So what I didn't mention is that in our implementation, once an IP
address successully passes greylisting, we no longer greylist it for
the next 45 days.  (It would probably be pointless... if an IP passes
greylisting once, it probably will keep passing it.)


that's nothing special and postgrey does the same, the whole point of
greylisting is that badly written bots don't try again (the same happens
if they connect to a backup-MX responding with 4xx)

also it don't help for clients which *do not* pass like large senders
with outbound clusters coming each time from a different IP

hence you skip greylisting based on DNSWL and spf-policyd because that
big legit senders hit DNSWL or have a proper SPF while random bots of
infected machines don't and this ones are your target for greylisting



Harald is right, the goal has to be "reject" spam asap, not to tell
"come again later", i.e i had 4 bot cons per second, this will run out
the system of smtp slots rapidly which means any good sender isnt able
to sent mail too, greylisting makes such situations more worst.


I'm no expert here, but postgrey is usually a purely local test. It
should terminate with a "currently busy, try again later" message very
quickly


yes, but when the total amount reaches your maximum of smtpd processes 
because 4 bots per second there are no longer slots für legit clients 
and if you then greylist a large amount fo legit clients which are all 
coming back (in case of high legit traffic) things get much worser


in times of postscreen (and "Using Postfix and Postgrey" with current 
software implies that it is available) that all is not mucha problem 
because most crap don't make it to smtpd


well, and finally limit the impact by just use iptables on the server

ctstate NEW recent: UPDATE seconds: 2 hit_count: 5 name: DEFAULT side: 
source mask: 255.255.255.255




signature.asc
Description: OpenPGP digital signature

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Kim Roar Foldøy Hauge]

On Sat, 30 Jul 2016, Robert Schetterer wrote:


Am 30.07.2016 um 03:34 schrieb Reindl Harald:



Am 29.07.2016 um 22:48 schrieb Dianne Skoll:

On Fri, 29 Jul 2016 22:39:15 +0200
Robert Schetterer <r...@sys4.de> wrote:


I don't use postfix or postscreen.

hm.. that does not fit the subject..why did you involved yourself ?


I am sorry.  I should have changed the thread subject.


you may get that quite better, i see
a lot of server greylisting useless ,only filling up others queues
waiting for a second slot ,so it may only cheap for you but not for
your partners
Dont slow down communication if you dont need to


So what I didn't mention is that in our implementation, once an IP
address successully passes greylisting, we no longer greylist it for
the next 45 days.  (It would probably be pointless... if an IP passes
greylisting once, it probably will keep passing it.)


that's nothing special and postgrey does the same, the whole point of
greylisting is that badly written bots don't try again (the same happens
if they connect to a backup-MX responding with 4xx)

also it don't help for clients which *do not* pass like large senders
with outbound clusters coming each time from a different IP

hence you skip greylisting based on DNSWL and spf-policyd because that
big legit senders hit DNSWL or have a proper SPF while random bots of
infected machines don't and this ones are your target for greylisting





Harald is right, the goal has to be "reject" spam asap, not to tell
"come again later", i.e i had 4 bot cons per second, this will run out
the system of smtp slots rapidly which means any good sender isnt able
to sent mail too, greylisting makes such situations more worst.



I'm no expert here, but postgrey is usually a purely local test. It should 
terminate with a "currently busy, try again later" message very quickly. 
SPF checks and white listing require dns lookups that can potentially take 
much longer. Several orders of magnitude longer.


Efficient handling of spam is all about doing the least expensive tests 
first in terms of cpu/time. Caching DNS can probably help a bit, but it 
will still require the occasional lookup now and then that take a lot 
longer than a good greylisting implementation should ever do.


Doing an expensive test on every mail when it's not needed is badly 
designed setup.


Many of the dns based lists also limit the amount of checks per day. Worst 
case scenario, you stop getting results from lists due to over use. If you 
use google's 8.8.8.8 servers for dns lookups one can quickly run into that 
problem, I did. A high volume of dns checks could force you into having to 
pay for the amount of traffic you cause.


Many expensive network (takes a long time) checks will probably make you 
run out of slots a lot faster than the reconnects due to greylisting will 
do due to the time spent waiting for the lookups to finish.


If speed of delivery is important, you could lower the amount of time mail 
stays greylisted. Ideally you'd like the mail delivered the first time 
a server tries to send it again. If a server tries to resend once, it will 
most likely try more than once anyway. Having a minimum time of 300 
seconds, the default of postgrey, is probably a bit excessive.


--
Kim Roar Foldøy Hauge
Event:Presse - The Gathering 2016
webmas...@samfunnet.no
Root@HC,HX,JH,LZ,OT,P,VH

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Robert Schetterer]

Am 30.07.2016 um 03:34 schrieb Reindl Harald:
> 
> 
> Am 29.07.2016 um 22:48 schrieb Dianne Skoll:
>> On Fri, 29 Jul 2016 22:39:15 +0200
>> Robert Schetterer <r...@sys4.de> wrote:
>>
>>>> I don't use postfix or postscreen.
>>> hm.. that does not fit the subject..why did you involved yourself ?
>>
>> I am sorry.  I should have changed the thread subject.
>>
>>> you may get that quite better, i see
>>> a lot of server greylisting useless ,only filling up others queues
>>> waiting for a second slot ,so it may only cheap for you but not for
>>> your partners
>>> Dont slow down communication if you dont need to
>>
>> So what I didn't mention is that in our implementation, once an IP
>> address successully passes greylisting, we no longer greylist it for
>> the next 45 days.  (It would probably be pointless... if an IP passes
>> greylisting once, it probably will keep passing it.)
> 
> that's nothing special and postgrey does the same, the whole point of
> greylisting is that badly written bots don't try again (the same happens
> if they connect to a backup-MX responding with 4xx)
> 
> also it don't help for clients which *do not* pass like large senders
> with outbound clusters coming each time from a different IP
> 
> hence you skip greylisting based on DNSWL and spf-policyd because that
> big legit senders hit DNSWL or have a proper SPF while random bots of
> infected machines don't and this ones are your target for greylisting
> 
> 
> 

Harald is right, the goal has to be "reject" spam asap, not to tell
"come again later", i.e i had 4 bot cons per second, this will run out
the system of smtp slots rapidly which means any good sender isnt able
to sent mail too, greylisting makes such situations more worst.





Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Reindl Harald]

Am 29.07.2016 um 22:48 schrieb Dianne Skoll:

On Fri, 29 Jul 2016 22:39:15 +0200
Robert Schetterer <r...@sys4.de> wrote:


I don't use postfix or postscreen.

hm.. that does not fit the subject..why did you involved yourself ?


I am sorry.  I should have changed the thread subject.


you may get that quite better, i see
a lot of server greylisting useless ,only filling up others queues
waiting for a second slot ,so it may only cheap for you but not for
your partners
Dont slow down communication if you dont need to


So what I didn't mention is that in our implementation, once an IP
address successully passes greylisting, we no longer greylist it for
the next 45 days.  (It would probably be pointless... if an IP passes
greylisting once, it probably will keep passing it.)


that's nothing special and postgrey does the same, the whole point of 
greylisting is that badly written bots don't try again (the same happens 
if they connect to a backup-MX responding with 4xx)


also it don't help for clients which *do not* pass like large senders 
with outbound clusters coming each time from a different IP


hence you skip greylisting based on DNSWL and spf-policyd because that 
big legit senders hit DNSWL or have a proper SPF while random bots of 
infected machines don't and this ones are your target for greylisting






signature.asc
Description: OpenPGP digital signature

Is greylisting effective? (was Re: Using Postfix and Postgrey - not scanning after hold)

[Dianne Skoll]

On Fri, 29 Jul 2016 22:39:15 +0200
Robert Schetterer <r...@sys4.de> wrote:

> > I don't use postfix or postscreen.  
> hm.. that does not fit the subject..why did you involved yourself ?

I am sorry.  I should have changed the thread subject.

> you may get that quite better, i see
> a lot of server greylisting useless ,only filling up others queues
> waiting for a second slot ,so it may only cheap for you but not for
> your partners
> Dont slow down communication if you dont need to

So what I didn't mention is that in our implementation, once an IP
address successully passes greylisting, we no longer greylist it for
the next 45 days.  (It would probably be pointless... if an IP passes
greylisting once, it probably will keep passing it.)

The impact on legitimate mail senders is therefore very minimal.

> its up to you using more up2date tec stuff, be sure postscreen
> is used in equal setups then yours and is well tested

I don't quite understand that last sentence...

Regards,

Dianne.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Gary Funck]

On 11/29/12 14:46:25, David F. Skoll wrote:
 We greylist after the end of DATA.  This wastes bandwidth, but lets us
 use the Subject: line as an additional mix in the greylisting tuple.
 This catches ratware that retries in the face of greylisting, but
 mutates the subject line with each retry.

We use grey listing on our low volume server, and as others have
noted, it works well because a high percentage of spam bots do
not bother to retry.  But as others have mentioned, it can be
painful waiting for the delayed confirmation on a registration to a web
site to come in an hour/two later, or email from a new client
who is waiting on a response.

Since this is a Spam Assassin list: Is there a way of disabling
grey listing, but still receiving some benefit from the principle
that mail received from a first time or infrequent sender should
be looked upon with some suspicion?

Assume that either some to-be-implemented SA filter, or some
mail gateway front-end (like MIMEDefang), adds a new tag/two,
for example: SENDER_FIRST_RCPT, SENDER_LOW_FREQ,
SENDER_HI_FREQ, or SENDER_HI_AVE_SA_SCORE? All these tags
might be based upon some look back period (say: 90 days).

Theoretically, these new tags could be calculated after the fact
when passing through a spam corpus.  And since many/most grey
listing systems differentiate by some form of (sender, recipient)
pairing this analysis can be reliably/repeatably performed by an
SA plug-in at the point of delivery to the user, if needed.

It would need to be shown that these new tags improve
the ability to discriminate spam from ham.  If the scheme
worked well, there might be no need for grey listing at all.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Matt]

 We greylist after the end of DATA.  This wastes bandwidth, but lets us
 use the Subject: line as an additional mix in the greylisting tuple.
 This catches ratware that retries in the face of greylisting, but
 mutates the subject line with each retry.

 We use grey listing on our low volume server, and as others have
 noted, it works well because a high percentage of spam bots do
 not bother to retry.  But as others have mentioned, it can be
 painful waiting for the delayed confirmation on a registration to a web
 site to come in an hour/two later, or email from a new client
 who is waiting on a response.

Using dnswl.org to whitelist against greylisting might help some.

 Since this is a Spam Assassin list: Is there a way of disabling
 grey listing, but still receiving some benefit from the principle
 that mail received from a first time or infrequent sender should
 be looked upon with some suspicion?

 Assume that either some to-be-implemented SA filter, or some
 mail gateway front-end (like MIMEDefang), adds a new tag/two,
 for example: SENDER_FIRST_RCPT, SENDER_LOW_FREQ,
 SENDER_HI_FREQ, or SENDER_HI_AVE_SA_SCORE? All these tags
 might be based upon some look back period (say: 90 days).

 Theoretically, these new tags could be calculated after the fact
 when passing through a spam corpus.  And since many/most grey
 listing systems differentiate by some form of (sender, recipient)
 pairing this analysis can be reliably/repeatably performed by an
 SA plug-in at the point of delivery to the user, if needed.

 It would need to be shown that these new tags improve
 the ability to discriminate spam from ham.  If the scheme
 worked well, there might be no need for grey listing at all.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Martin Gregorie]

On Mon, 2012-12-03 at 07:23 -0800, Gary Funck wrote:
 Since this is a Spam Assassin list: Is there a way of disabling
 grey listing, but still receiving some benefit from the principle
 that mail received from a first time or infrequent sender should
 be looked upon with some suspicion?
 
Yes. If you keep a list of the recipients of outgoing mail its easy to
whitelist any mail you receive from them. This approach does what you
want: a sender is treated as suspicious until you've sent mail to them
and recipient list maintenance is easy to automate.

I use a mail archive system as my recipients list because it has a
record of everybody I've sent mail to. I use an SA plugin to access the
archive. The combination of it and an associated rule will whitelist
anybody who is recorded in the archive as having received mail from me.

However, the database archives messages at 4-6 /sec, so this and/or the
storage requirements (4.3 GB to store 143,000 messages) may mean that,
if you're a high volume site and/or don't need an archive, you'd be
better off just keeping a list of the recipient(s) of outgoing messages.
 
I wrote my archive for personal use because I can find an old e-mail
with the archive search tool faster than I can by ferreting though a set
of mail folders: it was never designed as a high volume solution, but
should manage small business volumes quite easily with both it and SA
running on a typical desktop PC. Up to early this year I was using an
866 MHz P3 with 512MB RAM that easily kept up while PostgreSQL,the
archive, Postfix and SA. That is all now running on a 3GHz dual Athlon
with 4 GB RAM but not going any faster - an upgrade to Fedora 16 forced
the change because its installer wouldn't run in less than 1GB RAM.

If you think my SA plugin or the mail archive would be of use to you,
contact me off-list.


Martin

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[RW]

On Mon, 3 Dec 2012 07:23:59 -0800
Gary Funck wrote:

 Since this is a Spam Assassin list: Is there a way of disabling
 grey listing, but still receiving some benefit from the principle
 that mail received from a first time or infrequent sender should
 be looked upon with some suspicion?

Personally I wouldn't want to do it that way round - with a positive
score for unknown rather than a negative score for known. 

YMMV but almost all of the FPs I've had in the last ten years have been
that sort of mail because it's less likely to be recognised by Bayes.  

Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[David F. Skoll]

On Thu, 29 Nov 2012 14:36:45 -0500
vec...@vectro.org wrote:

 I've never had any
 complaints about delivery speed, but some senders have broken mail
 servers that don't retry on receiving a temporary failure.

Many such servers use broken SMTP implementations that can't handle
a 4xx code in response to RCPT properly.

We greylist after the end of DATA.  This wastes bandwidth, but lets us
use the Subject: line as an additional mix in the greylisting tuple.
This catches ratware that retries in the face of greylisting, but
mutates the subject line with each retry.

Also, once a given IP passes greylisting, we remember that and we don't
greylist that server for 40 days.  If you have a large-enough user population,
this can greatly mitigate the problems caused by initial greylisting delays.

Regards,

David.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Andrzej A. Filip]

On 11/29/2012 08:46 PM, David F. Skoll wrote:
 [...]
 Also, once a given IP passes greylisting, we remember that and we don't
 greylist that server for 40 days.  If you have a large-enough user population,
 this can greatly mitigate the problems caused by initial greylisting delays.
Do you treat yahoo like spam sources in the same way?

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Dave Warren]

On 11/29/2012 12:27, Andrzej A. Filip wrote:

On 11/29/2012 08:46 PM, David F. Skoll wrote:

[...]
Also, once a given IP passes greylisting, we remember that and we don't
greylist that server for 40 days.  If you have a large-enough user population,
this can greatly mitigate the problems caused by initial greylisting delays.

Do you treat yahoo like spam sources in the same way?


There's almost no point in greylisting an IP that you know will retry 
properly anyway, so why wouldn't you allow that IP to bypass greylisting 
in the future?


--
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Robert Schetterer]

Am 29.11.2012 20:46, schrieb David F. Skoll:
 On Thu, 29 Nov 2012 14:36:45 -0500
 vec...@vectro.org wrote:
 
 I've never had any
 complaints about delivery speed, but some senders have broken mail
 servers that don't retry on receiving a temporary failure.
 
 Many such servers use broken SMTP implementations that can't handle
 a 4xx code in response to RCPT properly.
 
 We greylist after the end of DATA.  This wastes bandwidth, but lets us
 use the Subject: line as an additional mix in the greylisting tuple.
 This catches ratware that retries in the face of greylisting, but
 mutates the subject line with each retry.
 
 Also, once a given IP passes greylisting, we remember that and we don't
 greylist that server for 40 days.  If you have a large-enough user population,
 this can greatly mitigate the problems caused by initial greylisting delays.
 
 Regards,
 
 David.
 

greylisting isnt state of art, however it might helpfull in some domains
( everyone has its own spam), using postscreen with postfix before
selective greylisting is a good choice

Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Andrzej A. Filip]

On 11/29/2012 09:31 PM, Dave Warren wrote:
 On 11/29/2012 12:27, Andrzej A. Filip wrote:
 On 11/29/2012 08:46 PM, David F. Skoll wrote:
 [...]
 Also, once a given IP passes greylisting, we remember that and we don't
 greylist that server for 40 days.  If you have a large-enough user
 population,
 this can greatly mitigate the problems caused by initial greylisting
 delays.
 Do you treat yahoo like spam sources in the same way?

 There's almost no point in greylisting an IP that you know will retry
 properly anyway, so why wouldn't you allow that IP to bypass
 greylisting in the future?

I assume that greylisting of yahoo like spam sources increases chances
of bulk detectors detecting spam. Is not it trues? [based on real data]

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[David F. Skoll]

On Thu, 29 Nov 2012 21:27:19 +0100
Andrzej A. Filip andrzej.fi...@gmail.com wrote:

 Do you treat yahoo like spam sources in the same way?

With respect to greylisting, of course.  If a machine passes greylisting once,
it's extremely likely to pass it in future and it's an utter waste of
time to greylist it.

Regards,

David.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Andrzej A. Filip]

On 11/29/2012 09:53 PM, David F. Skoll wrote:
 On Thu, 29 Nov 2012 21:27:19 +0100
 Andrzej A. Filip andrzej.fi...@gmail.com wrote:

 Do you treat yahoo like spam sources in the same way?
 With respect to greylisting, of course.  If a machine passes greylisting once,
 it's extremely likely to pass it in future and it's an utter waste of
 time to greylist it.
Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in
case of yahoo like spam sources?
[ based on your experience ]

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[David F. Skoll]

On Thu, 29 Nov 2012 21:59:45 +0100
Andrzej A. Filip andrzej.fi...@gmail.com wrote:

 Does greylisting increase chances of bulk detectors (razor/pyzor/dcc)
 in case of yahoo like spam sources?
 [ based on your experience ]

I suppose it might, but I don't use razor, pyzor, dcc or anything similar
so I have no personal experience.

Regards,

David.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Matt]

 I've never had any
 complaints about delivery speed, but some senders have broken mail
 servers that don't retry on receiving a temporary failure.

 Many such servers use broken SMTP implementations that can't handle
 a 4xx code in response to RCPT properly.

 We greylist after the end of DATA.  This wastes bandwidth, but lets us
 use the Subject: line as an additional mix in the greylisting tuple.
 This catches ratware that retries in the face of greylisting, but
 mutates the subject line with each retry.

 Also, once a given IP passes greylisting, we remember that and we don't
 greylist that server for 40 days.  If you have a large-enough user population,
 this can greatly mitigate the problems caused by initial greylisting delays.

Every 60 seconds we look at all messages that arrived in last 60
seconds.  If there Spamassassin score is less the 1 we add that server
to a whitelist for 6 months.  If its already on whitelist we update
the last message time.  If a message scores over 5 we remove it from
whitelist if its on it.  We do not greylist servers on the whitelist.
Works very well.  Even though we use greylisting our users very rarely
notice if at all due to this.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Axb]

Just wondering how many

boxes:
rcpt domains:
rcpt users:

you guys are sending through greylisting.

Axb

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[John Hardin]

On Thu, 29 Nov 2012, David F. Skoll wrote:


On Thu, 29 Nov 2012 21:27:19 +0100
Andrzej A. Filip andrzej.fi...@gmail.com wrote:


Do you treat yahoo like spam sources in the same way?


With respect to greylisting, of course.  If a machine passes greylisting 
once, it's extremely likely to pass it in future and it's an utter waste 
of time to greylist it.


Modulo spamvertised URIs and spam checksums sent via such hosts, 
particularly if they are freemail.


Filtering out the spambots who don't retry (and as trivial as that is to 
defeat, a large amount still gets blocked by this in my experience) is not 
the _only_ reason to greylist. Giving the URIBLs a chance to list a new 
URI and the checksum services a chance to recognize a new body are also 
benefits of greylisting. (But, as you said, you don't take advantage of 
those tools.)


Also, greylisting generally keys on host+sender, not just host.

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Bother, said Pooh as he struggled with /etc/sendmail.cf, it never
  does quite what I want. I wish Christopher Robin was here.
   -- Peter da Silva in a.s.r
---
 26 days until Christmas

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[David F. Skoll]

On Thu, 29 Nov 2012 22:47:45 +0100
Axb axb.li...@gmail.com wrote:

 boxes:

About 50 000

 rcpt domains:

About 2000

 rcpt users:

Lots.  I don't have an exact figure.

 you guys are sending through greylisting.

This is on our machines.  Our larger customers have significantly
higher numbers.

Regards,

David.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[John Levine]

Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in
case of yahoo like spam sources?

No.  A remarkable fraction of ratware still doesn't bother to retry,
so the most simple minded greylister will deter them.  That's why it's
useful.  I've never seen any support for the theory that greylisting
delays make it more likely that the host will be blacklisted when it
retries.

I haven't seen many legit senders that don't retry as David says he
has, but I don't have his volume of mail, either.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[John Hardin]

On Thu, 30 Nov 2012, John Levine wrote:


Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in
case of yahoo like spam sources?


No.  A remarkable fraction of ratware still doesn't bother to retry,
so the most simple minded greylister will deter them.  That's why it's
useful.  I've never seen any support for the theory that greylisting
delays make it more likely that the host will be blacklisted when it
retries.


It's not so much the host being blacklisted, as a checksum of the spam 
being published by pyzor et. al., or for spamvertised websites in the spam 
being published by URIBLs, so that when the sender tries again the score 
for that message will be higher than it would the first time around, 
hopefully high enough to classify it as spam rather than a FN.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Bother, said Pooh as he struggled with /etc/sendmail.cf, it never
  does quite what I want. I wish Christopher Robin was here.
   -- Peter da Silva in a.s.r
---
 26 days until Christmas

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[David F. Skoll]

On Thu, 29 Nov 2012 18:01:38 -0800 (PST)
John Hardin jhar...@impsec.org wrote:

 It's not so much the host being blacklisted, as a checksum of the
 spam being published by pyzor et. al., or for spamvertised websites
 in the spam being published by URIBLs, so that when the sender tries
 again the score for that message will be higher than it would the
 first time around, hopefully high enough to classify it as spam
 rather than a FN.

I would love to gather some hard data on this.  Maybe a research
project for the future... since we do our greylisting post-DATA,
we could in principle run all the content-filtering and URIBL lookups
and check if the score changes between the first attempt and the final
attempt after greylisting.  Or those who use SA without greylisting
could reprocess messages after an hour or two and see if the score
goes up.

[My gut instinct says that a reasonable greylisting interval is too
short for most DNSBLs to react.  Pyzor/Razor/DCC may be somewhat more
adept at reacting quickly.]

Regards,

David.

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Dave Warren]

On 11/29/2012 17:37, John Levine wrote:

Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in
case of yahoo like spam sources?

No.  A remarkable fraction of ratware still doesn't bother to retry,
so the most simple minded greylister will deter them.  That's why it's
useful.  I've never seen any support for the theory that greylisting
delays make it more likely that the host will be blacklisted when it
retries.


If I run my accepted-and-quarantined spam corpus through a filter to 
test against DNSBL effectiveness, I always see higher effectiveness 
ratings than what was shown during the SMTP phase. I haven't done so in 
recent enough memory to have any actual numbers, but when I last did a 
comparison, slow moving DNSBLs showed little/no change at all, 
fast-acting trap-driven ones show more of a difference.


Now I've not studied the exactly amount of time it takes for hosts to 
start getting listed, but since I only greylist questionable stuff 
already and since I whitelist aggressively, I've been able to set my 
greylisting in the 30-60 minute range without too many seizures from 
users and with higher rejection counts -- Since greylisting doesn't 
cause higher reject counts, I assume (yes, just assume) that it's due to 
higher hit rates.


I admit that it would make sense to do further testing, but for 
fast-acting DNSBLs, and body-hash based systems, it makes sense that the 
longer one defers a message, the greater the odds of a hit against a new 
zombie or a new spam-run.


--
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren

Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)

[Dave Warren]

On 11/29/2012 18:54, David F. Skoll wrote:

[My gut instinct says that a reasonable greylisting interval is too
short for most DNSBLs to react.  Pyzor/Razor/DCC may be somewhat more
adept at reacting quickly.]


Something trap-driven like NIX is a candidate. No, it's not safe enough 
to reject based on it's output, but it was worth use in a scoring 
system. Invalument too responds reasonably quickly, enough that it 
sometimes tripped during the greylist period.


The other trick is how you define reasonable. A reasonable greylist 
period for greylisting all mail is about 3 seconds, otherwise you'll 
have users screaming. However, if you only greylist questionable stuff 
to start with (rDNS failures, mismatches, etc, SPF fails, 
borderline-spammy stuff, DUL hits), you can get away with much longer 
times since most of it is crap anyway but a greylist period can help let 
the odd gem through.


--
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren

greylisting

[Per Jessen]

Giles Coochey wrote:

 This is not intended as a slag-off of greylisting, simply a statement
 that it requires constant maintenance if you are going to be sure that
 all legitimate mail is reaching its destination.

The main/only problem I have with greylisting are otherwise legit 
servers that don't do retries - usually unpatched Exchange 2003
servers. 
Apart from those, I hardly ever touch any of the greylisting setup. I
don't greylist everything though, maybe that is the difference. 


/Per Jessen, Zürich

Re: greylisting

[David F. Skoll]

On Tue, 17 May 2011 09:46:09 +0200
Per Jessen p...@computer.org wrote:

 The main/only problem I have with greylisting are otherwise legit 
 servers that don't do retries - usually unpatched Exchange 2003
 servers. 

I've never seen any Exchange server of any version fail greylisting.
(Again, I do it post-DATA which might explain it.)

Regards,

David.

Re: greylisting

[Per Jessen]

David F. Skoll wrote:

 On Tue, 17 May 2011 09:46:09 +0200
 Per Jessen p...@computer.org wrote:
 
 The main/only problem I have with greylisting are otherwise legit
 servers that don't do retries - usually unpatched Exchange 2003
 servers.
 
 I've never seen any Exchange server of any version fail greylisting.
 (Again, I do it post-DATA which might explain it.)

(I greylist before DATA.)
I think I see a new one about perhaps once a month.  The Exchange server
will build up a queue of emails and NDRs, which is only released when
the server is restarted.  I'm pretty certain this is the applicable
hotfix:

http://support.microsoft.com/kb/950757/


/Per Jessen, Zürich

Re: greylisting

[Ted Mittelstaedt]

On 5/17/2011 4:56 AM, Per Jessen wrote:

David F. Skoll wrote:


On Tue, 17 May 2011 09:46:09 +0200
Per Jessenp...@computer.org  wrote:


The main/only problem I have with greylisting are otherwise legit
servers that don't do retries - usually unpatched Exchange 2003
servers.


I've never seen any Exchange server of any version fail greylisting.
(Again, I do it post-DATA which might explain it.)


(I greylist before DATA.)
I think I see a new one about perhaps once a month.  The Exchange server
will build up a queue of emails and NDRs, which is only released when
the server is restarted.  I'm pretty certain this is the applicable
hotfix:

http://support.microsoft.com/kb/950757/


/Per Jessen, Zürich




Unfortunately a lot of people don't have automatic updates running on
their exchange servers.

Ted

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Steve Freegard]

On 19/01/11 15:02, David F. Skoll wrote:

On Wed, 19 Jan 2011 09:56:47 -0500
Lee Dilkiel...@dilkie.com  wrote:


The second was that I've found that the other spam-catching filtering
is doing a much better job than it was years ago and turning off
greylisting didn't adversely affect the amount of spam that got
through.


That's possibly true, but look at this.

A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

On a busy system, this can make a huge difference.  SpamAssassin scanning
is by no means cheap.



I know this thread is a bit old now; but at the time this was being 
discussed I was running a test as I decided to revisit greylisting and 
see if it was worth keeping in our products.


I found the results very interesting (to me at least), so I decided to 
write a whitepaper and share my results:


See http://www.fsl.com/index.php/resources/whitepapers/99

Kind regards,
Steve.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

Hi, Steve,

 http://www.fsl.com/index.php/resources/whitepapers/99

Interesting.  I think you should credit me for this:

Once that has been proven then that â is exempted from further
greylisting for 40 days since it was last seen.

Our CanIt system has been doing that since at least 2005, and AFAIK
was the first to do so.  I understand if you don't want to credit a direct
competitor :), but at least include my name.

Also, I mentioned greylisting before Evan Harris's paper.  It's here:

http://groups.google.com/group/comp.mail.sendmail/msg/ef7624049e1193a2?hl=en

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Tue, 08 Feb 2011 15:47:12 +
Steve Freegard st...@stevefreegard.com wrote:

 See http://www.fsl.com/index.php/resources/whitepapers/99

Once that has been proven then that 'hostid' is exempted from further
greylisting for 40 days since it was last seen.

:)  Our CanIt system has been doing this since at least 2005.

Also, I wrote about greylisting a few months before Evan Harris
published his paper:

http://groups.google.com/group/comp.mail.sendmail/msg/ef7624049e1193a2?hl=en

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Steve Freegard]

Hi David,

On 08/02/11 15:57, David F. Skoll wrote:

Hi, Steve,


http://www.fsl.com/index.php/resources/whitepapers/99


Interesting.  I think you should credit me for this:

Once that has been proven then that â is exempted from further
greylisting for 40 days since it was last seen.

Our CanIt system has been doing that since at least 2005, and AFAIK
was the first to do so.  I understand if you don't want to credit a direct
competitor :), but at least include my name.


I wasn't aware and have no way of verifying that.  Besides it's a pretty 
obvious thing to do when you look at what is trying to be proven.  We've 
been doing this for ages too (since v1 in 2007).



Also, I mentioned greylisting before Evan Harris's paper.  It's here:

http://groups.google.com/group/comp.mail.sendmail/msg/ef7624049e1193a2?hl=en



Sure - credit where it is due; I've you to the 'Thanks' section.

Cheers,
Steve.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Tue, 08 Feb 2011 17:04:37 +
Steve Freegard st...@stevefreegard.com wrote:

 Sure - credit where it is due; I've you to the 'Thanks' section.

Thanks.  And also, my apologies for posting to the list... that was supposed
to be a private message. :(

/me mutters something about email amateurs not understanding how email works...

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Rolf E. Sonneveld]

On 1/19/11 2:10 AM, David F. Skoll wrote:

On Tue, 18 Jan 2011 23:37:07 +0100
Rolf E. Sonneveldr.e.sonnev...@sonnection.nl  wrote:


I agree with you, looking at my own personal situation. However, many
mail admins (and maybe you too) are responsible for the e-mail
handling of many (tens/hundreds/thousands) of users. Most users have
unrealistic expectations of e-mail delivery times, and become
impatient when an e-mail they send is not delivered after a few
minutes. We can tell them they should not have these expectations,
but they just have them. User education is a tough task. How many
phone calls start with 'Hiname, how are you, did you receive my
mail?'.

User education can go a long way.  One of our (very large) customers
filters mail for about 900 000 email addresses and uses greylisting.
They've obviously decided the benefits outweigh the costs.

[...]


I wish everyone, using greylisting, would do what you did. That sure
would reduce collateral damage! I have a question about your setup:
do you automatically greylist senders to whom you sent mail the last
6 months?

We whitelist those senders...


Excuse me, I meant to say: whitelist instead of greylist.


If so, do you differentiate between interpersonal messages
(legit mail from you to that sender) and out-of-office type messages
(which can be the result of spam messages and can carry your mail
address, depending on what type of mail system you use)?

We attempt to.  We look for the standard Auto-Submitted header which
good auto-responders add.  And we use heuristics to try to catch the
crappy auto-responders (typically, any MTA made by a large company like
Microsoft or IBM qualifies as crappy) though those are not completely
accurate.

Another thing we could do in principle but don't currently is share data
about which machines pass greylisting.


Interesting idea... It would probably help to further minimize 
collateral damage for those organizations that can use this data.


Regards,
/rolf

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Lee Dilkie]

I recently gave up on greylisting after using it for years as well.

Two reasons really, one was the complaints from users (and I found that
they often asked folks to send mail to me twice to try and get mail to
work better and that was just embarrassing).

The second was that I've found that the other spam-catching filtering is
doing a much better job than it was years ago and turning off
greylisting didn't adversely affect the amount of spam that got through.

-lee


On 1/18/2011 5:41 PM, Warren Togami Jr. wrote:
 On 01/18/2011 12:31 PM, David F. Skoll wrote:
 On Tue, 18 Jan 2011 22:18:20 +
 Gary Forrestga...@netnorth.co.uk  wrote:

 Interesting 2 of our 3 scanning heads use a grey list system that
 uses /32 addresses as part of the process, these two servers have
 100's of emails delayed for well over a day. Our 3rd scanning head
 uses a grey list system that is less granular /24 ,  this does not.

 Ah, I should mention that we use a /24 for greylisting for IPv4 and a
 /64 for IPv6.  On the other hand, we also add a hash of the subject
 into the greylisting tuple so it becomes:

 I recently gave up entirely on greylisting after:

 * Last week I discovered /24 was not good enough for redelivery
 attempts at one major ISP.  All mail from that ISP was failing for the
 past month except in rare cases where randomly the same /24 attempted
 delivery within the time window.

 * Years of complaints of mail delivery delays or failures from my
 users.  They had began creating gmail accounts in order to bypass. 
 They kept running into too many cases of broken individual mail
 servers (major companies!) who failed to redeliver.

 Users don't care about so and so is violating RFC-XXX.  They are
 trying to get business done and it was simply causing too many problems.

 Warren

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Wed, 19 Jan 2011 09:56:47 -0500
Lee Dilkie l...@dilkie.com wrote:

 The second was that I've found that the other spam-catching filtering
 is doing a much better job than it was years ago and turning off
 greylisting didn't adversely affect the amount of spam that got
 through.

That's possibly true, but look at this.

A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

On a busy system, this can make a huge difference.  SpamAssassin scanning
is by no means cheap.

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Lee Dilkie]

On 1/19/2011 10:02 AM, David F. Skoll wrote:
 On Wed, 19 Jan 2011 09:56:47 -0500
 Lee Dilkie l...@dilkie.com wrote:

 The second was that I've found that the other spam-catching filtering
 is doing a much better job than it was years ago and turning off
 greylisting didn't adversely affect the amount of spam that got
 through.
 That's possibly true, but look at this.

 A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
 A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

 On a busy system, this can make a huge difference.  SpamAssassin scanning
 is by no means cheap.

 Regards,

 David.

Agreed there, I did have to install the compiled regex package to get SA
speeds up enough to handle the increased load (my server is not even
close to yours in performance but I did drop SA time from 10-30s to 3s).

Don't get me wrong, I liked GL but there are a number of big ISPs that
have quite long retry timeouts (for some reason, sympatico comes to
mind) and it got to be too annoying.

who knows, all the code is still there and I might switch it on again in
the future.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[John Hardin]

On Wed, 19 Jan 2011, Lee Dilkie wrote:

Don't get me wrong, I liked GL but there are a number of big ISPs that 
have quite long retry timeouts (for some reason, sympatico comes to 
mind) and it got to be too annoying.


...and when you encounter a big ISP that does this, do you notify their 
postmaster so they can fix the problem?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Taking my gun away because I *might* shoot someone is like cutting
  my tongue out because I *might* yell Fire! in a crowded theater.
  -- Peter Venetoklis
---
 4 days until John Moses Browning's 156th Birthday

Re: Suspicious URL:Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Daniel McDonald]

On 1/19/11 10:17 AM, John Hardin jhar...@impsec.org wrote:

 On Wed, 19 Jan 2011, Lee Dilkie wrote:
 
 Don't get me wrong, I liked GL but there are a number of big ISPs that
 have quite long retry timeouts (for some reason, sympatico comes to
 mind) and it got to be too annoying.
 
 ...and when you encounter a big ISP that does this, do you notify their
 postmaster so they can fix the problem?

Or add a grey-listing exception and publish it to the sqlgrey list so that
the rest of us can also add an exception?

I seldom have problems with large mailers.  Most of my greylisting issues
come from small organizations.  I usually end up exempting them from
grey-listing, after we get their DNS cleaned up


-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Matt]

 The legitimate mail that passes through my mail server comes from
 hosts / networks I might not hear from again for months, by which
 time I have to potentially wait 24 hours for the greylisting / mail
 server to try again.

I run greylisting on an email server with several thousand email
accounts and think its great.  Reduced system load drastically.  I
also have a perl script I have wrote that runs every minute and looks
at all messages that arrived in last 60 seconds and if spamassassin
gave them a score of less then 5 it adds the sending MTA to a
whitelist.  It also removes any from the whitelist that have sent a
message that scored over 5.  The whitelist goes back 6 months and is
continually refreshed by the script.  The whitelist typically has 40K
IP's in it.

As a result no one even notices the greylisting, AFAIK...

Re: Suspicious URL:Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Henrik K]

On Wed, Jan 19, 2011 at 11:14:29AM -0600, Daniel McDonald wrote:
 On 1/19/11 10:17 AM, John Hardin jhar...@impsec.org wrote:
 
  On Wed, 19 Jan 2011, Lee Dilkie wrote:
  
  Don't get me wrong, I liked GL but there are a number of big ISPs that
  have quite long retry timeouts (for some reason, sympatico comes to
  mind) and it got to be too annoying.
  
  ...and when you encounter a big ISP that does this, do you notify their
  postmaster so they can fix the problem?
 
 Or add a grey-listing exception and publish it to the sqlgrey list so that
 the rest of us can also add an exception?

Or use DNSWL and co which include many major relays?

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Ted Mittelstaedt]

On 1/19/2011 9:25 AM, Matt wrote:

The legitimate mail that passes through my mail server comes from
hosts / networks I might not hear from again for months, by which
time I have to potentially wait 24 hours for the greylisting / mail
server to try again.


I run greylisting on an email server with several thousand email
accounts and think its great.  Reduced system load drastically.  I
also have a perl script I have wrote that runs every minute and looks
at all messages that arrived in last 60 seconds and if spamassassin
gave them a score of less then 5 it adds the sending MTA to a
whitelist.  It also removes any from the whitelist that have sent a
message that scored over 5.  The whitelist goes back 6 months and is
continually refreshed by the script.  The whitelist typically has 40K
IP's in it.

As a result no one even notices the greylisting, AFAIK...


Is that the greylist whitelist or the SA whitelist?

Ted

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Matt]

 The legitimate mail that passes through my mail server comes from
 hosts / networks I might not hear from again for months, by which
 time I have to potentially wait 24 hours for the greylisting / mail
 server to try again.

 I run greylisting on an email server with several thousand email
 accounts and think its great.  Reduced system load drastically.  I
 also have a perl script I have wrote that runs every minute and looks
 at all messages that arrived in last 60 seconds and if spamassassin
 gave them a score of less then 5 it adds the sending MTA to a
 whitelist.  It also removes any from the whitelist that have sent a
 message that scored over 5.  The whitelist goes back 6 months and is
 continually refreshed by the script.  The whitelist typically has 40K
 IP's in it.

 As a result no one even notices the greylisting, AFAIK...

 Is that the greylist whitelist or the SA whitelist?

It whitelists those IP's from being greylisted in future.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Ted Mittelstaedt]

On 1/19/2011 8:06 AM, Lee Dilkie wrote:


On 1/19/2011 10:02 AM, David F. Skoll wrote:

On Wed, 19 Jan 2011 09:56:47 -0500
Lee Dilkiel...@dilkie.com  wrote:


The second was that I've found that the other spam-catching filtering
is doing a much better job than it was years ago and turning off
greylisting didn't adversely affect the amount of spam that got
through.

That's possibly true, but look at this.

A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

On a busy system, this can make a huge difference.  SpamAssassin scanning
is by no means cheap.

Regards,

David.


Agreed there, I did have to install the compiled regex package to get SA
speeds up enough to handle the increased load (my server is not even
close to yours in performance but I did drop SA time from 10-30s to 3s).

Don't get me wrong, I liked GL but there are a number of big ISPs that
have quite long retry timeouts (for some reason, sympatico comes to
mind) and it got to be too annoying.



In our experience the large ISPs don't have long retry timeouts.  What 
they have are multiple outbound mailservers.  They will try from 1 
server and when they get the error 4xx they shift the outbound message 
to another server.  This happens until all of their outbound servers

have been greylisted for the one message, then it goes through.

In my opinion (we use greylist-milter) the greylist developers are
basically their own worst enemies here.  They distribute a list of known
ISPs that round-robin outbound mail.  But the list is very old and
isn't a quarter of the ones that actually do it.

So an admin inexperienced with greylisting installs it and gets the 
experience your relating and then assumes that greylisting is no good.


Note that I am not assuming your inexperienced or that you don't already
know all about this problem.  You just didn't mention it so I didn't
want others who might come across this posting who might not be 
experienced with this to not know about it.


In our case greylisting is very cheap on CPU cycles.  But the regex 
matching and virus filtering is quite expensive.  And worse, we have

to pass everything to the users including the spam that we have tagged,
so we cannot do the logical thing and put SA first and then just throw
away from further processing everything that is determined to be spam.
Instead we have to put the virus filtering first (because we are allowed
to toss virus-infected mail) which means everything gets both scanned
for spam (except viruses) and virus filtered.

So we prefilter with greylist-milter and it really does indeed 
tremendously reduce the load on the server.  But you really do have to

explain to your users what is going on for it to work, and you
have to thoroughly investigate every mail complaint to make sure that
it's not caused by a round-robin ISP that you don't have in your
exception list.  And you have be alert for hosts like
craigslist.org because those bastards
fail mail delivery EVEN IF they get an error 4xx in violation of
the RFCs.

Ted


who knows, all the code is still there and I might switch it on again in
the future.

Re: Suspicious URL:Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[John Hardin]

On Wed, 19 Jan 2011, Daniel McDonald wrote:


On 1/19/11 10:17 AM, John Hardin jhar...@impsec.org wrote:


On Wed, 19 Jan 2011, Lee Dilkie wrote:


Don't get me wrong, I liked GL but there are a number of big ISPs that
have quite long retry timeouts (for some reason, sympatico comes to
mind) and it got to be too annoying.


...and when you encounter a big ISP that does this, do you notify their
postmaster so they can fix the problem?


Or add a grey-listing exception and publish it to the sqlgrey list so that
the rest of us can also add an exception?


Is the whitelist available standalone for those of us who don't use 
sqlgrey? I couldn't see it and didn't want to grab the entire tarball.


(As I was researching this I came across a posting to the sqlgrey list 
from 2005 mentioning a whitelist entry request on behalf of a C/R vendor, 
and my first thought was what, we want to _encourage_ C/R?)


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  One difference between a liberal and a pickpocket is that if you
  demand your money back from a pickpocket he will not question your
  motives.  -- William Rusher
---
 4 days until John Moses Browning's 156th Birthday

Re: Suspicious URL:Re: Suspicious URL:Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Daniel McDonald]

On 1/19/11 2:35 PM, John Hardin jhar...@impsec.org wrote:

 On Wed, 19 Jan 2011, Daniel McDonald wrote:
 
 On 1/19/11 10:17 AM, John Hardin jhar...@impsec.org wrote:
 
 On Wed, 19 Jan 2011, Lee Dilkie wrote:
 
 Don't get me wrong, I liked GL but there are a number of big ISPs that
 have quite long retry timeouts (for some reason, sympatico comes to
 mind) and it got to be too annoying.
 
 ...and when you encounter a big ISP that does this, do you notify their
 postmaster so they can fix the problem?
 
 Or add a grey-listing exception and publish it to the sqlgrey list so that
 the rest of us can also add an exception?
 
 Is the whitelist available standalone for those of us who don't use
 sqlgrey? I couldn't see it and didn't want to grab the entire tarball.
 
 (As I was researching this I came across a posting to the sqlgrey list
 from 2005 mentioning a whitelist entry request on behalf of a C/R vendor,
 and my first thought was what, we want to _encourage_ C/R?)
The files are accessible at
http://sqlgrey.bouton.name

The available files are MD5SUMS, README, clients_fqdn_whitelist,
clients_ip_whitelist, dyn_fqdn.regexp, smtp_server.regexp

There is a script in the tarball to retrieve the changed files by comparing
the published md5sum with that on disk and only pulling down those that are
different.


-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281

Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Tue, 18 Jan 2011 16:55:42 +0100
Giles Coochey gi...@coochey.net wrote:

 The legitimate mail that passes through my mail server comes from
 hosts / networks I might not hear from again for months, by which
 time I have to potentially wait 24 hours for the greylisting / mail
 server to try again.

My point is I've never encountered a client that waits 24h to retry.
Most retry within minutes and the longest I've seen is maybe 4h.

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Rolf E. Sonneveld]

On 1/18/11 4:58 PM, David F. Skoll wrote:

On Tue, 18 Jan 2011 16:55:42 +0100
Giles Coocheygi...@coochey.net  wrote:


The legitimate mail that passes through my mail server comes from
hosts / networks I might not hear from again for months, by which
time I have to potentially wait 24 hours for the greylisting / mail
server to try again.

My point is I've never encountered a client that waits 24h to retry.
Most retry within minutes and the longest I've seen is maybe 4h.


RFC821/RFC2821/RFC5321 points out that a client has to wait a minimum of 
30 minutes before a retry attempt should be made, unless the client is 
able to determine what the reason is for the delay:



The sender MUST delay retrying a particular destination after one
attempt has failed.  In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.


As greylisting has never been standardized, there is no way for the 
client to reliably determine after what interval a retry should be made.


Can you document that 'Most retry within minutes'? My experience is that 
greylisting is causing real collateral damage due to the fact that many 
MTA's use long retry intervals, at least longer than a few minutes*): 
people get confused because their postings to mailing lists are delayed 
while the discussion on the list goes on; people give up trying to 
register themselves with sites, which send a confirmation message to the 
customer: the customer is waiting for the confirmation mail and gives up 
after a few minutes.


But I think you're right: I've yet to see the first MTA that waits for 
24 hours before the first retry is done.


/rolf

*) The default retry interval for Postfix is 20 minutes, the default 
retry interval for Sendmail is 1 hour, the default for Exchange is 
(dependent on the role) 10 minutes or 30 minutes, default of Sun/Oracle 
Messaging Server is 30 minutes or more, etc. etc.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Tue, 18 Jan 2011 22:18:33 +0100
Rolf E. Sonneveld r.e.sonnev...@sonnection.nl wrote:

 RFC821/RFC2821/RFC5321 points out that a client has to wait a minimum
 of 30 minutes before a retry attempt should be made,

That's fine.  I don't care if an email from someone I've never heard
from before is delayed 30 minutes or even a couple of hours.

 Can you document that 'Most retry within minutes'?

I analyzed 1655 machines from my logs that successfully passed greylisting.
The mean retry time was about 13000 seconds, or about 3.6 hours.

The standard deviation was high at 45000 seconds (12.5 hours).

That being said, 1390 of the 1655 machines retried within 4 hours and
1586 of 1655 retried within a day.  Every one of the machines that took
longer than a day was actually a spam box that only accidentally passed
greylisting when it tried sending a second spam using the same sender
address.

 My experience is that greylisting is causing real collateral damage
 due to the fact that many MTA's use long retry intervals, at least
 longer than a few minutes*): people get confused because their
 postings to mailing lists are delayed while the discussion on the
 list goes on; people give up trying to register themselves with
 sites, which send a confirmation message to the customer: the
 customer is waiting for the confirmation mail and gives up after a
 few minutes.

Our system notices when a host retries and turns off greylisting for 40 days
for that host.  That can greatly reduce the collateral damage.  It also
won't greylist senders to whom I've sent mail within the last 6 months.

For me, greylisting is so cheap and so effective I'm willing to live with
the small amount of colateral damage it introduces.

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Gary Forrest]

Hi All

To answer David's post, extract from our scanning system for today.

*Jan 18 01:53:19 sendmail[8404]: p0I1rIDg008404: 
from=debenhams5-boun...@shopdebenhams.com, size=43048, class=0, 
nrcpts=1, msgid=debenh...@shopdebenhams.com, proto=ESMTP, 
daemon=MTA, relay=revd138.shopdebenhams.com [195.154.153.138]
Jan 18 01:53:19 sendmail[8404]: p0I1rIDg008404: Milter add: header: 
X-Greylist: Delayed for 117:43:55 by milter-greylist-4.0.1 
(avs3.netnorth.co.uk [82.148.225.24]); Tue, 18 Jan 2011 01:53:19 + (GMT)


that's a greylist delay of **117:43:55* - almost 5 days !

Interesting 2 of our 3 scanning heads use a grey list system that uses 
/32 addresses as part of the process, these two servers have 100's of 
emails delayed for well over a day. Our 3rd scanning head uses a grey 
list system that is less granular /24 ,  this does not.



Gary


*
*On 18/01/2011 15:58, David F. Skoll wrote:

On Tue, 18 Jan 2011 16:55:42 +0100
Giles Coocheygi...@coochey.net  wrote:


The legitimate mail that passes through my mail server comes from
hosts / networks I might not hear from again for months, by which
time I have to potentially wait 24 hours for the greylisting / mail
server to try again.

My point is I've never encountered a client that waits 24h to retry.
Most retry within minutes and the longest I've seen is maybe 4h.

Regards,

David.





 Message Scanning REF:470-avs3-1295366379 



--

|Gary Forrest
|(Director)
|Email: ga...@netnorth.co.uk
|Tel: 0845 058 2001
|Fax: 01204 900719
|
|Netnorth Limited
|Units 7 and 8 Queensbrook
|Bolton Technology Exchange
|Spa Road
|Bolton
|BL1 4AY
|
|Sales queries:  sa...@netnorth.co.uk
|Domain name queries: doma...@netnorth.co.uk
|Support queries: supp...@netnorth.co.uk
|Accounts queries: accou...@netnorth.co.uk

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Tue, 18 Jan 2011 22:18:20 +
Gary Forrest ga...@netnorth.co.uk wrote:

 Interesting 2 of our 3 scanning heads use a grey list system that
 uses /32 addresses as part of the process, these two servers have
 100's of emails delayed for well over a day. Our 3rd scanning head
 uses a grey list system that is less granular /24 ,  this does not.

Ah, I should mention that we use a /24 for greylisting for IPv4 and a
/64 for IPv6.  On the other hand, we also add a hash of the subject
into the greylisting tuple so it becomes:

   {sender_address, recipient_address, sender_ip/24, subject_hash}

because we found a number of spammers bypassing greylisting but mutating
the message subject each time.

Regards,

David.

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Rolf E. Sonneveld]

On 1/18/11 11:02 PM, David F. Skoll wrote:

On Tue, 18 Jan 2011 22:18:33 +0100
Rolf E. Sonneveldr.e.sonnev...@sonnection.nl  wrote:


RFC821/RFC2821/RFC5321 points out that a client has to wait a minimum
of 30 minutes before a retry attempt should be made,

That's fine.  I don't care if an email from someone I've never heard
from before is delayed 30 minutes or even a couple of hours.


I agree with you, looking at my own personal situation. However, many 
mail admins (and maybe you too) are responsible for the e-mail handling 
of many (tens/hundreds/thousands) of users. Most users have unrealistic 
expectations of e-mail delivery times, and become impatient when an 
e-mail they send is not delivered after a few minutes. We can tell them 
they should not have these expectations, but they just have them. User 
education is a tough task. How many phone calls start with 'Hi name, 
how are you, did you receive my mail?'.



Can you document that 'Most retry within minutes'?

I analyzed 1655 machines from my logs that successfully passed greylisting.
The mean retry time was about 13000 seconds, or about 3.6 hours.

The standard deviation was high at 45000 seconds (12.5 hours).

That being said, 1390 of the 1655 machines retried within 4 hours and
1586 of 1655 retried within a day.  Every one of the machines that took
longer than a day was actually a spam box that only accidentally passed
greylisting when it tried sending a second spam using the same sender
address.


Thanks for sharing these figures, they're really useful even though the 
standard deviation is high.



My experience is that greylisting is causing real collateral damage
due to the fact that many MTA's use long retry intervals, at least
longer than a few minutes*): people get confused because their
postings to mailing lists are delayed while the discussion on the
list goes on; people give up trying to register themselves with
sites, which send a confirmation message to the customer: the
customer is waiting for the confirmation mail and gives up after a
few minutes.

Our system notices when a host retries and turns off greylisting for 40 days
for that host.  That can greatly reduce the collateral damage.  It also
won't greylist senders to whom I've sent mail within the last 6 months.


I wish everyone, using greylisting, would do what you did. That sure 
would reduce collateral damage! I have a question about your setup: do 
you automatically greylist senders to whom you sent mail the last 6 
months? If so, do you differentiate between interpersonal messages 
(legit mail from you to that sender) and out-of-office type messages 
(which can be the result of spam messages and can carry your mail 
address, depending on what type of mail system you use)?



For me, greylisting is so cheap and so effective I'm willing to live with
the small amount of colateral damage it introduces.


The right anti-spam defense is always a balance between catching as much 
spam as possible and avoiding false positives (and other collateral 
damage) as much as possible. IMHO too many mail admins focus on the 
former, forgetting about the latter. The net result is that the 
reliability of Internet e-mail system has degraded over the last 15 years.


/rolf

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[Warren Togami Jr.]

On 01/18/2011 12:31 PM, David F. Skoll wrote:

On Tue, 18 Jan 2011 22:18:20 +
Gary Forrestga...@netnorth.co.uk  wrote:


Interesting 2 of our 3 scanning heads use a grey list system that
uses /32 addresses as part of the process, these two servers have
100's of emails delayed for well over a day. Our 3rd scanning head
uses a grey list system that is less granular /24 ,  this does not.


Ah, I should mention that we use a /24 for greylisting for IPv4 and a
/64 for IPv6.  On the other hand, we also add a hash of the subject
into the greylisting tuple so it becomes:


I recently gave up entirely on greylisting after:

* Last week I discovered /24 was not good enough for redelivery attempts 
at one major ISP.  All mail from that ISP was failing for the past month 
except in rare cases where randomly the same /24 attempted delivery 
within the time window.


* Years of complaints of mail delivery delays or failures from my users. 
 They had began creating gmail accounts in order to bypass.  They kept 
running into too many cases of broken individual mail servers (major 
companies!) who failed to redeliver.


Users don't care about so and so is violating RFC-XXX.  They are 
trying to get business done and it was simply causing too many problems.


Warren

Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

[David F. Skoll]

On Tue, 18 Jan 2011 23:37:07 +0100
Rolf E. Sonneveld r.e.sonnev...@sonnection.nl wrote:

 I agree with you, looking at my own personal situation. However, many 
 mail admins (and maybe you too) are responsible for the e-mail
 handling of many (tens/hundreds/thousands) of users. Most users have
 unrealistic expectations of e-mail delivery times, and become
 impatient when an e-mail they send is not delivered after a few
 minutes. We can tell them they should not have these expectations,
 but they just have them. User education is a tough task. How many
 phone calls start with 'Hi name, how are you, did you receive my
 mail?'.

User education can go a long way.  One of our (very large) customers
filters mail for about 900 000 email addresses and uses greylisting.
They've obviously decided the benefits outweigh the costs.

[...]

 I wish everyone, using greylisting, would do what you did. That sure 
 would reduce collateral damage! I have a question about your setup:
 do you automatically greylist senders to whom you sent mail the last
 6 months?

We whitelist those senders...

 If so, do you differentiate between interpersonal messages 
 (legit mail from you to that sender) and out-of-office type messages 
 (which can be the result of spam messages and can carry your mail 
 address, depending on what type of mail system you use)?

We attempt to.  We look for the standard Auto-Submitted header which
good auto-responders add.  And we use heuristics to try to catch the
crappy auto-responders (typically, any MTA made by a large company like
Microsoft or IBM qualifies as crappy) though those are not completely
accurate.

Another thing we could do in principle but don't currently is share data
about which machines pass greylisting.  We have a reputation-reporting system
that reports on various events, and two of the events it reports are
Machine X was greylisted and Machine X passed greylisting.  We could
publish a DNS zone that you could look up and decide not to greylist
a given machine.

Right now, we have event data on 15.9 million IPv4 addresses.  Of those, about
7.1 million have never passed greylisting (which means we know of about
8.8 million machines that are probably pointless to greylist.)

Regards,

David.

Greylisting (was Re: Anti-Perl rant (was Re: Issuing rollback DBI Mysql))

[David F. Skoll]

On Mon, 27 Dec 2010 12:37:00 -0800
Ted Mittelstaedt t...@ipinc.net wrote:

 greylisting, though, is by far the best.  But I have noticed an 
 increasing number of sites out there - and this is large sites - who
 apparently are honked-off that people greylist, and they will bounce
 delivery of mail that is issued an error 4xx in violation of the
 standard.  Off the top of my head I seem to remember seeing this from
 several airline company mailers that send out the advertisements to
 their frequent flyer members, and that send out electronic ticketing
 receipts.  Jerks!

What you may be seeing is marginal SMTP client software that doesn't
know how to handle a 4xx response to RCPT.  There was even some
commercial software that couldn't deal with this properly (Novell
Groupwise, I believe, though it has long since been fixed in that
product.)

BTW, this is another reason we do our greylisting post-DATA.  Although
it's slower and uses more bandwidth, it does avoid problems with
marginal SMTP clients and it does let us use the Subject: as part of
the greylisting tuple, which greatly increases greylisting
effectiveness.

We do not find virus-scanning before spam-scanning to be effective.  A
tiny percentage of our mail is flagged as containing a virus, so it
doesn't really reduce the amount of mail that would need to be
spam-scanned.

Regards,

David.

Re: Greylisting (was Re: Anti-Perl rant (was Re: Issuing rollback DBI Mysql))

[Ted Mittelstaedt]

On 12/27/2010 12:42 PM, David F. Skoll wrote:

On Mon, 27 Dec 2010 12:37:00 -0800
Ted Mittelstaedtt...@ipinc.net  wrote:


greylisting, though, is by far the best.  But I have noticed an
increasing number of sites out there - and this is large sites - who
apparently are honked-off that people greylist, and they will bounce
delivery of mail that is issued an error 4xx in violation of the
standard.  Off the top of my head I seem to remember seeing this from
several airline company mailers that send out the advertisements to
their frequent flyer members, and that send out electronic ticketing
receipts.  Jerks!


What you may be seeing is marginal SMTP client software that doesn't
know how to handle a 4xx response to RCPT.  There was even some
commercial software that couldn't deal with this properly (Novell
Groupwise, I believe, though it has long since been fixed in that
product.)

BTW, this is another reason we do our greylisting post-DATA.  Although
it's slower and uses more bandwidth, it does avoid problems with
marginal SMTP clients and it does let us use the Subject: as part of
the greylisting tuple, which greatly increases greylisting
effectiveness.

We do not find virus-scanning before spam-scanning to be effective.  A
tiny percentage of our mail is flagged as containing a virus,


That's subject to interpretation I think.  I would guess that your 
LEGITIMATE mail is ALSO a tiny percentage of your total received mail. ;-)


The real question is, do you get viruses that would make it past SA?  We
do, for the simple reason that we have some users who regularly get mail
that is normally flagged as spam - and they WANT that mail - so we
list them in the exemption (all spam to) list.  The virus filtering 
makes sure that they don't get hosed down.


Of course, you can do virus scanning post-SA to capture these.

Ted

 so it

doesn't really reduce the amount of mail that would need to be
spam-scanned.

Regards,

David.

Re: Greylisting (was Re: Anti-Perl rant (was Re: Issuing rollback DBI Mysql))

[David F. Skoll]

On Mon, 27 Dec 2010 13:36:39 -0800
Ted Mittelstaedt t...@ipinc.net wrote:

[...]

  We do not find virus-scanning before spam-scanning to be
  effective.  A tiny percentage of our mail is flagged as containing
  a virus,

 That's subject to interpretation I think.  I would guess that your 
 LEGITIMATE mail is ALSO a tiny percentage of your total received
 mail. ;-)

No, not really.  Here are the statistics for 30 days' worth of mail for
messages that made it past greylisting:

- About 600 000 non-spam messages
- About 530 000 spam or suspected-spam messages
- About 65 000 messages blocked for various reasons other than
  content-filtering (on DNSBL, sender blacklisted by end-user, etc.)
- 774 viruses as detected by ClamAV

As you see, viruses make up a tiny percentage of mail volume.
Non-spam makes up about 50% of the post-greylisting volume
or about 20% of total volume including greylisting.

During that same period, about 2.4 million messages were greylisted, of
which just under 50 000 were retried correctly and made it past the
greylisting hurdle.  Greylisting remains tremendously effective.

 The real question is, do you get viruses that would make it past SA?

I can't answer that because we scan for viruses before SA.  I would
guess yes.  It would be more efficient to scan for viruses after
scanning for spam, even though we still do it the other way around.

Regards,

David.

Re: Greylisting (was Re: Anti-Perl rant (was Re: Issuing rollback DBI Mysql))

[Daniel McDonald]

On 12/27/10 4:07 PM, David F. Skoll d...@roaringpenguin.com wrote:

 On Mon, 27 Dec 2010 13:36:39 -0800
 Ted Mittelstaedt t...@ipinc.net wrote:
 
 The real question is, do you get viruses that would make it past SA?
 
 I can't answer that because we scan for viruses before SA.  I would
 guess yes.  It would be more efficient to scan for viruses after
 scanning for spam, even though we still do it the other way around.

I scan for viruses first, (actually second, after grey-listing) because
clamav with the unofficial signatures identifies a fair amount of spam, and
the non-virus findings are added to the spamassassin score...

-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281

Re: on greylisting...

[Jonas Eckerman]

On 2010-04-01 19:06, Adam Katz wrote:

For what it's worth, I reconfigured my greylisting relay from a
blanket delay to delaying only spamcop neighbors, anything that hits a
DNSBL, and any Windows *desktop* (using p0f).


I once tried that, had had to refrain from it. The groupware system 
FirstClass installed on Windows NT+ (of different flavors, including 
desktop OSes) machines is (or was) popular with swedish disability 
NGOs, and beeing an NGO for deafblind people, we need to be able to 
communicate those systems.


I probably should analyze our current mail stream to see if we still get 
lots of mail from FC systems, and what OSes those seem to be running on 
nowadays.


(The fact that admins of above mentioned FirstClass systems tended to 
configure outgoing SMTP in odd ways also amde m putin some 
country/domainbased exemptions...)



If I recall correctly, Jonas's implementation also uses p0f and could
therefore benefit from my analysis.


Yes, my implementation can use p0f. It uses a list of tests that are 
checked in order to decide wether a sending system sould be handled by 
the grylist or not.


I'm currently using tests for OS (p0f), DNS black- and white-lists, 
RDNS, MX, SPF, country (GeoIP), sender domain, local spam/ham history 
and local otgoing hitory to make that desicion.



p0f's results with the (perl-compatible) regular expression
 /Windows (?:XP|2000(?!SP4)|Vista)/
will safely block only desktops.


Interesting. I hope I'll have time to check that against or logs. It 
would nice to have windows desktops greylisted while still beeing able 
to exempt windows mail and groupware systems.


Regards
/Jonas
--
Jonas Eckerman
Fruktträdet  Förbundet Sveriges Dövblinda
http://www.fsdb.org/
http://www.frukt.org/
http://whatever.frukt.org/

on greylisting...

[Adam Katz]

 On 2010-03-30 01:29, Brent Kennedy wrote:
 Graylisting does work.

Jonas Eckerman wrote:
 I know it works. That's why I said I like it because it stops spam.
 Been using my own implementation for years.

For what it's worth, I reconfigured my greylisting relay from a
blanket delay to delaying only spamcop neighbors, anything that hits a
DNSBL, and any Windows *desktop* (using p0f).

The move reduced the fatal delay of 80-90% of my incoming mail down to
64%, which is pretty reasonable given the fact that the inconvenience
caused by greylisting has all-but vanished: only 3.3% of those delayed
windows desktops makes it through, and more than half of them get
rejected by spamassassin.  (I don't have comparable stats from before
the move.  Also of note:  90% is a BIG number, so there may be a flaw
in my counting, but since this is relative anyway, it doesn't matter.)

My configuration notes were posted to the milter-greylist wiki at
http://milter-greylist.wikidot.com/using-p0f  and my original post is
at  http://tech.groups.yahoo.com/group/milter-greylist/message/5496

If I recall correctly, Jonas's implementation also uses p0f and could
therefore benefit from my analysis.  The gist of it is that matching
p0f's results with the (perl-compatible) regular expression
/Windows (?:XP|2000(?!SP4)|Vista)/
will safely block only desktops.  (Though half of the Windows systems
I see mail from use Windows 2000 SP4, XP SP1+ and it has to be
excluded from the desktops list because there are sooo many MS
Exchange servers out there still running on win2k.  I'd love to see
p0f overcome that limitation...)

-Adam

Re: [SA] on greylisting...

[Adam Katz]

Adam Katz wrote:
 matching p0f's results with the (perl-compatible) regular expression
 /Windows (?:XP|2000(?!SP4)|Vista)/
 will safely block only desktops.

Oops, that lost a space after the 2000 part. That should read:

/Windows (?:XP|2000 (?!SP4)|Vista)/

Also note that p0f is currently incapable of detecting newer items
like Vista, 2008, and 7.  I've included Vista it in my regex and 2008
in my milter-greylist exceptions for completeness in case of updates.

Re: on greylisting...

[Ned Slider]

Adam Katz wrote:


For what it's worth, I reconfigured my greylisting relay from a
blanket delay to delaying only spamcop neighbors, anything that hits a
DNSBL, and any Windows *desktop* (using p0f).

The move reduced the fatal delay of 80-90% of my incoming mail down to
64%, which is pretty reasonable given the fact that the inconvenience



Some implementations, postgrey for example, keep a whitelist of servers 
known to retry, so once a small number of mails (3 by default iirc) have 
been successfully delivered from a given server (or servers in the same 
/24 subnet), it is auto-whitelisted on the basis that there is little 
point continually greylisting servers that you *know* will retry anyway.


This approach works extremely well and after a few weeks normal usage 
very few legitimate mails are delayed by greylisting.

Re: on greylisting...

[Matt]

 For what it's worth, I reconfigured my greylisting relay from a
 blanket delay to delaying only spamcop neighbors, anything that hits a
 DNSBL, and any Windows *desktop* (using p0f).

 The move reduced the fatal delay of 80-90% of my incoming mail down to
 64%, which is pretty reasonable given the fact that the inconvenience


 Some implementations, postgrey for example, keep a whitelist of servers
 known to retry, so once a small number of mails (3 by default iirc) have
 been successfully delivered from a given server (or servers in the same /24
 subnet), it is auto-whitelisted on the basis that there is little point
 continually greylisting servers that you *know* will retry anyway.

 This approach works extremely well and after a few weeks normal usage very
 few legitimate mails are delayed by greylisting.

I wrote a perl script that whitelists any servers from greylisting for
6 months that send a message that scores less then 1 by spamassassin.
If it later sends a message that scores greater then 5 it is removed
from the whitelist.  Works great.  After having a few months to learn
almost no legitimate servers are greylisted.

Matt

Re: on greylisting...

[Adam Katz]

Ned Slider wrote:
 Some implementations, postgrey for example, keep a whitelist of
 servers known to retry, so once a small number of mails (3 by
 default iirc) have been successfully delivered from a given server
 (or servers in the same /24 subnet), it is auto-whitelisted on the
 basis that there is little point continually greylisting servers
 that you *know* will retry anyway.
 
 This approach works extremely well and after a few weeks normal
 usage very few legitimate mails are delayed by greylisting.

My grey time is 35 days, which is effectively the same thing.  By
greylisting only Windows desktops, I can ensure emails sent during a
conference call are received without delay, during a support call,
etc. (though the support address is configured to bypass greylisting
during the work day).  A server's first connection matters!

That said, it would be nice to revoke the grey time for a server that
didn't come back.  That might even get me to make the grey time
bigger, though the larger the database, the longer it takes to search
(which must be a larger problem with postgrey with all that
non-expiring data...).

Re: on greylisting...

[Adam Katz]

Matt wrote:
 I wrote a perl script that whitelists any servers from greylisting for
 6 months that send a message that scores less then 1 by spamassassin.
 If it later sends a message that scores greater then 5 it is removed
 from the whitelist.  Works great.  After having a few months to learn
 almost no legitimate servers are greylisted.

Milter-greylist can do that too.  I haven't had the time yet, but I'll
be eventually moving my relays to use milter-greylist for greylisting
AND spamassassin, which would also let me mix and match.

That could look something like this in your greylist.conf:

dacl whitelist spamd  1
dacl blacklist spamd  8 msg Message blocked for spam content.
dacl  greylist spamd  5

That last line shows what the milter can do, though I wouldn't combine
the two like that.  The development version can also tarpit (which I
would combine with SA) ... see this commercial tarpit's explanation:
http://mailchannels.com/literature/osterman-research-traffic-shaping.pdf

Big benefit here:  that's an SMTP-time REJECT command (which
spamass-milter also does, but amavis, mailscanner, and procmail+spamc
do not).  It generates a bounce from the sending server (read: *not*
backscatter), so senders will always know if their message has not
been delivered (and smarter mailing lists, including some that
spammers use, will auto-unsubscribe a bouncing email account).

I flag at 5 and reject at 8.  Users can then filter/delete flagged
mail as they see fit without being overwhelmed by volume.  I consider
this feature mandatory.

Re: opinions on greylisting and others

[Arvid Picciani]

thanks for your responses.  unfortunatly i lost all my local mail when 
my laptop exploded friday :(

does this list have an online archive?