Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-02-08 Thread Steve Freegard

On 19/01/11 15:02, David F. Skoll wrote:

On Wed, 19 Jan 2011 09:56:47 -0500
Lee Dilkiel...@dilkie.com  wrote:


The second was that I've found that the other spam-catching filtering
is doing a much better job than it was years ago and turning off
greylisting didn't adversely affect the amount of spam that got
through.


That's possibly true, but look at this.

A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

On a busy system, this can make a huge difference.  SpamAssassin scanning
is by no means cheap.



I know this thread is a bit old now; but at the time this was being 
discussed I was running a test as I decided to revisit greylisting and 
see if it was worth keeping in our products.


I found the results very interesting (to me at least), so I decided to 
write a whitepaper and share my results:


See http://www.fsl.com/index.php/resources/whitepapers/99

Kind regards,
Steve.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-02-08 Thread David F. Skoll
Hi, Steve,

 http://www.fsl.com/index.php/resources/whitepapers/99

Interesting.  I think you should credit me for this:

Once that has been proven then that â is exempted from further
greylisting for 40 days since it was last seen.

Our CanIt system has been doing that since at least 2005, and AFAIK
was the first to do so.  I understand if you don't want to credit a direct
competitor :), but at least include my name.

Also, I mentioned greylisting before Evan Harris's paper.  It's here:

http://groups.google.com/group/comp.mail.sendmail/msg/ef7624049e1193a2?hl=en

Regards,

David.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-02-08 Thread David F. Skoll
On Tue, 08 Feb 2011 15:47:12 +
Steve Freegard st...@stevefreegard.com wrote:

 See http://www.fsl.com/index.php/resources/whitepapers/99

Once that has been proven then that 'hostid' is exempted from further
greylisting for 40 days since it was last seen.

:)  Our CanIt system has been doing this since at least 2005.

Also, I wrote about greylisting a few months before Evan Harris
published his paper:

http://groups.google.com/group/comp.mail.sendmail/msg/ef7624049e1193a2?hl=en

Regards,

David.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-02-08 Thread Steve Freegard

Hi David,

On 08/02/11 15:57, David F. Skoll wrote:

Hi, Steve,


http://www.fsl.com/index.php/resources/whitepapers/99


Interesting.  I think you should credit me for this:

Once that has been proven then that â is exempted from further
greylisting for 40 days since it was last seen.

Our CanIt system has been doing that since at least 2005, and AFAIK
was the first to do so.  I understand if you don't want to credit a direct
competitor :), but at least include my name.


I wasn't aware and have no way of verifying that.  Besides it's a pretty 
obvious thing to do when you look at what is trying to be proven.  We've 
been doing this for ages too (since v1 in 2007).



Also, I mentioned greylisting before Evan Harris's paper.  It's here:

http://groups.google.com/group/comp.mail.sendmail/msg/ef7624049e1193a2?hl=en



Sure - credit where it is due; I've you to the 'Thanks' section.

Cheers,
Steve.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-02-08 Thread David F. Skoll
On Tue, 08 Feb 2011 17:04:37 +
Steve Freegard st...@stevefreegard.com wrote:

 Sure - credit where it is due; I've you to the 'Thanks' section.

Thanks.  And also, my apologies for posting to the list... that was supposed
to be a private message. :(

/me mutters something about email amateurs not understanding how email works...

Regards,

David.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Rolf E. Sonneveld

On 1/19/11 2:10 AM, David F. Skoll wrote:

On Tue, 18 Jan 2011 23:37:07 +0100
Rolf E. Sonneveldr.e.sonnev...@sonnection.nl  wrote:


I agree with you, looking at my own personal situation. However, many
mail admins (and maybe you too) are responsible for the e-mail
handling of many (tens/hundreds/thousands) of users. Most users have
unrealistic expectations of e-mail delivery times, and become
impatient when an e-mail they send is not delivered after a few
minutes. We can tell them they should not have these expectations,
but they just have them. User education is a tough task. How many
phone calls start with 'Hiname, how are you, did you receive my
mail?'.

User education can go a long way.  One of our (very large) customers
filters mail for about 900 000 email addresses and uses greylisting.
They've obviously decided the benefits outweigh the costs.

[...]


I wish everyone, using greylisting, would do what you did. That sure
would reduce collateral damage! I have a question about your setup:
do you automatically greylist senders to whom you sent mail the last
6 months?

We whitelist those senders...


Excuse me, I meant to say: whitelist instead of greylist.


If so, do you differentiate between interpersonal messages
(legit mail from you to that sender) and out-of-office type messages
(which can be the result of spam messages and can carry your mail
address, depending on what type of mail system you use)?

We attempt to.  We look for the standard Auto-Submitted header which
good auto-responders add.  And we use heuristics to try to catch the
crappy auto-responders (typically, any MTA made by a large company like
Microsoft or IBM qualifies as crappy) though those are not completely
accurate.

Another thing we could do in principle but don't currently is share data
about which machines pass greylisting.


Interesting idea... It would probably help to further minimize 
collateral damage for those organizations that can use this data.


Regards,
/rolf


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Lee Dilkie
I recently gave up on greylisting after using it for years as well.

Two reasons really, one was the complaints from users (and I found that
they often asked folks to send mail to me twice to try and get mail to
work better and that was just embarrassing).

The second was that I've found that the other spam-catching filtering is
doing a much better job than it was years ago and turning off
greylisting didn't adversely affect the amount of spam that got through.

-lee


On 1/18/2011 5:41 PM, Warren Togami Jr. wrote:
 On 01/18/2011 12:31 PM, David F. Skoll wrote:
 On Tue, 18 Jan 2011 22:18:20 +
 Gary Forrestga...@netnorth.co.uk  wrote:

 Interesting 2 of our 3 scanning heads use a grey list system that
 uses /32 addresses as part of the process, these two servers have
 100's of emails delayed for well over a day. Our 3rd scanning head
 uses a grey list system that is less granular /24 ,  this does not.

 Ah, I should mention that we use a /24 for greylisting for IPv4 and a
 /64 for IPv6.  On the other hand, we also add a hash of the subject
 into the greylisting tuple so it becomes:

 I recently gave up entirely on greylisting after:

 * Last week I discovered /24 was not good enough for redelivery
 attempts at one major ISP.  All mail from that ISP was failing for the
 past month except in rare cases where randomly the same /24 attempted
 delivery within the time window.

 * Years of complaints of mail delivery delays or failures from my
 users.  They had began creating gmail accounts in order to bypass. 
 They kept running into too many cases of broken individual mail
 servers (major companies!) who failed to redeliver.

 Users don't care about so and so is violating RFC-XXX.  They are
 trying to get business done and it was simply causing too many problems.

 Warren


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread David F. Skoll
On Wed, 19 Jan 2011 09:56:47 -0500
Lee Dilkie l...@dilkie.com wrote:

 The second was that I've found that the other spam-catching filtering
 is doing a much better job than it was years ago and turning off
 greylisting didn't adversely affect the amount of spam that got
 through.

That's possibly true, but look at this.

A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

On a busy system, this can make a huge difference.  SpamAssassin scanning
is by no means cheap.

Regards,

David.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Lee Dilkie

On 1/19/2011 10:02 AM, David F. Skoll wrote:
 On Wed, 19 Jan 2011 09:56:47 -0500
 Lee Dilkie l...@dilkie.com wrote:

 The second was that I've found that the other spam-catching filtering
 is doing a much better job than it was years ago and turning off
 greylisting didn't adversely affect the amount of spam that got
 through.
 That's possibly true, but look at this.

 A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
 A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

 On a busy system, this can make a huge difference.  SpamAssassin scanning
 is by no means cheap.

 Regards,

 David.

Agreed there, I did have to install the compiled regex package to get SA
speeds up enough to handle the increased load (my server is not even
close to yours in performance but I did drop SA time from 10-30s to 3s).

Don't get me wrong, I liked GL but there are a number of big ISPs that
have quite long retry timeouts (for some reason, sympatico comes to
mind) and it got to be too annoying.

who knows, all the code is still there and I might switch it on again in
the future.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread John Hardin

On Wed, 19 Jan 2011, Lee Dilkie wrote:

Don't get me wrong, I liked GL but there are a number of big ISPs that 
have quite long retry timeouts (for some reason, sympatico comes to 
mind) and it got to be too annoying.


...and when you encounter a big ISP that does this, do you notify their 
postmaster so they can fix the problem?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Taking my gun away because I *might* shoot someone is like cutting
  my tongue out because I *might* yell Fire! in a crowded theater.
  -- Peter Venetoklis
---
 4 days until John Moses Browning's 156th Birthday


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Matt
 The legitimate mail that passes through my mail server comes from
 hosts / networks I might not hear from again for months, by which
 time I have to potentially wait 24 hours for the greylisting / mail
 server to try again.

I run greylisting on an email server with several thousand email
accounts and think its great.  Reduced system load drastically.  I
also have a perl script I have wrote that runs every minute and looks
at all messages that arrived in last 60 seconds and if spamassassin
gave them a score of less then 5 it adds the sending MTA to a
whitelist.  It also removes any from the whitelist that have sent a
message that scored over 5.  The whitelist goes back 6 months and is
continually refreshed by the script.  The whitelist typically has 40K
IP's in it.

As a result no one even notices the greylisting, AFAIK...


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Ted Mittelstaedt

On 1/19/2011 9:25 AM, Matt wrote:

The legitimate mail that passes through my mail server comes from
hosts / networks I might not hear from again for months, by which
time I have to potentially wait 24 hours for the greylisting / mail
server to try again.


I run greylisting on an email server with several thousand email
accounts and think its great.  Reduced system load drastically.  I
also have a perl script I have wrote that runs every minute and looks
at all messages that arrived in last 60 seconds and if spamassassin
gave them a score of less then 5 it adds the sending MTA to a
whitelist.  It also removes any from the whitelist that have sent a
message that scored over 5.  The whitelist goes back 6 months and is
continually refreshed by the script.  The whitelist typically has 40K
IP's in it.

As a result no one even notices the greylisting, AFAIK...


Is that the greylist whitelist or the SA whitelist?

Ted


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Matt
 The legitimate mail that passes through my mail server comes from
 hosts / networks I might not hear from again for months, by which
 time I have to potentially wait 24 hours for the greylisting / mail
 server to try again.

 I run greylisting on an email server with several thousand email
 accounts and think its great.  Reduced system load drastically.  I
 also have a perl script I have wrote that runs every minute and looks
 at all messages that arrived in last 60 seconds and if spamassassin
 gave them a score of less then 5 it adds the sending MTA to a
 whitelist.  It also removes any from the whitelist that have sent a
 message that scored over 5.  The whitelist goes back 6 months and is
 continually refreshed by the script.  The whitelist typically has 40K
 IP's in it.

 As a result no one even notices the greylisting, AFAIK...

 Is that the greylist whitelist or the SA whitelist?

It whitelists those IP's from being greylisted in future.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-19 Thread Ted Mittelstaedt

On 1/19/2011 8:06 AM, Lee Dilkie wrote:


On 1/19/2011 10:02 AM, David F. Skoll wrote:

On Wed, 19 Jan 2011 09:56:47 -0500
Lee Dilkiel...@dilkie.com  wrote:


The second was that I've found that the other spam-catching filtering
is doing a much better job than it was years ago and turning off
greylisting didn't adversely affect the amount of spam that got
through.

That's possibly true, but look at this.

A greylisted message: mimedefang[17175]: p0I4xvRE017628: Filter time is 85ms
A scanned message:mimedefang[17175]: p0I50ACP017683: Filter time is 906ms

On a busy system, this can make a huge difference.  SpamAssassin scanning
is by no means cheap.

Regards,

David.


Agreed there, I did have to install the compiled regex package to get SA
speeds up enough to handle the increased load (my server is not even
close to yours in performance but I did drop SA time from 10-30s to 3s).

Don't get me wrong, I liked GL but there are a number of big ISPs that
have quite long retry timeouts (for some reason, sympatico comes to
mind) and it got to be too annoying.



In our experience the large ISPs don't have long retry timeouts.  What 
they have are multiple outbound mailservers.  They will try from 1 
server and when they get the error 4xx they shift the outbound message 
to another server.  This happens until all of their outbound servers

have been greylisted for the one message, then it goes through.

In my opinion (we use greylist-milter) the greylist developers are
basically their own worst enemies here.  They distribute a list of known
ISPs that round-robin outbound mail.  But the list is very old and
isn't a quarter of the ones that actually do it.

So an admin inexperienced with greylisting installs it and gets the 
experience your relating and then assumes that greylisting is no good.


Note that I am not assuming your inexperienced or that you don't already
know all about this problem.  You just didn't mention it so I didn't
want others who might come across this posting who might not be 
experienced with this to not know about it.


In our case greylisting is very cheap on CPU cycles.  But the regex 
matching and virus filtering is quite expensive.  And worse, we have

to pass everything to the users including the spam that we have tagged,
so we cannot do the logical thing and put SA first and then just throw
away from further processing everything that is determined to be spam.
Instead we have to put the virus filtering first (because we are allowed
to toss virus-infected mail) which means everything gets both scanned
for spam (except viruses) and virus filtered.

So we prefilter with greylist-milter and it really does indeed 
tremendously reduce the load on the server.  But you really do have to

explain to your users what is going on for it to work, and you
have to thoroughly investigate every mail complaint to make sure that
it's not caused by a round-robin ISP that you don't have in your
exception list.  And you have be alert for hosts like
craigslist.org because those bastards
fail mail delivery EVEN IF they get an error 4xx in violation of
the RFCs.

Ted


who knows, all the code is still there and I might switch it on again in
the future.




Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread Rolf E. Sonneveld

On 1/18/11 4:58 PM, David F. Skoll wrote:

On Tue, 18 Jan 2011 16:55:42 +0100
Giles Coocheygi...@coochey.net  wrote:


The legitimate mail that passes through my mail server comes from
hosts / networks I might not hear from again for months, by which
time I have to potentially wait 24 hours for the greylisting / mail
server to try again.

My point is I've never encountered a client that waits 24h to retry.
Most retry within minutes and the longest I've seen is maybe 4h.


RFC821/RFC2821/RFC5321 points out that a client has to wait a minimum of 
30 minutes before a retry attempt should be made, unless the client is 
able to determine what the reason is for the delay:



The sender MUST delay retrying a particular destination after one
attempt has failed.  In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.


As greylisting has never been standardized, there is no way for the 
client to reliably determine after what interval a retry should be made.


Can you document that 'Most retry within minutes'? My experience is that 
greylisting is causing real collateral damage due to the fact that many 
MTA's use long retry intervals, at least longer than a few minutes*): 
people get confused because their postings to mailing lists are delayed 
while the discussion on the list goes on; people give up trying to 
register themselves with sites, which send a confirmation message to the 
customer: the customer is waiting for the confirmation mail and gives up 
after a few minutes.


But I think you're right: I've yet to see the first MTA that waits for 
24 hours before the first retry is done.


/rolf

*) The default retry interval for Postfix is 20 minutes, the default 
retry interval for Sendmail is 1 hour, the default for Exchange is 
(dependent on the role) 10 minutes or 30 minutes, default of Sun/Oracle 
Messaging Server is 30 minutes or more, etc. etc.




Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread David F. Skoll
On Tue, 18 Jan 2011 22:18:33 +0100
Rolf E. Sonneveld r.e.sonnev...@sonnection.nl wrote:

 RFC821/RFC2821/RFC5321 points out that a client has to wait a minimum
 of 30 minutes before a retry attempt should be made,

That's fine.  I don't care if an email from someone I've never heard
from before is delayed 30 minutes or even a couple of hours.

 Can you document that 'Most retry within minutes'?

I analyzed 1655 machines from my logs that successfully passed greylisting.
The mean retry time was about 13000 seconds, or about 3.6 hours.

The standard deviation was high at 45000 seconds (12.5 hours).

That being said, 1390 of the 1655 machines retried within 4 hours and
1586 of 1655 retried within a day.  Every one of the machines that took
longer than a day was actually a spam box that only accidentally passed
greylisting when it tried sending a second spam using the same sender
address.

 My experience is that greylisting is causing real collateral damage
 due to the fact that many MTA's use long retry intervals, at least
 longer than a few minutes*): people get confused because their
 postings to mailing lists are delayed while the discussion on the
 list goes on; people give up trying to register themselves with
 sites, which send a confirmation message to the customer: the
 customer is waiting for the confirmation mail and gives up after a
 few minutes.

Our system notices when a host retries and turns off greylisting for 40 days
for that host.  That can greatly reduce the collateral damage.  It also
won't greylist senders to whom I've sent mail within the last 6 months.

For me, greylisting is so cheap and so effective I'm willing to live with
the small amount of colateral damage it introduces.

Regards,

David.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread Gary Forrest

Hi All

To answer David's post, extract from our scanning system for today.

*Jan 18 01:53:19 sendmail[8404]: p0I1rIDg008404: 
from=debenhams5-boun...@shopdebenhams.com, size=43048, class=0, 
nrcpts=1, msgid=debenh...@shopdebenhams.com, proto=ESMTP, 
daemon=MTA, relay=revd138.shopdebenhams.com [195.154.153.138]
Jan 18 01:53:19 sendmail[8404]: p0I1rIDg008404: Milter add: header: 
X-Greylist: Delayed for 117:43:55 by milter-greylist-4.0.1 
(avs3.netnorth.co.uk [82.148.225.24]); Tue, 18 Jan 2011 01:53:19 + (GMT)


that's a greylist delay of **117:43:55* - almost 5 days !

Interesting 2 of our 3 scanning heads use a grey list system that uses 
/32 addresses as part of the process, these two servers have 100's of 
emails delayed for well over a day. Our 3rd scanning head uses a grey 
list system that is less granular /24 ,  this does not.



Gary


*
*On 18/01/2011 15:58, David F. Skoll wrote:

On Tue, 18 Jan 2011 16:55:42 +0100
Giles Coocheygi...@coochey.net  wrote:


The legitimate mail that passes through my mail server comes from
hosts / networks I might not hear from again for months, by which
time I have to potentially wait 24 hours for the greylisting / mail
server to try again.

My point is I've never encountered a client that waits 24h to retry.
Most retry within minutes and the longest I've seen is maybe 4h.

Regards,

David.





 Message Scanning REF:470-avs3-1295366379 



--

|Gary Forrest
|(Director)
|Email: ga...@netnorth.co.uk
|Tel: 0845 058 2001
|Fax: 01204 900719
|
|Netnorth Limited
|Units 7 and 8 Queensbrook
|Bolton Technology Exchange
|Spa Road
|Bolton
|BL1 4AY
|
|Sales queries:  sa...@netnorth.co.uk
|Domain name queries: doma...@netnorth.co.uk
|Support queries: supp...@netnorth.co.uk
|Accounts queries: accou...@netnorth.co.uk



Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread David F. Skoll
On Tue, 18 Jan 2011 22:18:20 +
Gary Forrest ga...@netnorth.co.uk wrote:

 Interesting 2 of our 3 scanning heads use a grey list system that
 uses /32 addresses as part of the process, these two servers have
 100's of emails delayed for well over a day. Our 3rd scanning head
 uses a grey list system that is less granular /24 ,  this does not.

Ah, I should mention that we use a /24 for greylisting for IPv4 and a
/64 for IPv6.  On the other hand, we also add a hash of the subject
into the greylisting tuple so it becomes:

   {sender_address, recipient_address, sender_ip/24, subject_hash}

because we found a number of spammers bypassing greylisting but mutating
the message subject each time.

Regards,

David.


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread Rolf E. Sonneveld

On 1/18/11 11:02 PM, David F. Skoll wrote:

On Tue, 18 Jan 2011 22:18:33 +0100
Rolf E. Sonneveldr.e.sonnev...@sonnection.nl  wrote:


RFC821/RFC2821/RFC5321 points out that a client has to wait a minimum
of 30 minutes before a retry attempt should be made,

That's fine.  I don't care if an email from someone I've never heard
from before is delayed 30 minutes or even a couple of hours.


I agree with you, looking at my own personal situation. However, many 
mail admins (and maybe you too) are responsible for the e-mail handling 
of many (tens/hundreds/thousands) of users. Most users have unrealistic 
expectations of e-mail delivery times, and become impatient when an 
e-mail they send is not delivered after a few minutes. We can tell them 
they should not have these expectations, but they just have them. User 
education is a tough task. How many phone calls start with 'Hi name, 
how are you, did you receive my mail?'.



Can you document that 'Most retry within minutes'?

I analyzed 1655 machines from my logs that successfully passed greylisting.
The mean retry time was about 13000 seconds, or about 3.6 hours.

The standard deviation was high at 45000 seconds (12.5 hours).

That being said, 1390 of the 1655 machines retried within 4 hours and
1586 of 1655 retried within a day.  Every one of the machines that took
longer than a day was actually a spam box that only accidentally passed
greylisting when it tried sending a second spam using the same sender
address.


Thanks for sharing these figures, they're really useful even though the 
standard deviation is high.



My experience is that greylisting is causing real collateral damage
due to the fact that many MTA's use long retry intervals, at least
longer than a few minutes*): people get confused because their
postings to mailing lists are delayed while the discussion on the
list goes on; people give up trying to register themselves with
sites, which send a confirmation message to the customer: the
customer is waiting for the confirmation mail and gives up after a
few minutes.

Our system notices when a host retries and turns off greylisting for 40 days
for that host.  That can greatly reduce the collateral damage.  It also
won't greylist senders to whom I've sent mail within the last 6 months.


I wish everyone, using greylisting, would do what you did. That sure 
would reduce collateral damage! I have a question about your setup: do 
you automatically greylist senders to whom you sent mail the last 6 
months? If so, do you differentiate between interpersonal messages 
(legit mail from you to that sender) and out-of-office type messages 
(which can be the result of spam messages and can carry your mail 
address, depending on what type of mail system you use)?



For me, greylisting is so cheap and so effective I'm willing to live with
the small amount of colateral damage it introduces.


The right anti-spam defense is always a balance between catching as much 
spam as possible and avoiding false positives (and other collateral 
damage) as much as possible. IMHO too many mail admins focus on the 
former, forgetting about the latter. The net result is that the 
reliability of Internet e-mail system has degraded over the last 15 years.


/rolf


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread Warren Togami Jr.

On 01/18/2011 12:31 PM, David F. Skoll wrote:

On Tue, 18 Jan 2011 22:18:20 +
Gary Forrestga...@netnorth.co.uk  wrote:


Interesting 2 of our 3 scanning heads use a grey list system that
uses /32 addresses as part of the process, these two servers have
100's of emails delayed for well over a day. Our 3rd scanning head
uses a grey list system that is less granular /24 ,  this does not.


Ah, I should mention that we use a /24 for greylisting for IPv4 and a
/64 for IPv6.  On the other hand, we also add a hash of the subject
into the greylisting tuple so it becomes:


I recently gave up entirely on greylisting after:

* Last week I discovered /24 was not good enough for redelivery attempts 
at one major ISP.  All mail from that ISP was failing for the past month 
except in rare cases where randomly the same /24 attempted delivery 
within the time window.


* Years of complaints of mail delivery delays or failures from my users. 
 They had began creating gmail accounts in order to bypass.  They kept 
running into too many cases of broken individual mail servers (major 
companies!) who failed to redeliver.


Users don't care about so and so is violating RFC-XXX.  They are 
trying to get business done and it was simply causing too many problems.


Warren


Re: Greylisting delay (was Re: Q about short-circuit over ruling blacklisting rule)

2011-01-18 Thread David F. Skoll
On Tue, 18 Jan 2011 23:37:07 +0100
Rolf E. Sonneveld r.e.sonnev...@sonnection.nl wrote:

 I agree with you, looking at my own personal situation. However, many 
 mail admins (and maybe you too) are responsible for the e-mail
 handling of many (tens/hundreds/thousands) of users. Most users have
 unrealistic expectations of e-mail delivery times, and become
 impatient when an e-mail they send is not delivered after a few
 minutes. We can tell them they should not have these expectations,
 but they just have them. User education is a tough task. How many
 phone calls start with 'Hi name, how are you, did you receive my
 mail?'.

User education can go a long way.  One of our (very large) customers
filters mail for about 900 000 email addresses and uses greylisting.
They've obviously decided the benefits outweigh the costs.

[...]

 I wish everyone, using greylisting, would do what you did. That sure 
 would reduce collateral damage! I have a question about your setup:
 do you automatically greylist senders to whom you sent mail the last
 6 months?

We whitelist those senders...

 If so, do you differentiate between interpersonal messages 
 (legit mail from you to that sender) and out-of-office type messages 
 (which can be the result of spam messages and can carry your mail 
 address, depending on what type of mail system you use)?

We attempt to.  We look for the standard Auto-Submitted header which
good auto-responders add.  And we use heuristics to try to catch the
crappy auto-responders (typically, any MTA made by a large company like
Microsoft or IBM qualifies as crappy) though those are not completely
accurate.

Another thing we could do in principle but don't currently is share data
about which machines pass greylisting.  We have a reputation-reporting system
that reports on various events, and two of the events it reports are
Machine X was greylisted and Machine X passed greylisting.  We could
publish a DNS zone that you could look up and decide not to greylist
a given machine.

Right now, we have event data on 15.9 million IPv4 addresses.  Of those, about
7.1 million have never passed greylisting (which means we know of about
8.8 million machines that are probably pointless to greylist.)

Regards,

David.