forgot to CC: misc@
---------- Forwarded message ----------
From: Gilles Chehade <gilles.cheh...@gmail.com>
Date: 2013/10/17
Subject: Re: Sending to Hotmail, Yahoo etc
To: Simon Brock <s...@widearea.co.uk>
Woops, I accidentally deleted this mail from my real mail account and just
bumped into it on my gmail account ;-)
Answers inlined and a few questions at the bottom:
2013/10/10 Simon Brock <s...@widearea.co.uk>
> Hi
> After being a happy user of zmailer for a long time, we decided to move
> our newsletter mailings to OpenSMTD. Zmailer was great when it was written
> and had some great features but has not been updated in a long time. The
> architecture of OpenSMTPD is very similar to Zmailer.
>
> We send a lot of newsletters to mail systems like GMail, Hotmail and Yahoo
> etc but Hotmail and Yahoo provide particular problems. As such we have been
> playing with some of the obscure options which only seem to get mentioned
> in limits.c e.g.
>
> limit mta for domain hotmail.com inet4 session-mail-max 10
> max-conn-per-host 40 max-conn-per-route 20 max-conn-per-source 200
> max-conn-per-connector 40 max-conn-per-relay 400 max-conn-per-domain 400
> session-transaction-delay 0
>
This is very likely to get you blacklisted very fast, you're essentially
telling OpenSMTPD to blast the hotmail servers by making tons of connexions
and by not introducing any delay between your transactions. While this
might work for a little while or with little volume, they're going to give
your IP addresses and domain a very bad reputation.
I am completely sure we have not got the right set of options but I am not
> completely sure the options we need exist. Before we started trying to make
> the change, we wanted to check we were doing the right thing.
>
The problem to be solved is how these services create temporary failures --
> this is the service telling you to go away and then seeing when you come
> back. For example, Hotmail and others will start producing temporary
> failures which is a hint to slow down or stop. It seems to me that
> OpenSMTPD doesn't take the hint but keeps trying other messages from the
> queue. At this point, Hotmail then thinks you are a 'bot' and characterises
> you as such in their SNDS service.
>
>
Actually it is a lot more complex than that, OpenSMTPD keeps a routing
table and keeps track of failures for every route it has managed to
establish.
It can even differentiate between different kind of failures (no MX
available, MX are available but do not accept mail, MX has started to
produce many errors in a row, ...).
When a provider produces many temporary failures in a row, OpenSMTPD will
assume the MX is having a temporary failure and will mark it unavailable
for a while using a quadratic delay strategy + a penalty on the envelopes
that were temporary failed so that they aren't retried in the same order in
case this was just a coincidence.
> Similarly Yahoo and others will stop accepting connections if they think
> you are sending too many messages. At this point, OpenSMTPD backs away and
> the quadratic back off cuts in which increases the retry time.
> Unfortunately these services will accept messages in a finite time.
>
>
I don't quite get the "accept messages in a finite time" part.
Yahoo will fire the per-recipient temporary failures after a certain number
of envelopes are submitted and unless you stop submitted for a while, it
will assume you're a spammer.
With the "MX has started to produce many errors in a row" strategy,
OpenSMTPD detects this and penalizes the route to avoid more submissions.
When the route is available again, envelopes are transmitted as long as we
don't receive a serie of temporary failure again.
To get round these problems, we have been playing with 'smtpctl schedule'
> and restarting the server. In particular, with Yahoo we can see this
> behaviour:
>
> * messages not being delivered;
> * schedule the messages -- still not being delivered;
> * stop and start the daemon;
> * schedule the messages and they are delivered;
>
Problem with this is that the routing information is runtime information
which gets lost across a restart.
If you're having problems with the default limits, we should really
understand what is the problem, what limits should be used and then make
them the default.
> A few other observations:
>
> * we use a config line like this to handle returns:
>
> accept from any for domain "domain.com" virtual { "@" => nobody } deliver
> to mda "mail-handler.pl"
>
> Mapping all virtual addresses to the user nobody took us a few minutes to
> find. What would be useful (but probably only to us) would be to have a
> regexp to define the email addresses. This is because there are a lot of
> envelope addresses which are in a standard form and a few recipients. We
> could use the MySQL table option but that might be overkill.
>
>
This is a good idea, you should fill a feature request on our bug tracker
or this will be lost/forgotten
> * it might also be useful to allow wildcards in 'limit mta domain
> hotmail.com' as there are many hotmails etc.
>
Same as above, just keep in mind that there should really be no reason to
tweak limits as we're supposed to ship proper limits that work for 99% of
people.
The problems you mentioned above should be covered and if they aren't then
we have a bug we have to sort out, you shouldn't have to deal with limits.
Any help, gratefully received -- particularly if it stops me from having to
> dive further into the code!
>
>
What would really help us help you is to provide some figures:
- what version of OpenSMTPD are you using ?
- what kind of volumes are you sending ?
- how many sources are you using ?
- over how much time are you sending these volumes ?
We know of systems that use the default limits to exchange hundreds of
thousands of mails with gmail / yahoo / hotmail daily, I'm curious about
what could break your setup :-)
Gilles
