Gilles Thanks for the reply -- have trimmed out some bit and answered your questions.
> We send a lot of newsletters to mail systems like GMail, Hotmail and Yahoo > etc but Hotmail and Yahoo provide particular problems. As such we have been > playing with some of the obscure options which only seem to get mentioned in > limits.c e.g. > > limit mta for domain hotmail.com inet4 session-mail-max 10 max-conn-per-host > 40 max-conn-per-route 20 max-conn-per-source 200 max-conn-per-connector 40 > max-conn-per-relay 400 max-conn-per-domain 400 session-transaction-delay 0 > > This is very likely to get you blacklisted very fast, you're essentially > telling OpenSMTPD to blast the hotmail servers by making tons of connexions > and by not introducing any delay between your transactions. While this might > work for a little while or with little volume, they're going to give your IP > addresses and domain a very bad reputation. > I am not completely sure this is a problem. We have historically used much bigger numbers that this without an issue. We had a a busy day yesterday so I turned this into a default configuration and had no particular problems (except for one machine). > > The problem to be solved is how these services create temporary failures -- > this is the service telling you to go away and then seeing when you come back > > > Actually it is a lot more complex than that, OpenSMTPD keeps a routing table > and keeps track of failures for every route it has managed to establish. > It can even differentiate between different kind of failures (no MX > available, MX are available but do not accept mail, MX has started to produce > many errors in a row, ...). > When a provider produces many temporary failures in a row, OpenSMTPD will > assume the MX is having a temporary failure and will mark it unavailable for > a while using a quadratic delay strategy + a penalty on the envelopes that > were temporary failed so that they aren't retried in the same order in case > this was just a coincidence. > And the standard quadratic back off maximum delay is four hours. To compare with old zmailer (http://www.zmailer.org/man/scheduler.8zm.html), we could set retry intervals (e.g. interval and retries -- see the examples). I can understand the reasoning behind quadratic but in the worst case scenario, it will not detect that a remote system is back for four hours. I guess my question is do I have a way of making OpenSMTPD put an MX into temporary failure sooner rather than later. Also the 'different order' can cause interesting problems -- newer emails in the queue overtaking older ones. > Similarly Yahoo and others will stop accepting connections if they think you > are sending too many messages. At this point, OpenSMTPD backs away and the > quadratic back off cuts in which increases the retry time. Unfortunately > these services will accept messages in a finite time. > > > I don't quite get the "accept messages in a finite time" part. Yahoo doesn't seem to mind being contacted later -- it seems to be that OpenSMTPD backs off quadratically while Yahoo (and others such as Mimecast) don't mind being retried in a finite time (e.g. 15 mins). > > To get round these problems, we have been playing with 'smtpctl schedule' and > restarting the server. In particular, with Yahoo we can see this behaviour: > > * messages not being delivered; > * schedule the messages -- still not being delivered; > * stop and start the daemon; > * schedule the messages and they are delivered; > > Problem with this is that the routing information is runtime information > which gets lost across a restart. > If you're having problems with the default limits, we should really > understand what is the problem, what limits should be used and then make them > the default. > I completely agree and am happy to try things. Again to back into ancient history, the zmailer boilerplate scheduler.conf file used to have examples of different strategies (which are mentioned in the manual page above and also on the old zmailer website). > > A few other observations: > > * we use a config line like this to handle returns: > > accept from any for domain "domain.com" virtual { "@" => nobody } deliver to > mda "mail-handler.pl" > > Mapping all virtual addresses to the user nobody took us a few minutes to > find. What would be useful (but probably only to us) would be to have a > regexp to define the email addresses. This is because there are a lot of > envelope addresses which are in a standard form and a few recipients. We > could use the MySQL table option but that might be overkill. > > > This is a good idea, you should fill a feature request on our bug tracker or > this will be lost/forgotten > > I will add it RSN and might look at adding it -- I have used pcre code before. > > * it might also be useful to allow wildcards in 'limit mta domain > hotmail.com' as there are many hotmails etc. > > Same as above, just keep in mind that there should really be no reason to > tweak limits as we're supposed to ship proper limits that work for 99% of > people. > The problems you mentioned above should be covered and if they aren't then we > have a bug we have to sort out, you shouldn't have to deal with limits. > > Again will do -- I did also note that I can set system wide limits by removing [domain etc]. I guess also there ought to be a feature request to put these config options in the manual page (not everyone wants to read limit.c). > Any help, gratefully received -- particularly if it stops me from having to > dive further into the code! > > > What would really help us help you is to provide some figures: > > - what version of OpenSMTPD are you using ? I think we are on opensmtpd-201310101759p1 on all machines. I take the approach of downloading the most current version, try it on one machine and then push it to others. We are running on CentOS 6. > - what kind of volumes are you sending ? About 100K per weekday -- less at weekends. > - how many sources are you using ? Do you mean how many machines? Can I be evasive here and say a few. > - over how much time are you sending these volumes ? > It tends to be very bursty -- people like their newsletters to land very soon after they push the send button. > We know of systems that use the default limits to exchange hundreds of > thousands of mails with gmail / yahoo / hotmail daily, I'm curious about what > could break your setup :-) > Gmail is not an issue -- it happily swallows everything we send at it. Hotmail and Yahoo are still seeing some issues. Mimecast 'grey listing' remains a problem but that might also the fact we cannot use SPF or DKIM with that client maybe a contributing factor. Mimecast can be solved by the 'stop, start and reschedule' which satisfies their grey listing. TIA Simon.
signature.asc
Description: Message signed with OpenPGP using GPGMail
