On 11/11/2010 03:14 PM, Pedro Axelrud wrote:
Hello guys,
We have a startup with an email marketing web app, we are very rigid
with our clients to not allow spammers.
One of our client's wants to send a message to two million contacts
HOW will they be sending these messages ?
That's the most important factor.
If they try to maildrop 2M messages (or 2000 messages with 1000
recipients each) into your postfix server, without consultation with
you, things will get hairy.
If they wish to send a bulk mail to 2M recipients, you probably want to
tell them to implement a steady delivery mechanism on their end.
There's several ways that can go; you could require that they spread out
destinations over multiple messages, to spread the concurrency load.
Or the exact opposite: require them to group recipients by domain into
separate messages, so you can take advantage of the fact that postfix
will also deliver them as one message.
The right solution will probably be somewhere in the middle; but it
won't require you to process and queue 2 million messages.
It should be perfectly doable with about 20000 messages or so, each with
100 recipients.
and I am worried with some details. I am not worried about the time
that will take to send all those messages, but I am worried about
respecting the server's policies.
Today we use two dedicated servers as mailservers, each one with four
IPs. I want to make some benchmarks to check if we need more and how many.
We use different transports according to the domain, but what limits
it's sending rate is /transport_destination_rate_delay /that is a
fixed value, and also we use the
/transport_destination_concurrency_limit /with
/transport_concurrency_negative_feedback /and
/transport_concurrency_positive_feedback /to control the sending rates
according to recipient policies.
But I am worried that with this quantity of messages only this
restrictions wont be able to manage the recipient's policies and our
deliverability would be terrible.
That's what they are designed for, so normally, they should work fine.
However, deliverability isn't exactly under your control; bad or
obstinate MTAs may make trouble where there technically isn't any (a
well-known one would be Yahoo).
It helps markedly if you know this in advance and configure a
slower-delivering transport for these domains.
You say the time taken to deliver 2M messages isn't a factor - so why
risk deferrals or rejections by edging close to the (unknown) limits of
remote servers ?
Just throttle postfix back to deliver ~10 messages per second, and
slightly less than that amount concurrently per destination.
This would take less than an hour to deliver 20000 messages - that's
insignificant for most purposes.
One problem that postfix don't deal is when an IP is blocked by an
ISP, message rate should drop a lot while the IP does not get unblocked.
Yes it does. Postfix keeps tabs on undeliverable locations, and doesn't
try again unnecessarily.
Note that there is a big difference between *blocking* an IP, i.e. the
mail server is unreachable, and postfix temporarily marks it as such,
and *rejecting* messages, which means postfix can connect to the server
just fine, but receives a 4xx status code, which according to the RFCs
means postfix must queue the mail and try again later.
The latter incurs overhead, and may produce large queues with
misconfigured servers or bad recipient lists. The former has little impact.
Also when one IP gets greylisted it should stop sending to that domain
or send only a few messages while greylisted.
Again, most if this is auto-tuned.
As long as the mail is in your deferred queue it doesn't impact delivery
of subsequent messages - unless the deferred queue grows to unmanageable
sizes.
You may want to very specifically raise your minimal_backoff_time
setting so the first retry doesn't happen until, say, an hour later (the
default is 5 minutes).
This would spread out deferred messages (move them away from being
retried while still delivering original mail to that location).
I want to ask you if is there any optimization in the configs or any
postfix extensions that can help me deal with this.
Sure, there are a lot of settings; you may want to start here:
http://www.postfix.org/TUNING_README.html
There are source/sink benchmark tools at the end of that, allowing you
to simulate high volumes.
--
J.
