Re: relay server - mass mailing tuning

[Philip Paeps]

On 2017-04-11 14:04:08 (-0400), Viktor Dukhovni  
wrote:

On Apr 11, 2017, at 1:55 PM, Philip Paeps  wrote:
It is worth repeating that the spinning rust actually matters in this 
case: Postfix fsync()s when accepting a message into the queue.  The 
time to it takes to enqueue a message is at least the time it takes to 
write it to disk, not simply the time it takes to hit the buffer 
cache.


For high-volume senders, the main problem is usually getting the mail *out*
faster with less friction from the receiving systems.  Accepting mail faster
is not necessarily a good idea, if that just means a bigger backlog.


I'm aware of that. :)  I merely wanted to remind the OP (and anyone 
stumbling across this thread in the archives) that the time it takes to 
enqueue mail should not be underestimated.  Many common workloads don't 
have to care too much about disk latency because they're protected by 
the buffer cache.  This is not the case when handling large volumes of 
mail.


Ideally you want to be able to get mail out as fast as it's coming in or 
at least not very much slower but you don't have complete control over 
that (sites you need to deliver to can go down or refuse to take mail 
from you for a while, etc).


If you know you need to relay half a million messages in a short period 
of time weighing about 100Gbytes in total, you need to be prepared to 
enqueue an appreciable fraction of that volume.


The OPs efforts should go towards understanding what it takes to send 
email at the intended volume.  Receiving it fast enough is a secondary 
issue.


It sounds like the OP is largely unprepared for the task at hand.  This 
requires real experience and dedicated effort.  Just asking for a few 
ideas on this list is unlikely to be sufficient.


I agree.

A sane approach is to start small, grow gradually, and solve small 
problems as they come up, before they become big problems.  Going from 
nothing to sending a very large volume of mail is not easy.


At the very least, the OP should spend what little time he has before he 
needs to handle this load becoming as familiar as possible with the 
TUNING_README and probably also QSHAPE_README.


That will hopefully prepare him enough to balance the rate of mail 
coming in against the rate he can get it out.  It still might not be 
very pretty though.


Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information

Re: relay server - mass mailing tuning

[Viktor Dukhovni]

> On Apr 11, 2017, at 1:55 PM, Philip Paeps  wrote:
> 
> It is worth repeating that the spinning rust actually matters in this case:
> Postfix fsync()s when accepting a message into the queue.  The time to it
> takes to enqueue a message is at least the time it takes to write it to
> disk, not simply the time it takes to hit the buffer cache.

For high-volume senders, the main problem is usually getting the mail *out*
faster with less friction from the receiving systems.  Accepting mail faster
is not necessarily a good idea, if that just means a bigger backlog.

The OPs efforts should go towards understanding what it takes to send email
at the intended volume.  Receiving it fast enough is a secondary issue.

It sounds like the OP is largely unprepared for the task at hand.  This
requires real experience and dedicated effort.  Just asking for a few ideas
on this list is unlikely to be sufficient.

A sane approach is to start small, grow gradually, and solve small problems
as they come up, before they become big problems.  Going from nothing to
sending a very large volume of mail is not easy.

-- 
Viktor.

Re: relay server - mass mailing tuning

[Philip Paeps]

On 2017-04-11 10:02:21 (-0400), Wietse Venema  wrote:

Fazzina, Angelo:
I would think they would have to tell you the volume and or rate at 
which they intend to send the mail.


He mentioned message count and size, about 100GB of traffic, so that 
might take a while, especially when the queue is on spinning disks.


It is worth repeating that the spinning rust actually matters in this 
case: Postfix fsync()s when accepting a message into the queue.  The 
time to it takes to enqueue a message is at least the time it takes to 
write it to disk, not simply the time it takes to hit the buffer cache.


People occasionally forget that Postfix actually tries to be reliable in 
addition to fast. :)


Mounting /var/spool on an SSD is definitely a good idea.  Ideally a 
mirrored pair of SSDs for reliability.


If you're running on ZFS, you want your ZIL on fast SSDs (again: plural, 
and remember to mirror them, by default ZFS load-balances log devices).


(If you don't care about reliability, mount /var/spool on a tmpfs.  Or 
build Postfix without HAS_FSYNC.  If it breaks you get to keep the 
pieces though.)



- If the sender's DNS setup is borked, Postfix will lose time doing
DNS lookup for the SMTP client name/address.


Running a local resolver (e.g. Unbound) can mitigate most of these 
issues.  Also be sure to check that the TTL of the SMTP client's DNS 
name / zone are configured sensibly.  If it's got a TTL of 5 seconds, 
you can do all the caching you like and you'll still be doing a lot of 
waiting for DNS.  (This can be mitigated with the cache-min-ttl setting 
in Unbound).


Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information

Re: relay server - mass mailing tuning

[Wietse Venema]

Fazzina, Angelo:
> Hi,
> I would think they would have to tell you the volume and or rate
> at which they intend to send the mail.

He mentioned message count and size, about 100GB of traffic, so that 
might take a while, especially when the queue is on spinning disks.

> By default Postfix lets the queue grow to 20,000 emails I think,
> and then may stop accepting until it can process some, and start
> accepting more.

That is incorrect. Postfix keeps accepting mail until queue_minfree
limit is reached. That happens when mail arrives faster than Postfix
can deliver it.

I suggest that the original poster does some experiments with
smtp-source and smtp-sink, to find out where the bottlenecks are.

Things to keep in mind:

- Multiple parallel inbound connections may get the mail faster
into Postfix, but that also overloads the file system and prevents
Postfix from delivering mail. There is an optimal input volume that
depends on file system performance.

- By default, smtpd_client_connection_count_limit is set to half
of the default process limit, so that is something that the senders
need to be aware of.

- If the sender's DNS setup is borked, Postfix will lose time doing
DNS lookup for the SMTP client name/address.

Wietse

RE: relay server - mass mailing tuning

[Fazzina, Angelo]

Hi,
I would think they would have to tell you the volume and or rate at which they 
intend to send the mail.
By default Postfix lets the queue grow to 20,000 emails I think, and then may 
stop accepting until it can process some, and start accepting more.

Any reason you haven’t been reading this?
http://www.postfix.org/TUNING_README.html


And making some educated guesses ?  You know the total volume 300-500K. all you 
care about is Rate, how fast are they going to send them ?
Good Luck.
-ALF

-Angelo Fazzina
Operating Systems Programmer / Analyst
University of Connecticut,  UITS, SSG, Server Systems
860-486-9075

From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of Zalezny Niezalezny
Sent: Tuesday, April 11, 2017 4:24 AM
To: Postfix users 
Subject: relay server - mass mailing tuning

Hi,
in the next days our external service provider will push to our relay server 
(Postfix) regulary ~300 000 - 500 000 E-mails with size 60-500kb.
External system provider has a SMTP server farm, so it will send messages from 
~20 IP`s to our single Postfix instance.
My question is, do I need to tune anything in our Postfix relay to receive and 
transport such a big amount of messages in single session ?
What kind of settings should I setup to keep performance on the optimal level 
and to avoid situation with too many connections etc.etc.

Could You please advice here ?


Thanks in advance for any support.

With kind regards
Konrad Wawryn

relay server - mass mailing tuning

[Zalezny Niezalezny]

Hi,

in the next days our external service provider will push to our relay
server (Postfix) regulary ~300 000 - 500 000 E-mails with size 60-500kb.

External system provider has a SMTP server farm, so it will send messages
from ~20 IP`s to our single Postfix instance.

My question is, do I need to tune anything in our Postfix relay to receive
and transport such a big amount of messages in single session ?

What kind of settings should I setup to keep performance on the optimal
level and to avoid situation with too many connections etc.etc.


Could You please advice here ?



Thanks in advance for any support.


With kind regards

Konrad Wawryn