On 12.5.2013 0:33, Wietse Venema wrote:
Patrik Rak:
This largely solves the problem, and is much simpler to configure:
# Out of a total of $default_process_limit (100), leaving 20
# for fresh mail. Adjust appropriately when master.cf or
# default_process_limit are changed.
#
smtp_deferred_concurrency_limit = 80
Even simpler: stop reading the deferred queue when more than N% of
the maximal number of recipient slots is from deferred mail.
Hmm, I have spent quite more time thinking about all this while
gardening today. Let me put it like this:
I am very well aware of the performance postfix is able to achieve, so I
find it quite embarrassing how surprisingly little bogus recipients can
stall legitimate mail. Whatever the solution, postfix should know better
than that out of the box.
Can we agree on that?
Now we know of two problems which cause this:
A) Congested active queue, which can be currently solved either by
increasing its size considerably, or by passing the deferred mail to
separate server or instance,
B) Congested delivery agents, which happens much earlier and which can
be somewhat remedied by increasing the transport process limit
sufficiently, or again by using separate server or instance.
In both cases we are talking about congestion by mail coming from the
deferred queue here. While it can theoretically happen with new mail
from the incoming queue as well, we have already spent considerable
effort in the past to make sure that doesn't happen, so I dare to say
that that is not an issue anymore.
The proposals to solving these two problems we have heard so far are:
0) Using separate server or instance.
1) Limiting amount of deferred mail moved into the active queue.
2) Allowing delivery of mail from deferred queue via dedicated transport.
3) Dedicating some amount of delivery agents to new mail only.
Now, how do these fare compared to each other:
- The solution 0 solves both A and B problems, and is something I would
definitely recommend for any enterprise setup anyway. Unfortunately, it
doesn't work out of the box and requires nontrivial amount of work to
set up, so it is not a proper solution for the problem I described as such.
- The solution 1 solves problem A quite elegantly, and I believe we are
all in agreement that it would be worth implementing. It can be IMO
easily made the default setting as well.
Technically speaking, it could be used to solve problem B as well, by
making the deferred message limit smaller than the delivery agent
process limit, but imposing such a low limit would quite certainly
affect throughput of deferred mail, so it is definitely not something we
would want to make the default, so it is out of question as the out of
the box solution for problem B as well.
- The solution 2 doesn't address problem A at all. It merely solves
problem B. While I was quite glad to see that it would separate the
deferred mail from new mail entirely, it's true that its perhaps too
pervasive to be made out of the box default either.
- The solution 3 is similar to solution 2 in that it only solves problem
B. The advantage here is that it doesn't introduce new transport or
anything, which makes me feel it can be used quite safely for a default
setting as well.
So, from a summary like this, it seems to me that we would want both
solution 1 and 3 as that would solve both problems A and B out of the box.
Therefore, I suggest that I'll do my best to review the current
algorithm and see how solution 3 could be implemented. Every time I
tried before I concluded that separating the two as much as possible
would be best for the sake of simplicity, but today I was trying to see
the whole problem as yet another limit similar to concurrency window
applied to jobs originating from deferred messages only. And as qmgr has
quite sophisticated support for dealing with blocker jobs and getting
them out of the way already, perhaps it's not that difficult to try to
approach it this way in the end. I'll see...
Any objections?
Patrik
P.S. As I finish this, your most recent responses to my earlier mails
just hit my mailbox. Unfortunately I will have to read them a bit later,
so my apologies for not taking them into account yet in this mail.
