On Mon, May 13, 2013 at 06:55:12AM -0400, Wietse Venema wrote:
> Viktor Dukhovni:
> > The reasonable response to latency spikes is creating concurrency
> > spikes.
>
> By design, Postfix MUST be able to run in a fixed resource budget.
> Your on-demand concurrency spikes break this principle and will
> result in unexpected resource exhaustion.
No, there are two different process limits one for non-slow deliveries,
that protects exagainst excessive network concurrency, and another for
slow deliveries that protects against memory exhaustion. We can set the
blocked process limit to zero for backwards compatible process ceilings
if you like. That restores legacy behaviour.
> If you want to run more processes increase the process limit. Then,
> Postfix's resource budget can be validated with smtp-sink/source
> like tools.
I want to run more processes (up to a limit) when deliveries are
slow, than when they are not. Just increasing the process limit
while effective for the slow case, risks too much contention for
the fast case.
> Recall that the automatic response to smtpd overload was NOT running
> more smtpd processes. Instead, the solution was allowing them to
> run with shorter timeouts. That approach respects the "fixed budget"
> requirement.
>
> Please consider a "stress" like equivalent for delivery agents.
Is a shorter timeout wise here? With remote connections we don't
have any control over the connection rate, so we set lower idle
timeouts under stress.
With delivery agents, we have full control of the concurrency, so
there is no need to drop timeouts and we're never idle deliberately,
only blocked on DNS lookups and connection attempts, ...
So raising process concurrency to a higher ceiling is quite ok.
There is little risk of memory issues, each additional smtp(8)
consumes a trivial amount of RAM. The text pages are shared,
and the data footprint of smtp(8) is low.
Memory pressure based on smtp(8) delivery agent process count isnot
not an issue anymore. Each process uses O(100KB) of RAM. We can
run 1,000 delivery agents in 100MB of RAM, which is rather tiny
these days, and the default process limit is still 100, mostly
because 100 parallel outbound streams is reasonable from a network
viewpoint on bandwidth constrained links, which are still common.
If you feel that under stress smtp(8) connection timeouts should
be low, perhaps that's reasonable, but we have no control over DNS
timeouts and short HELO timeouts may be unwise, poorly connectied sites
may never get their mail from a loaded MTA.
In any case, we are at least talking about solving the right problem,
managing concurrency and latency of actual deliveries as they happen.
--
Viktor.
