qshape by V. Duchovni, with explanation of the various postfix queues
================================
This document describes the "qshape" program which helps the
administrator understand the Postfix queue message distribution
in time and by sender domain or recipient domain.
The "qshape" program was developed by Victor Duchovni of Morgan Stanley.
Purpose of this software
========================
When mail is draining slowly or the queue is unexpectedly large, run
"qshape" as the super-user (root) to help zero in on the problem.
Postfix queues:
===============
- The "maildrop" queue.
Messages that have been submitted via the "sendmail" command, but not
yet brought in the main Postfix queue by the "pickup" service await
processing the the "maildrop" queue. Messages can be added to the
"maildrop" queue even when the Postfix system is not running, they
will begin to be processed once Postfix is started.
The "maildrop" queue is drained by the single threaded "pickup"
service scanning the queue directory peridiocally or when notified
of new message arrival by the "postdrop" program. The "postdrop"
program is a setgid helper that allows the unprivileged "sendmail"
program to inject mail into the "maildrop" queue and to notify the
"pickup" service of its arrival.
All mail that enters the main Postfix queue does so via the "cleanup"
service. The cleanup service is responsible for envelope and header
rewriting, header and body regular expression checks, automatic bcc
recipient processing and guaranteed insertion of the message into the
Postfix "incoming" queue.
In the absense of excessive CPU consumption in "cleanup" header or
body regular expression checks or other software consuming all available
CPU resources, Postfix performance is disk I/O bound. The rate at which
the "pickup" service can inject messages into the queue is largely
determined by disk access times, since the "cleanup" service must commit
the message to stable storage before returning success. The same is
true of the "postdrop" program writing the message to the "maildrop"
directory.
As the pickup service is single threaded, it can only deliver one
message at a time at a rate that does not exceed the reciprocal disk
I/O latency (+ CPU if not negligible) of the cleanup service.
Congestion in this queue is indicative of an excessive local message
submission rate or perhaps excessive CPU consumption in the "cleanup"
service due to excessive "body_checks".
Note, that once the active queue is full, the cleanup service will
attempt to slow down message injection by pausing $in_flow_delay
for each message. In this case "maildrop" queue congestion may be a
consequence of congestion downstream, rather than a problem in its
own right.
Note also, that one should not attempt to deliver large volumes of mail
via the "pickup" service. High volume sites must avoid using content
filters that reinject scanned mail via "sendmail" and "postdrop".
A high arrival rate of locally submitted mail may be an indication
of an uncaught forwarding loop, or a run-away notification program.
Try to keep the volume of local mail injection to a moderate level.
The "postsuper -r" command can place selected messages into the
"maildrop" queue for reprocessing. This is most useful for resetting
any stale content_filter settings. Requeuing a large number of messages
using "postsuper -r" can clearly cause a spike in the size of the
"maildrop" queue.
- The "hold" queue.
The administrator can define "smtpd" access(5) policies, or "cleanup"
header/body checks that cause messages to be automatically diverted
from normal processing and placed indefinitely in the "hold" queue.
Messages placed in the "hold" queue stay there until the administrator
intervenes. No periodic delivery attempts are made for messages in the
"hold" queue. The postsuper(8) can be used to manually put messages
on "hold" on release them into the "deferred" queue.
Messages can potentially stay in the "hold" queue for a time in excess
of normal maximal queue lifetime (after which undelivered messages
are bounced back to the sender). If such "old" messages need to be
released from the "hold" queue, they should typically be moved into
the "maildrop" queue, so that the message gets a new timestamp and
is given more than 1 opportunity to be delivered. Messages that are
"young" can be moved directly into the "deferred" queue.
The "hold" queue plays little role in Postfix performance, and
monitoring of the "hold" queue is typically more closely motivated by
tracking spam and malware, than by performance issues.
- The "incoming" queue.
All new mail entering the Postfix queue is written by the "cleanup"
service into the "incoming" queue. New queue files are created owned
by the "postfix" user with an access bitmask (or mode) of 0600. Once
a queue file is ready for further processing the "cleanup" service
changes the queue file mode to 0700 and notifies the queue manager
of new mail arrival. The queue manager ignores incomplete queue files
whose mode is 0600, these are still being written by cleanup.
The queue manager scans the incoming queue bringing any new mail
into the "active" queue if the active queue resource limits have not
been exceeded. By default the active queue accomodates at most 20000
messages. Once the active queue message limit is reached, the queue
manager stops scanning the incoming (and deferred, see below) queue.
Under normal conditions the incoming queue is nearly empty (has only
mode 0600 files), with the queue manager able to import new messages
into the active queue as soon as they become available.
The incoming queue grows when the message input rate spikes above
the rate at which the queue manager can import messages into the
active queue. The main factor slowing down the queue manager is
transport queries to the trivial-rewrite service. If the queue
manager is routinely not keeping up, consider not using "slow"
lookup services (MySQL, LDAP, ...) for transport lookups or
speeding up the hosts that provide the lookup service.
The in_flow_delay parameter is used to clamp the input
rate when the queue manager starts to fall behind. The cleanup
service will pause for $in_flow_delay seconds before creating
a new queue file if it cannot obtain a "token" from the queue
manager.
Since the number of cleanup processes is limited in most cases by the
SMTP server concurrency, the input rate can exceed the output rate by
at most "SMTP connection count" / $in_flow_delay messages per second.
With a default process limit of 100, and an in_flow_delay of 1s,
the coupling is strong enough to limit a single run-away injector
to 1 message per second, but is not strong enough to deflect an
excessive input rate from many sources at the same time.
If a server is being hammered from multiple directions, consider
raising the in_flow_delay to 10 or even 30 seconds, but only if
the incoming queue is growing even while the active queue is not
full and the trivial-rewrite service is using a fast transport
lookup mechanism.
- The "active" queue.
The queue manager is a delivery agent scheduler, it works to
ensure fast and fair delivery of mail to all destinations within
designated resource limits.
The active queue is somewhat analogous to an operating systems
process run queue. Messages in the active queue are read to be
sent (runnable), but are not necessarily in the process of being
sent (running).
While most Postfix administrators think of the "active" queue as
a directory on disk, the real "active" queue is a set of data
structures in the memory of the queue manager process.
Messages in the "maildrop", "hold", "incoming" and "deferred" (see
below) queues do not occupy memory, they are safely stored on disk
waiting for their turn to be processed. The envelope information for
messages in the "active" queue is managed in memory, allowing the
queue manager to do global scheduling allocating available delivery
agent processes to an appropriate message in the active queue.
Within the active queue (multi-recipient) messages are broken up into
groups of recipients that share the same transport/nexthop combination,
the group size is capped by the transport's recipient concurrency limit.
Multiple recipient groups (from one or more messages) are queued
for delivery via the common transport/nexthop combination. The
destination concurrency limit for the transports caps the number
of simultenous delivery attempts for each nexthop. Transports with
a recipient concurrency limit of 1 are special, these the grouped by
the actual recipient address rather than the nexthop, thereby enabling
per-recipient concurrency limits rather than per-domain concurrency
limits. Per recipient-limits are appropriate when performing final
delivery to mailboxes rather relaying to a remote server.
Congestion occurs in the active queue when one or more destinations
drain more slowly than the corresponding message input rate. If the
destination is down for some time, the queue manager will mark it dead,
and immediately defer all mail for the destination without trying
to assign it to a delivery agent, in this case the messages will
quickly leave the active queue and end up in the deferred queue. If
the destination is instead simply slow, or there is a problem causing
an excessive arrival rate the active queue queue will grow and will
become dominated by mail to the congested destination.
The only way to reduce congestion is to either reduce the input rate
or increase the throughput. Increasing the throughput requires either
increasing the concurrency or reducing the latency of deliveries.
Another common cause of congestion is unwarranted flushing of the
entire deferred queue. The deferred queue holds messages that are
likely to fail to be delivered and are also likely to be slow to
fail delivery (timeouts). This means that the most common reaction
to a large deferred queue (flush it!) is more than likely counter-
productive, and is likely to make the problem worse. Do not flush the
deferred queue unless you expect that most of its content has recently
become deliverable (e.g. relayhost back up after an outage)!
Note that whenever the queue manager is restarted, there may already
be messages in the active queue directory, but the "real" active
in memory is empty. In order recover the in-memory state, the queue
manager moves all the active directory messages back into the incoming
queue, and then uses its normal incoming queue scan to refill the
active queue. The process of moving all the messages back and forth,
redoing transport table (trivial-rewrite resolve service) lookups,
and re-importing the messages back into memory is expensive. At
all costs avoid frequent restarts of the queue manager.
- The "deferred" queue.
When all the deliverable recipients for a message are delivered, and
for some recipients delivery failed for a transient reason (it might
succeed later), the message is placed in the deferred queue.
The queue manager scans the deferred queue periodically, the scan
interval is controlled by the "queue_run_delay" parameter. While
a deferred queue scan is in progress, if an incoming queue scan is
also in progress (ideally these are brief since the incoming queue
should be short), the queue manager alternates between bringing a new
"incoming" message and a new "deferred" message into the queue. This
"round-robin" strategy prevent starvation of either the incoming or
the deferred queues,
Each deferred queue scan only brings a fraction of the deferred queue
back into the active queue for a retry, this is because each message
in the deferred queue is assigned a "cool-off" time when it is deferred.
This is done by time-warping the modification times of the queue file
into the future. The queue file is not eligible for a retry if its
modification time is not yet reached.
The "cool-off" time is at least $minimal_backoff_time and at most
$maximal_backoff_time. The next retry time is set by doubling the
message's age in the queue, and adjusting up or down to lie within
the limits. This means that young messages are initially retried more
often than old messages.
If a high volume site routinely has large deferred queues, it may
be useful to adjust the queue_run_delay, minimal_backoff_time and
maximal_backoff_time to provide short enough delays on first failure,
with perhaps longer delays after multiple failures, to reduce the
retransmission rate of old messages and thereby reduce the quantity
of previously deferred mail in the active queue.
One common reason for large deferred queues is failure to do recipient
validation at the SMTP input stage. Since spammers routinely launch
dictionary attacks from unrepliable sender addresses the bounces
for invalid recipient addresses clog the deferred queue (and at high
volumes proportionally the active queue). Recipient validation is a
must.
When a host with lots of deferred mail is down for some time, it
is possible for the entire deferred queue to reach its retry time
simulteneously. This can lead to a very full active queue once the
host comes back up. The phenomenon can repeat approximately every
maximal backoff time seconds if the messages are again deferred after
a brief burst of congestion. Ideally in the future Postfix will add a
random offset to the retry time (or use a combination of strategies)
to reduce the chances of repeated complete deferred queue flushes.
Using qshape:
=============
The "qshape" program displays a tabular view of the Postfix queue contents.
On the horizontal axis it displays the queue age with fine granularity for
recent messages and (geometrically) less fine granularity for older messages.
The the vertical axis it displays the destination (or with the "-s" switch
the sender) domain.
For example in the output below, we see the top 10 lines of the (mostly
forged) sender domain distribution for captured spam in the a "hold"
queue (qshape -s hold):
T 5 10 20 40 80 160 320 640 1280 1280+
TOTAL 400 0 0 0 1 1 3 10 10 15 360
yahoo.com 11 0 0 0 0 0 0 0 0 0 11
ms35.hinet.net 11 0 0 0 0 0 0 0 1 2 8
hotmail.com 10 0 0 0 0 0 0 0 1 1 8
winnersdaily.net 9 0 0 0 0 0 0 0 0 0 9
extremepricecuts.net 9 0 0 0 0 0 0 0 0 0 9
telus.net 5 0 0 0 0 0 0 0 1 0 4
attbi.com 5 0 0 0 0 0 0 1 0 1 3
The "T" column shows the total (in this case sender) count for each
domain. The columns with numbers above them, show counts for messages
aged fewer than that many minutes, but not younger than the age limit
for the previos column. The row labeled "TOTAL" shows the total count
for all domains.
In this case for example, there 10 messages allegedly from hotmail.com,
1 between 320 and 639 minutes old, 1 between 640 and 1279 minutes old
and 8 older than 1280 minutes.
By default qshape shows statistics for the union of the incoming and
active queues which are directly relevant to Postfix performance.
One can request an alternate list of queues:
$ qshape deferred
$ qshape incoming active deferred
this will show the age distribution of the deferred queue or the union of the
incoming active and deferred queues.
Large numbers in the qshape output represent a large number of messages
that are destined to (or alleged to come from) a particular domain.
It should be possible to tell at a glance which domains dominate the
queue sender or recipient counts, approximately when the spike started,
and when it stopped.
The problem destinations or sender domains appear near the top left corner
of the output table. Remember that the active queue can accomodate up to
20000 ($qmgr_message_active_limit) messages. To check wether this limit
has been reached, use:
$ qshape -s active | head
if the total sender count is below 20000 the active queue is not yet
saturated, any high volume sender domains show near the top of the output.
The active queue is also limited to at most 20000 recipient addresses
($qmgr_message_recipient_limit). To check for exhaustion of this limit
use:
$ qshape active | head
Command line options control the number of display "buckets", the age
limit for the smallest bucket, display of parent domain counts and so
on. The "-h" option outputs a summary of the available switches.
Having found the high volume domains, it is often useful to search the
logs for recent messages pertaining to the domains in question.
# Find deliveries to example.com
#
tail -10000 /var/log/maillog |
egrep -i ': to=<[EMAIL PROTECTED]>,' |
less
# Find messages from example.com
#
tail -10000 /var/log/maillog |
egrep -i ': from=<[EMAIL PROTECTED]>,' |
less
You may want to drill in on some specific queue ids:
# Find all messages for a specific queue id.
#
tail -10000 /var/log/maillog | egrep ': 2B2173FF68: '
Also look for queue manager warning messages in the log, these can
suggest strategies to reduce congestion.
When all else fails try the Postfix mailing list for help, don't forget
to include the top 10 or 20 lines of "qshape" output.
