On 19/05/11 15:43, W B Hacker wrote:
W B Hacker wrote:
Phil Pennock wrote:
On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
On 19/05/11 11:50, Phil Pennock wrote:
New queuing system refers to an approach to scale up the spool
directory
to something more queue-like, with segregated admin-defined queues
(eg,
"big_freemail_provider_x"). This is because while Exim is
excellent at
inbound mail, it doesn't always scale as well as some would like for
outbound mail which can't be immediately delivered. Nothing has been
done on this. Patches welcome.
Is this AKA bug 336? It sounds quite interesting, so I think I might
have a look at making some inroads into the problem.
If there are are any notes/thoughts about behaviour/config/use it
would
be handy to hear them.
I wasn't aware of 336, but yes it's related.
At present, there's split_spool_directory, which divides things up with
one level of hashing, and then some people script their own
queue-runner
launchers, running in parallel over sub-trees of the split spool
instead
of having the Exim daemon launch runners over everything, which compete
with each other.
Nothing more specific was discussed, that I either recall or find in
the
minutes; we all understood the general problem.
Known bound, yes.
Problem?
Not sure.
I stand on the position that WHEN one is loading Exim so heavily that it
HITS that sort of bound, one has far too many eggs in one basket
downtime-risk-wise, and should split that load over multiple Exim
(free)....
... on multiple separate boxen (generally cheap). Or at least usually
'cheapER' than answering complaints from such a huge user base - or
rushing to a remote data center.
If we use a new sub-directory of spool_directory to hold named queues,
then previous Exim installs won't know of the content, but it should be
fairly easy to script a rollback tool which recombines queues into the
original queue.
*snip*
..or doesn't need to do so at all...
Here's a use-what-we-have scenario for high load and costly (leased?)
single-box.
Simulate a cluster of virtual servers, but w/o the noise and nuisance
of jails or actual VDM's:
- create a directory tree, preferably on separate devices/mounts, ELSE
SSDD (Disk Drive, not Different Day) - optionally [sub] named per
listener-IP.
- install or clone exim1 thru exim(n), each with its executables and
~/configure renamed to suit, and its own listener IP, master script to
control sequence & time-spacing of startup.
These CAN (with same-group privs) all write to a common mailstore on
yet-another mount to make life easier for POP/IMAP. Or those can ALSO
remain separate - for similar load management reasons.
Each of these umm, 'partitions'..? .. will have its own queue - hints
db, logs (or not) .. everything else.
So long as the BOX and OS have the guts to pull it anyway, the Exim
queue issue - and all else - is compartmentalized to smaller chunks of
the load...
CAVEAT: Separate HDD spindles and head positioners, eg whole drives -
de riguer. ELSE, solid-state aside - r/w load creates the enviable,
but ultimately costly, sound of self-fornicating machinery of the
sewing machine phylum.
I'd even bet this has been done already...
;-)
Bill
I think the interesting use of multiple queues comes in when policy is
involved in deciding where to go: the simplest case where they are all
equal 'bins' is, as you say, readily served by having multiple
independent instances.
Obvious candidates would be a queue for messages that are relatively
time-sensitive (SMS-Email, host status alerts etc.), which should be
delivered ASAP even if there are 50,000 in the default queue. Or the
inverse, where there is a source of an occasional torrent of messages,
which should be delayed as necessary to avoid impacting the flow of
routine mail - say allowing queue_only in a per-spool context.
Another one would be a queue for relaying to a [local] server, which can
and is prepared to accept mail at a substantially higher rate than a
typical recipient, or where the same rate to remote domains would
negatively impact WAN performance. Or its inverse where there is a
particular recipient or set thereof that require a lower rate limit.
Another again might be to put messages with questionable spam status in
a queue with lower priority/frequency.
Dom
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
