Viktor Dukhovni:
> > On Jun 16, 2020, at 12:39 PM, Wietse Venema <wie...@porcupine.org> wrote:
> >
> > - Adding layer of sender-based round-robin selection. Not sure if
> > that would explode at large scale.
>
> When the input mix approximates steady-state, FIFO is essentially optimal,
> with each type of message getting its average share of the output in a fair
> and timely manner. If/when we stray from FIFO, we need to be carefuly not
> be simply underserving some fraction of the expected steady-state load
> distribution.
Specifically, we don't want to forever deprioritize a request, and
that is usually done over time by decreasing the thing that causes
a request to be postponed. In the next approach, artificial cost,
we would monotonically decrease the artificial cost over time
so that a delivery request is not permanently starved.
> This would mean being able to somehow detect a "burst" of traffic and
> characterise to distinguish the messages that are contributing to the
> burst from other messages. That's a tricky problem. The sender address
> for bulk traffic is liable to have VERP tags, the client IP of interest
> may be a few hops back from the edge system that encounters delays in
> delivering an input burst to remote organizations (ADMDs in the language
> of RFC5598).
Of course any sender-based metrics would be based on sender address
without the address extension (i.e. the part that VERP adds).
> > - Adding some artificial 'cost' value that is computed while delivery
> > requests are added to per-destination queues. Cost could depend on
> > the number of delivery requests per sender email address, the message
> > size, and so on. Then, the scheduler could choose what-to-deliver
> > based on artificial cost in addition to the things that it already
> > considers now.
>
> Ideally we'd have an algorithm that could group related messages into
> a set of logical "bulk" sources, and apply the current bulk message
> pre?mption algorithm not only to multi-recipient mail, but also to
> multi-message bulk sources. The hard part is the classification,
> especially in a single-threaded queue manager that needs to do this
> in O(1 millisecond).
With artificial cost done well, it should make little difference
if a message has 10000 recipients in distinct domains, or if those
same recipients are enqueued in 10000 single-recipient submissions.
In both cases we would have 10000 delivery requests, and in both
cases the artificial cost would be similar.
I think that would be an improvement.
I'd like to avoid grouping as too diffult, and consider artificial
cost (with monotonic decrease) as more feasible, provided that the
'current artificial cost' metric is easily updated (for example a
hash table indexed by sender that counts the number of delivery
requests with that sender). Not that the number of delivery requests
measures performance impact better than number of recipients.
Each new delivery request is then assigned an artificial cost that
recflects the 'current cost' metrics at that point in time. Over
time, the artificial cost for that delivery request is decreased
monotonically so that it will not be starved.
> Perhaps the best proxy for related messages is origin domain (ignoring
> localpart) + approximate message size. Related messages are likely to
> carry similar content of approximately the same size to all recipients.
> But completely ignoring the localpart may be too coarse. It is not
> obvious to me how to extract common elements from a per-recipient-salted
> localpart...
>
> A traditional multi-recipient message automatically qualifies as a single
> source (same message size for all recipients and same origin domain).
See my above comment abount the equivalence of multi-recipient
list mail versus single-recipient.
Look, Mom! No grouping needed.
Wietse
> We could attempt to group "related" messages in a fuzzy manner as above,
> and then apply the existing pre?mption algorithm. Still not sure how to
> do a good job of the aggregation.
>
> --
> --
> Viktor.
>
>
