Re: Postfix mailq priority
vi...@vheuser.com: > On 2020/06/16 13:42 PM, Wietse Venema wrote: > > vi...@vheuser.com: > >> Obviously I am above my pay grade here, > >> but can this? "Adding some artificial 'cost' value" currently be done? > > No. There is an existing solution that works for multi-recipient > > list mail. That solution does not work for single-recipient list mail, > > which we suspect is your use case. > > > > Viktor proposes to schedule single-recipient list mail from the > > same sender as if it is multi-recipient list mail (post-facto grouping). > > > > I'm proposing to just put a cost on each delivery request, initially > > based on how many other requests a sender already has in flight, > > that decays over time until the request is selected for delivery. > > This guarantees that all mail will eventually be delivered. > > > > This also would give us the option to make the scheduling dependent > > on message size, or quality-of-service indicators. > > > > Wietse > Just fyi - > Mailman domain concurrency is a yes/no/full option. > I've limited postfix to 5 and set Mailman to yes > and that sped things up dramatically > with no rejections from TWC. Perhaps you set some Postfix xxx_destination_recipient_limit=1. As documented that makes the Postfix concurrency limit a per-recipient property, which increases the total concurrency for a destinaion domain. Wietse
Re: Postfix mailq priority
On 2020/06/16 13:42 PM, Wietse Venema wrote: vi...@vheuser.com: Obviously I am above my pay grade here, but can this? "Adding some artificial 'cost' value" currently be done? No. There is an existing solution that works for multi-recipient list mail. That solution does not work for single-recipient list mail, which we suspect is your use case. Viktor proposes to schedule single-recipient list mail from the same sender as if it is multi-recipient list mail (post-facto grouping). I'm proposing to just put a cost on each delivery request, initially based on how many other requests a sender already has in flight, that decays over time until the request is selected for delivery. This guarantees that all mail will eventually be delivered. This also would give us the option to make the scheduling dependent on message size, or quality-of-service indicators. Wietse Just fyi - Mailman domain concurrency is a yes/no/full option. I've limited postfix to 5 and set Mailman to yes and that sped things up dramatically with no rejections from TWC.
Re: Postfix mailq priority
vi...@vheuser.com: > Obviously I am above my pay grade here, > but can this? "Adding some artificial 'cost' value" currently be done? No. There is an existing solution that works for multi-recipient list mail. That solution does not work for single-recipient list mail, which we suspect is your use case. Viktor proposes to schedule single-recipient list mail from the same sender as if it is multi-recipient list mail (post-facto grouping). I'm proposing to just put a cost on each delivery request, initially based on how many other requests a sender already has in flight, that decays over time until the request is selected for delivery. This guarantees that all mail will eventually be delivered. This also would give us the option to make the scheduling dependent on message size, or quality-of-service indicators. Wietse > >
Re: Postfix mailq priority
On Tue, Jun 16, 2020 at 01:25:41PM -0400, Wietse Venema wrote: > > I will look up how to create transports and assign mail to them. > > But it appears that there would be no way to redirect what is > > already in the queue. If you already have a lot of messages in the "active" queue, then those already are already resolve to a transport, but you can get the transport to be recomputed by putting all the messages "on hold", and then immediately releassing them (into the deferred queue) via "postsuper -H". > > Is that correct? > > The transport is NOT stored in the queue file. The transport map > is read every time the queue manager tries to deliver mail. More precisely, whenever the queue manager imports a message into the active queue and queues it for delivery. So recomputed each time a message is retried (from "deferred" or perhaps "hold"). -- Viktor.
Re: Postfix mailq priority
vi...@vheuser.com: > Postfix steadily delivers, but would like to de-prioritize the list emails > so regular traffic is not delayed. That's what we're talking about - making that automatic so you don't have to babysit Postfix. Otherwise, we'd be wasting everyone's time. > I will look up how to create transports and assign mail to them. > But it appears that there would be no way to redirect what is > already in the queue. > Is that correct? The transport is NOT stored in the queue file. The transport map is read every time the queue manager tries to deliver mail. Wietse
Re: Postfix mailq priority
On 2020/06/16 10:39 AM, Wietse Venema wrote: Viktor Dukhovni: On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote: Postfix has only one queue, and concurrency and process count is per-destination, not based on where the mail came from. Consider using a separate postfix instance for delivering mail list messages to prevent them from interfering with regular mail. See MULTI_INSTANCE_README There's one active queue, but within that active queue messages are queued to a particular (transport,nexthop) pair, and scheduling is round-robin by transport, and then FIFO, subject to concurrency and process limits and amortised pre?mption of multi-recipient messages by messages with fewer recipients, so that messages with lots of recipients don't hog the queue too long before some other messages with fewer recipients that arrived later get to use a delivery slot. If one just wants to put messages in multiple "lines" for delivery scheduling, but otherwise all settings are the same, then using multiple transports is simpler and often just as effective as multiple instances. The more Postfix can do by itself, the better. That could be: - Adding layer of sender-based round-robin selection. Not sure if that would explode at large scale. - Adding some artificial 'cost' value that is computed while delivery requests are added to per-destination queues. Cost could depend on the number of delivery requests per sender email address, the message size, and so on. Then, the scheduler could choose what-to-deliver based on artificial cost in addition to the things that it already considers now. Wietse Obviously I am above my pay grade here, but can this "Adding some artificial 'cost' value" currently be done? How?
Re: Postfix mailq priority
On 2020/06/16 11:32 AM, Viktor Dukhovni wrote: On Jun 16, 2020, at 12:39 PM, Wietse Venema wrote: - Adding layer of sender-based round-robin selection. Not sure if that would explode at large scale. When the input mix approximates steady-state, FIFO is essentially optimal, with each type of message getting its average share of the output in a fair and timely manner. If/when we stray from FIFO, we need to be carefuly not be simply underserving some fraction of the expected steady-state load distribution. This would mean being able to somehow detect a "burst" of traffic and characterise to distinguish the messages that are contributing to the burst from other messages. That's a tricky problem. The sender address for bulk traffic is liable to have VERP tags, the client IP of interest may be a few hops back from the edge system that encounters delays in delivering an input burst to remote organizations (ADMDs in the language of RFC5598). - Adding some artificial 'cost' value that is computed while delivery requests are added to per-destination queues. Cost could depend on the number of delivery requests per sender email address, the message size, and so on. Then, the scheduler could choose what-to-deliver based on artificial cost in addition to the things that it already considers now. Ideally we'd have an algorithm that could group related messages into a set of logical "bulk" sources, and apply the current bulk message preëmption algorithm not only to multi-recipient mail, but also to multi-message bulk sources. The hard part is the classification, especially in a single-threaded queue manager that needs to do this in O(1 millisecond). Perhaps the best proxy for related messages is origin domain (ignoring localpart) + approximate message size. Related messages are likely to carry similar content of approximately the same size to all recipients. But completely ignoring the localpart may be too coarse. It is not obvious to me how to extract common elements from a per-recipient-salted localpart... A traditional multi-recipient message automatically qualifies as a single source (same message size for all recipients and same origin domain). We could attempt to group "related" messages in a fuzzy manner as above, and then apply the existing preëmption algorithm. Still not sure how to do a good job of the aggregation. I've got a lot to learn here. Didn't expect so much detail! Ideally, one could flag the big list somehow and not worry about the little ones. I turned off concurrency because TWC.com was rejecting bundles of emails. Serious slow down. Postfix steadily delivers, but would like to de-prioritize the list emails so regular traffic is not delayed. I will look up how to create transports and assign mail to them. But it appears that there would be no way to redirect what is already in the queue. Is that correct? (Sheepish grin) Feature request? (-:
Re: Postfix mailq priority
Viktor Dukhovni: > > On Jun 16, 2020, at 12:39 PM, Wietse Venema wrote: > > > > - Adding layer of sender-based round-robin selection. Not sure if > > that would explode at large scale. > > When the input mix approximates steady-state, FIFO is essentially optimal, > with each type of message getting its average share of the output in a fair > and timely manner. If/when we stray from FIFO, we need to be carefuly not > be simply underserving some fraction of the expected steady-state load > distribution. Specifically, we don't want to forever deprioritize a request, and that is usually done over time by decreasing the thing that causes a request to be postponed. In the next approach, artificial cost, we would monotonically decrease the artificial cost over time so that a delivery request is not permanently starved. > This would mean being able to somehow detect a "burst" of traffic and > characterise to distinguish the messages that are contributing to the > burst from other messages. That's a tricky problem. The sender address > for bulk traffic is liable to have VERP tags, the client IP of interest > may be a few hops back from the edge system that encounters delays in > delivering an input burst to remote organizations (ADMDs in the language > of RFC5598). Of course any sender-based metrics would be based on sender address without the address extension (i.e. the part that VERP adds). > > - Adding some artificial 'cost' value that is computed while delivery > > requests are added to per-destination queues. Cost could depend on > > the number of delivery requests per sender email address, the message > > size, and so on. Then, the scheduler could choose what-to-deliver > > based on artificial cost in addition to the things that it already > > considers now. > > Ideally we'd have an algorithm that could group related messages into > a set of logical "bulk" sources, and apply the current bulk message > pre?mption algorithm not only to multi-recipient mail, but also to > multi-message bulk sources. The hard part is the classification, > especially in a single-threaded queue manager that needs to do this > in O(1 millisecond). With artificial cost done well, it should make little difference if a message has 1 recipients in distinct domains, or if those same recipients are enqueued in 1 single-recipient submissions. In both cases we would have 1 delivery requests, and in both cases the artificial cost would be similar. I think that would be an improvement. I'd like to avoid grouping as too diffult, and consider artificial cost (with monotonic decrease) as more feasible, provided that the 'current artificial cost' metric is easily updated (for example a hash table indexed by sender that counts the number of delivery requests with that sender). Not that the number of delivery requests measures performance impact better than number of recipients. Each new delivery request is then assigned an artificial cost that recflects the 'current cost' metrics at that point in time. Over time, the artificial cost for that delivery request is decreased monotonically so that it will not be starved. > Perhaps the best proxy for related messages is origin domain (ignoring > localpart) + approximate message size. Related messages are likely to > carry similar content of approximately the same size to all recipients. > But completely ignoring the localpart may be too coarse. It is not > obvious to me how to extract common elements from a per-recipient-salted > localpart... > > A traditional multi-recipient message automatically qualifies as a single > source (same message size for all recipients and same origin domain). See my above comment abount the equivalence of multi-recipient list mail versus single-recipient. Look, Mom! No grouping needed. Wietse > We could attempt to group "related" messages in a fuzzy manner as above, > and then apply the existing pre?mption algorithm. Still not sure how to > do a good job of the aggregation. > > -- > -- > Viktor. > >
Re: Postfix mailq priority
> On Jun 16, 2020, at 12:39 PM, Wietse Venema wrote: > > - Adding layer of sender-based round-robin selection. Not sure if > that would explode at large scale. When the input mix approximates steady-state, FIFO is essentially optimal, with each type of message getting its average share of the output in a fair and timely manner. If/when we stray from FIFO, we need to be carefuly not be simply underserving some fraction of the expected steady-state load distribution. This would mean being able to somehow detect a "burst" of traffic and characterise to distinguish the messages that are contributing to the burst from other messages. That's a tricky problem. The sender address for bulk traffic is liable to have VERP tags, the client IP of interest may be a few hops back from the edge system that encounters delays in delivering an input burst to remote organizations (ADMDs in the language of RFC5598). > - Adding some artificial 'cost' value that is computed while delivery > requests are added to per-destination queues. Cost could depend on > the number of delivery requests per sender email address, the message > size, and so on. Then, the scheduler could choose what-to-deliver > based on artificial cost in addition to the things that it already > considers now. Ideally we'd have an algorithm that could group related messages into a set of logical "bulk" sources, and apply the current bulk message preëmption algorithm not only to multi-recipient mail, but also to multi-message bulk sources. The hard part is the classification, especially in a single-threaded queue manager that needs to do this in O(1 millisecond). Perhaps the best proxy for related messages is origin domain (ignoring localpart) + approximate message size. Related messages are likely to carry similar content of approximately the same size to all recipients. But completely ignoring the localpart may be too coarse. It is not obvious to me how to extract common elements from a per-recipient-salted localpart... A traditional multi-recipient message automatically qualifies as a single source (same message size for all recipients and same origin domain). We could attempt to group "related" messages in a fuzzy manner as above, and then apply the existing preëmption algorithm. Still not sure how to do a good job of the aggregation. -- -- Viktor.
Re: Postfix mailq priority
Viktor Dukhovni: > On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote: > > > Postfix has only one queue, and concurrency and process count is > > per-destination, not based on where the mail came from. > > > > Consider using a separate postfix instance for delivering mail list > > messages to prevent them from interfering with regular mail. See > > MULTI_INSTANCE_README > > There's one active queue, but within that active queue messages are > queued to a particular (transport,nexthop) pair, and scheduling is > round-robin by transport, and then FIFO, subject to concurrency and > process limits and amortised pre?mption of multi-recipient messages by > messages with fewer recipients, so that messages with lots of recipients > don't hog the queue too long before some other messages with fewer > recipients that arrived later get to use a delivery slot. > > If one just wants to put messages in multiple "lines" for delivery > scheduling, but otherwise all settings are the same, then using multiple > transports is simpler and often just as effective as multiple instances. The more Postfix can do by itself, the better. That could be: - Adding layer of sender-based round-robin selection. Not sure if that would explode at large scale. - Adding some artificial 'cost' value that is computed while delivery requests are added to per-destination queues. Cost could depend on the number of delivery requests per sender email address, the message size, and so on. Then, the scheduler could choose what-to-deliver based on artificial cost in addition to the things that it already considers now. Wietse
Re: Postfix mailq priority
On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote: > Postfix has only one queue, and concurrency and process count is > per-destination, not based on where the mail came from. > > Consider using a separate postfix instance for delivering mail list > messages to prevent them from interfering with regular mail. See > MULTI_INSTANCE_README There's one active queue, but within that active queue messages are queued to a particular (transport,nexthop) pair, and scheduling is round-robin by transport, and then FIFO, subject to concurrency and process limits and amortised preëmption of multi-recipient messages by messages with fewer recipients, so that messages with lots of recipients don't hog the queue too long before some other messages with fewer recipients that arrived later get to use a delivery slot. If one just wants to put messages in multiple "lines" for delivery scheduling, but otherwise all settings are the same, then using multiple transports is simpler and often just as effective as multiple instances. -- Viktor.
Re: Postfix mailq priority
> On Jun 15, 2020, at 7:13 PM, "vi...@vheuser.com" wrote: > > The problem is periodic, completely irregular, mailing list loads. > It would be nice to lower the list priority > so that regular mail got delivered first. > Postfix has only one queue, and concurrency and process count is per-destination, not based on where the mail came from. Consider using a separate postfix instance for delivering mail list messages to prevent them from interfering with regular mail. See MULTI_INSTANCE_README — Noel Jones
Re: Postfix mailq priority
On 2020/06/15 19:01 PM, Viktor Dukhovni wrote: On Mon, Jun 15, 2020 at 04:50:16PM -0400, vi...@vheuser.com wrote: Is there any way to change the order of delivery of items in the mail queue? Not as such, but see below. Is there any way of changing priority of mail when adding to the queue? Not as such, but: - Mail delivery within a given transport is largely FIFO. - But messages for destinations that have reached their destination concurrency limit (for the transport) are delayed until enough earlier messages are delivered. - Mail delivery is round-robin by transport, but deliveries via transports that have reached their process limit are delayed until free process slots become available. Therefore, you can prioritise some messages by using a dedicated transport that is not congested with other low-priority messages and sends to a destination that is willing to accept messages quickly enough at the configured concurrency. Sometimes a dedicated transport is used to separate sluggish low-priority traffic from normal traffic, so that normal traffic delivery is unimpeded. You get to put messages in different queues, some queues may move faster or slower than others. The express checkout line in a supermarket is great if most shoppers are filling carts, and just a few are grabbing a quick couple of items. But if one day everyone wants a carton of milk and a pack of chewing gum, the express line could turn out to be much slower. Thanks, Viktor. The problem is periodic, completely irregular, mailing list loads. It would be nice to lower the list priority so that regular mail got delivered first. I had to force some lists to lower concurrency but now they clog the queue. Guess I had better read up on alternative transports, but thanks to the response.
Re: Postfix mailq priority
On Mon, Jun 15, 2020 at 04:50:16PM -0400, vi...@vheuser.com wrote: > Is there any way to change the order of delivery of items in the mail queue? Not as such, but see below. > Is there any way of changing priority of mail when adding to the queue? Not as such, but: - Mail delivery within a given transport is largely FIFO. - But messages for destinations that have reached their destination concurrency limit (for the transport) are delayed until enough earlier messages are delivered. - Mail delivery is round-robin by transport, but deliveries via transports that have reached their process limit are delayed until free process slots become available. Therefore, you can prioritise some messages by using a dedicated transport that is not congested with other low-priority messages and sends to a destination that is willing to accept messages quickly enough at the configured concurrency. Sometimes a dedicated transport is used to separate sluggish low-priority traffic from normal traffic, so that normal traffic delivery is unimpeded. You get to put messages in different queues, some queues may move faster or slower than others. The express checkout line in a supermarket is great if most shoppers are filling carts, and just a few are grabbing a quick couple of items. But if one day everyone wants a carton of milk and a pack of chewing gum, the express line could turn out to be much slower. -- Viktor.
Postfix mailq priority
Is there any way to change the order of delivery of items in the mail queue? Is there any way of changing priority of mail when adding to the queue? Searched the docs and found nqmgr but not this specific question. Thank you.
