Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="

2020-06-16 Thread Scott Kitterman
On Wednesday, June 17, 2020 12:39:06 AM EDT Bob Proulx wrote:
> It resulted in this configuration entry.
> 
> root@turmoil:~# postconf relayhost
> relayhost = smtp.proulx.com:587
> 
> And so it is possible to use that Debian specific installation
> option.  Any more details here should be on a Debian specific mailing
> list.

Those are both good points, but it's also worth noting that typically 
submission requires SMTP Auth which you will have to configure manually after 
doing the above.

Scott K





Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="

2020-06-16 Thread Bob Proulx
Gary Aitken wrote:
> Wietse Venema wrote:
> > Perhaps you're better of with
> > - uninstall Postfix
> > - reinstall Postfix
> >
> > and only after doing that edit Postfix config files.
>
> A simple uninstall and reinstall of postfix could not be used, as the 
> uninstall
> would remove another package (automysqlbackup) which depended on postfix.
> That package was already working, and I didn't want to disturb it.  I got
> around that by installing nullmailer, which satisfied the default-mta and
> mail-transfer-agent dependencies from automysqlbackup; then uninstalled
> postfix without removing/uninstalling anything else.  Then purge postfix
> to remove the config files, then reinstall postfix.

For future reference it is also possible to use dpkg to remove postfix
ignoring the dependency and then install it again satisfying the
dependency.

  dpkg --purge --force-depends postfix
  ...verify /etc/postfix/ and other locations are clean...
  apt-get install postfix
  ...configure it as desired...

That would have avoided the clever use of an alternate MTA such as
nullmailer as a dependency placeholder. :-)

> This is a satellite system which should receive no mail, but which needs to
> post mail from root and other daemon-users.  I thought I had it set up by
> answering the setup questions as:
>   Type of system: Satellite
>   Domain: xbiologix.net
>   mail relay: aspmx3.googlemail.com
...
> When attempting to send mail, /var/log/mail.log shows the following:
>
> Jun 16 17:18:11 xblgx-ops postfix/smtp[26224]: connect to 
> aspmx3.googlemail.com[142.250.10.27]:25: Connection timed out
>
> It is using default mail port, 25; I need port 465 or 587 because it's my
> understanding google blocks everything on port 25.
> During the setup, I was not (I don't think) given the option of specifying
> smtps or the port.

It was there.  I ran through a test just now.  dpkg --purge as above
and the installed.  Selected Satellite.  I see this:

Please specify a domain, host, host:port, [address] or [address]:port. Use 
the form
[destination] to turn off MX lookups. Leave this blank for no relay host.

Do not specify more than one host.

The relayhost parameter specifies the default host to send mail to when no 
entry is
matched in the optional transport(5) table. When no relay host is given, 
mail is routed
directly to the destination.

SMTP relay host (blank for none):

When I specified the following to that form:

smtp.proulx.com:587

It resulted in this configuration entry.

root@turmoil:~# postconf relayhost
relayhost = smtp.proulx.com:587

And so it is possible to use that Debian specific installation
option.  Any more details here should be on a Debian specific mailing
list.

However personally I always simply choose Internet Site and then
configure everything I want on top of that template.  It's a good
default to start from but I always want more specific customizations
on top of it.

Bob


Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="

2020-06-16 Thread Gary Aitken

On 6/16/20 9:49 PM, Scott Kitterman wrote:

On Tuesday, June 16, 2020 11:36:27 PM EDT Gary Aitken wrote:

It is using default mail port, 25; I need port 465 or 587 because
it's my understanding google blocks everything on port 25. During
the setup, I was not (I don't think) given the option of
specifying smtps or the port.


There's no preconfigured option for that setup.  The satellite system
option does relay on port 25.  If you want Postfix to do submission,
you need to set that up manually.


ok, I'm willing, but I haven't the faintest idea what setting it up manually
means.  I presume some settings in main.cf, but postconf -d shows 979 things
to choose from.  suggestions?

Thanks,

Gary


Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="

2020-06-16 Thread Scott Kitterman
On Tuesday, June 16, 2020 11:36:27 PM EDT Gary Aitken wrote:
> It is using default mail port, 25; I need port 465 or 587 because it's my
> understanding google blocks everything on port 25.
> During the setup, I was not (I don't think) given the option of specifying
> smtps or the port.

There's no preconfigured option for that setup.  The satellite system option 
does relay on port 25.  If you want Postfix to do submission, you need to set 
that up manually.

Scott K




Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="

2020-06-16 Thread Gary Aitken

On 6/12/20 12:26 PM, Wietse Venema wrote:

Gary Aitken:

I had previously edited main.cf to set
sendmail_path = /usr/sbin/sendmail
setgid_group = postdrop

$ sudo postfix check
postfix: fatal: bad string length 0 < 1: mailq_path =

Not sure what mailq_path should be set to... /var/spool/postfix/ ?


http://www.postfix.org/postconf.5.html#mailq_path

 Sendmail compatibility feature that specifies where the Postfix
 mailq(1) command is installed. This command can be used to list
 the Postfix mail queue.


Do all of these need to be set?  I thought the re-configure should
have taken care of this, and reasonable defaults would be applied?
Do I need to remove main.cf before doing a dpkg-reconfigure?


Perhaps you're better of with

- uninstall Postfix
- reinstall Postfix

and only after doing that edit Postfix config files.


A simple uninstall and reinstall of postfix could not be used, as the uninstall
would remove another package (automysqlbackup) which depended on postfix.
That package was already working, and I didn't want to disturb it.  I got
around that by installing nullmailer, which satisfied the default-mta and
mail-transfer-agent dependencies from automysqlbackup; then uninstalled
postfix without removing/uninstalling anything else.  Then purge postfix
to remove the config files, then reinstall postfix.

So, now I have it installed "properly".  yea!
However...

This is a satellite system which should receive no mail, but which needs to
post mail from root and other daemon-users.  I thought I had it set up by
answering the setup questions as:
  Type of system: Satellite
  Domain: xbiologix.net
  mail relay: aspmx3.googlemail.com

I have modified main.cf by adding:

smtpd_tls_cert_file=/etc/letsencrypt/live/xbiologix.net/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/xbiologix.net/privkey.key
...
smtp_tls_security_level = may
smtp_tls_loglevel = 1

$ postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
html_directory = /usr/share/doc/postfix/html
inet_interfaces = loopback-only
inet_protocols = all
mailbox_size_limit = 0
mydestination =
myhostname = xblgx-ops.c.insidexblgx.internal
mynetworks = 127.0.0.0/8 [:::127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
readme_directory = /usr/share/doc/postfix
recipient_delimiter = +
relay_domains =
relayhost = aspmx3.googlemail.com
smtp_tls_loglevel = 1
smtp_tls_security_level = may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated 
defer_unauth_destination
smtpd_tls_cert_file = /etc/letsencrypt/live/xbiologix.net/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/xbiologix.net/privkey.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes

When attempting to send mail, /var/log/mail.log shows the following:

Jun 16 17:18:11 xblgx-ops postfix/smtp[26224]: connect to 
aspmx3.googlemail.com[142.250.10.27]:25: Connection timed out

It is using default mail port, 25; I need port 465 or 587 because it's my
understanding google blocks everything on port 25.
During the setup, I was not (I don't think) given the option of specifying
smtps or the port.

There is supposedly a default egress rule from google cloud vm instances,
but I don't see it listed in my firewall rules.  However, I think the listed
firewall rules do not include the default *implied* rules imposed at a higher
level; at least the google cloud docs kind of imply this:

"Firewall rules must allow egress traffic from the instance. Unless overridden by a 
higher priority rule, the implied allow rule for egress traffic permits outbound traffic 
from all instances."

In any case, I added an explicit rule to allow egress on port 465.
tcpdump shows attempts going out, but on port 25, not 465.

struggling along, any help much appreciated

Gary


_restrictions

2020-06-16 Thread Helmut Schneider

Hi,

I'm cleaning up my postfix configs and am wondering if I can improve / 
should change my _restrictions on postfix 3.3 / 3.5:


local postfix instance:

smtpd_client_restrictions =
permit_mynetworks
permit_sasl_authenticated
smtpd_helo_restrictions =
permit_mynetworks
permit_sasl_authenticated
smtpd_sender_restrictions =
permit_mynetworks
permit_sasl_authenticated
smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_destination
smtpd_data_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_pipelining

relaying instance inbound:

smtpd_client_restrictions =
permit_mynetworks
permit_sasl_authenticated
warn_if_reject check_client_access hash:/etc/postfix-in/client_access
warn_if_reject reject_unknown_client_hostname
warn_if_reject reject_unknown_reverse_client_hostname
warn_if_reject reject_rbl_client ix.dnsbl.manitu.net
warn_if_reject reject_rbl_client zen.spamhaus.org
smtpd_helo_restrictions =
permit_mynetworks
permit_sasl_authenticated
warn_if_reject check_helo_access hash:/etc/postfix-in/helo_access
warn_if_reject reject_non_fqdn_helo_hostname
warn_if_reject reject_invalid_helo_hostname
warn_if_reject reject_unknown_helo_hostname
smtpd_sender_restrictions =
permit_mynetworks
permit_sasl_authenticated
warn_if_reject check_sender_access hash:/etc/postfix-in/sender_access
warn_if_reject reject_non_fqdn_sender
warn_if_reject reject_unknown_sender_domain
smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unlisted_recipient
reject_unauth_destination
smtpd_data_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_pipelining
smtpd_relay_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_destination

relaying instance outbound:

smtpd_client_restrictions =
permit_mynetworks
permit_sasl_authenticated
check_client_access cidr:/etc/postfix-out/client_access
warn_if_reject reject_unknown_client_hostname
warn_if_reject reject_unknown_reverse_client_hostname
smtpd_helo_restrictions =
permit_mynetworks
permit_sasl_authenticated
check_helo_access hash:/etc/postfix-out/helo_access
reject_non_fqdn_helo_hostname
reject_invalid_helo_hostname
warn_if_reject reject_unknown_helo_hostname
smtpd_sender_restrictions =
permit_mynetworks
permit_sasl_authenticated
smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
check_client_access cidr:/etc/postfix-out/client_access
reject_unauth_destination
smtpd_data_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_pipelining
smtpd_relay_restrictions =
permit_mynetworks
permit_sasl_authenticated
check_client_access cidr:/etc/postfix-out/client_access
reject_unauth_destination

Anything missing / redundant / unneccessary?

Thank you!



Re: Postfix mailq priority

2020-06-16 Thread Wietse Venema
vi...@vheuser.com:
> On 2020/06/16 13:42 PM, Wietse Venema wrote:
> > vi...@vheuser.com:
> >> Obviously I am above my pay grade here,
> >> but can this? "Adding some artificial 'cost' value" currently be done?
> > No. There is an existing solution that works for multi-recipient
> > list mail. That solution does not work for single-recipient list mail,
> > which we suspect is your use case.
> >
> > Viktor proposes to schedule single-recipient list mail from the
> > same sender as if it is multi-recipient list mail (post-facto grouping).
> >
> > I'm proposing to just put a cost on each delivery request, initially
> > based on how many other requests a sender already has in flight,
> > that decays over time until the request is selected for delivery.
> > This guarantees that all mail will eventually be delivered.
> >
> > This also would give us the option to make the scheduling dependent
> > on message size, or quality-of-service indicators.
> >
> > Wietse
> Just fyi -
> Mailman domain concurrency is a yes/no/full option.
> I've limited postfix to 5 and set Mailman to yes
> and that sped things up dramatically
> with no rejections from TWC.

Perhaps you set some Postfix xxx_destination_recipient_limit=1.  As
documented that makes the Postfix concurrency limit a per-recipient
property, which increases the total concurrency for a destinaion domain.

Wietse


Re: Postfix mailq priority

2020-06-16 Thread vi...@vheuser.com

On 2020/06/16 13:42 PM, Wietse Venema wrote:

vi...@vheuser.com:

Obviously I am above my pay grade here,
but can this? "Adding some artificial 'cost' value" currently be done?

No. There is an existing solution that works for multi-recipient
list mail. That solution does not work for single-recipient list mail,
which we suspect is your use case.

Viktor proposes to schedule single-recipient list mail from the
same sender as if it is multi-recipient list mail (post-facto grouping).

I'm proposing to just put a cost on each delivery request, initially
based on how many other requests a sender already has in flight,
that decays over time until the request is selected for delivery.
This guarantees that all mail will eventually be delivered.

This also would give us the option to make the scheduling dependent
on message size, or quality-of-service indicators.

Wietse

Just fyi -
Mailman domain concurrency is a yes/no/full option.
I've limited postfix to 5 and set Mailman to yes
and that sped things up dramatically
with no rejections from TWC.



Re: Postfix mailq priority

2020-06-16 Thread Wietse Venema
vi...@vheuser.com:
> Obviously I am above my pay grade here,
> but can this? "Adding some artificial 'cost' value" currently be done?

No. There is an existing solution that works for multi-recipient
list mail. That solution does not work for single-recipient list mail,
which we suspect is your use case.

Viktor proposes to schedule single-recipient list mail from the
same sender as if it is multi-recipient list mail (post-facto grouping).

I'm proposing to just put a cost on each delivery request, initially
based on how many other requests a sender already has in flight,
that decays over time until the request is selected for delivery.
This guarantees that all mail will eventually be delivered.

This also would give us the option to make the scheduling dependent
on message size, or quality-of-service indicators.

Wietse
> 
> 


Re: Postfix mailq priority

2020-06-16 Thread Viktor Dukhovni
On Tue, Jun 16, 2020 at 01:25:41PM -0400, Wietse Venema wrote:

> > I will look up how to create transports and assign mail to them.
> > But it appears that there would be no way to redirect what is
> > already in the queue.

If you already have a lot of messages in the "active" queue, then those
already are already resolve to a transport, but you can get the
transport to be recomputed by putting all the messages "on hold", and
then immediately releassing them (into the deferred queue) via
"postsuper -H".

> > Is that correct?
> 
> The transport is NOT stored in the queue file. The transport map
> is read every time the queue manager tries to deliver mail.

More precisely, whenever the queue manager imports a message into
the active queue and queues it for delivery.  So recomputed each
time a message is retried (from "deferred" or perhaps "hold").

-- 
Viktor.


Re: Postfix mailq priority

2020-06-16 Thread Wietse Venema
vi...@vheuser.com:
> Postfix steadily delivers, but would like to de-prioritize the list emails
> so regular traffic is not delayed.

That's what we're talking about  - making that automatic so you
don't have to babysit Postfix. Otherwise, we'd be wasting everyone's
time.

> I will look up how to create transports and assign mail to them.
> But it appears that there would be no way to redirect what is
> already in the queue.
> Is that correct?

The transport is NOT stored in the queue file. The transport map
is read every time the queue manager tries to deliver mail.

Wietse


Re: Postfix mailq priority

2020-06-16 Thread vi...@vheuser.com



On 2020/06/16 10:39 AM, Wietse Venema wrote:

Viktor Dukhovni:

On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote:


Postfix has only one queue, and concurrency and process count is
per-destination, not based on where the mail came from.

Consider using a separate postfix instance for delivering mail list
messages to prevent them from interfering with regular mail. See
MULTI_INSTANCE_README

There's one active queue, but within that active queue messages are
queued to a particular (transport,nexthop) pair, and scheduling is
round-robin by transport, and then FIFO, subject to concurrency and
process limits and amortised pre?mption of multi-recipient messages by
messages with fewer recipients, so that messages with lots of recipients
don't hog the queue too long before some other messages with fewer
recipients that arrived later get to use a delivery slot.

If one just wants to put messages in multiple "lines" for delivery
scheduling, but otherwise all settings are the same, then using multiple
transports is simpler and often just as effective as multiple instances.

The more Postfix can do by itself, the better. That could be:

- Adding layer of sender-based round-robin selection. Not sure if
that would explode at large scale.

- Adding some artificial 'cost' value that is computed while delivery
requests are added to per-destination queues. Cost could depend on
the number of delivery requests per sender email address, the message
size, and so on. Then, the scheduler could choose what-to-deliver
based on artificial cost in addition to the things that it already
considers now.

Wietse


Obviously I am above my pay grade here,
but can this  "Adding some artificial 'cost' value" currently be done?  How?



Re: Postfix mailq priority

2020-06-16 Thread vi...@vheuser.com



On 2020/06/16 11:32 AM, Viktor Dukhovni wrote:

On Jun 16, 2020, at 12:39 PM, Wietse Venema  wrote:

- Adding layer of sender-based round-robin selection. Not sure if
that would explode at large scale.

When the input mix approximates steady-state, FIFO is essentially optimal,
with each type of message getting its average share of the output in a fair
and timely manner.  If/when we stray from FIFO, we need to be carefuly not
be simply underserving some fraction of the expected steady-state load
distribution.

This would mean being able to somehow detect a "burst" of traffic and
characterise to distinguish the messages that are contributing to the
burst from other messages.  That's a tricky problem.  The sender address
for bulk traffic is liable to have VERP tags, the client IP of interest
may be a few hops back from the edge system that encounters delays in
delivering an input burst to remote organizations (ADMDs in the language
of RFC5598).


- Adding some artificial 'cost' value that is computed while delivery
requests are added to per-destination queues. Cost could depend on
the number of delivery requests per sender email address, the message
size, and so on. Then, the scheduler could choose what-to-deliver
based on artificial cost in addition to the things that it already
considers now.

Ideally we'd have an algorithm that could group related messages into
a set of logical "bulk" sources, and apply the current bulk message
preëmption algorithm not only to multi-recipient mail, but also to
multi-message bulk sources.  The hard part is the classification,
especially in a single-threaded queue manager that needs to do this
in O(1 millisecond).

Perhaps the best proxy for related messages is origin domain (ignoring
localpart) + approximate message size.  Related messages are likely to
carry similar content of approximately the same size to all recipients.
But completely ignoring the localpart may be too coarse.  It is not
obvious to me how to extract common elements from a per-recipient-salted
localpart...

A traditional multi-recipient message automatically qualifies as a single
source (same message size for all recipients and same origin domain).

We could attempt to group "related" messages in a fuzzy manner as above,
and then apply the existing preëmption algorithm.  Still not sure how to
do a good job of the aggregation.


I've got a lot to learn here.  Didn't expect so much detail!

Ideally, one could flag the big list somehow and not worry about the little 
ones.
I turned off concurrency because TWC.com was rejecting bundles of emails.
Serious slow down.
Postfix steadily delivers, but would like to de-prioritize the list emails
so regular traffic is not delayed.

I will look up how to create transports and assign mail to them.
But it appears that there would be no way to redirect what is already in the 
queue.
Is that correct?
(Sheepish grin) Feature request?  (-:



Re: Postfix mailq priority

2020-06-16 Thread Wietse Venema
Viktor Dukhovni:
> > On Jun 16, 2020, at 12:39 PM, Wietse Venema  wrote:
> > 
> > - Adding layer of sender-based round-robin selection. Not sure if
> > that would explode at large scale.
> 
> When the input mix approximates steady-state, FIFO is essentially optimal,
> with each type of message getting its average share of the output in a fair
> and timely manner.  If/when we stray from FIFO, we need to be carefuly not
> be simply underserving some fraction of the expected steady-state load
> distribution.

Specifically, we don't want to forever deprioritize a request, and
that is usually done over time by decreasing the thing that causes
a request to be postponed. In the next approach, artificial cost,
we would monotonically decrease the artificial cost over time
so that a delivery request is not permanently starved.

> This would mean being able to somehow detect a "burst" of traffic and
> characterise to distinguish the messages that are contributing to the
> burst from other messages.  That's a tricky problem.  The sender address
> for bulk traffic is liable to have VERP tags, the client IP of interest
> may be a few hops back from the edge system that encounters delays in
> delivering an input burst to remote organizations (ADMDs in the language
> of RFC5598).

Of course any sender-based metrics would be based on sender address
without the address extension (i.e. the part that VERP  adds).

> > - Adding some artificial 'cost' value that is computed while delivery
> > requests are added to per-destination queues. Cost could depend on
> > the number of delivery requests per sender email address, the message
> > size, and so on. Then, the scheduler could choose what-to-deliver
> > based on artificial cost in addition to the things that it already
> > considers now.
> 
> Ideally we'd have an algorithm that could group related messages into
> a set of logical "bulk" sources, and apply the current bulk message
> pre?mption algorithm not only to multi-recipient mail, but also to
> multi-message bulk sources.  The hard part is the classification,
> especially in a single-threaded queue manager that needs to do this
> in O(1 millisecond).

With artificial cost done well, it should make little difference
if a message has 1 recipients in distinct domains, or if those
same recipients are enqueued in 1 single-recipient submissions.
In both cases we would have 1 delivery requests, and in both
cases the artificial cost would be similar.

I think that would be an improvement.

I'd like to avoid grouping as too diffult, and consider artificial
cost (with monotonic decrease) as more feasible, provided that the
'current artificial cost' metric is easily updated (for example a
hash table indexed by sender that counts the number of delivery
requests with that sender). Not that the number of delivery requests
measures performance impact better than number of recipients.

Each new delivery request is then assigned an artificial cost that
recflects the 'current cost' metrics at that point in time. Over
time, the artificial cost for that delivery request is decreased
monotonically so that it will not be starved.

> Perhaps the best proxy for related messages is origin domain (ignoring
> localpart) + approximate message size.  Related messages are likely to
> carry similar content of approximately the same size to all recipients.
> But completely ignoring the localpart may be too coarse.  It is not
> obvious to me how to extract common elements from a per-recipient-salted
> localpart...
> 
> A traditional multi-recipient message automatically qualifies as a single
> source (same message size for all recipients and same origin domain).

See my above comment abount the equivalence of multi-recipient
list mail versus single-recipient.

Look, Mom! No grouping needed.

Wietse

> We could attempt to group "related" messages in a fuzzy manner as above,
> and then apply the existing pre?mption algorithm.  Still not sure how to
> do a good job of the aggregation.
> 
> -- 
> -- 
>   Viktor.
> 
> 


Re: Postfix mailq priority

2020-06-16 Thread Viktor Dukhovni
> On Jun 16, 2020, at 12:39 PM, Wietse Venema  wrote:
> 
> - Adding layer of sender-based round-robin selection. Not sure if
> that would explode at large scale.

When the input mix approximates steady-state, FIFO is essentially optimal,
with each type of message getting its average share of the output in a fair
and timely manner.  If/when we stray from FIFO, we need to be carefuly not
be simply underserving some fraction of the expected steady-state load
distribution.

This would mean being able to somehow detect a "burst" of traffic and
characterise to distinguish the messages that are contributing to the
burst from other messages.  That's a tricky problem.  The sender address
for bulk traffic is liable to have VERP tags, the client IP of interest
may be a few hops back from the edge system that encounters delays in
delivering an input burst to remote organizations (ADMDs in the language
of RFC5598).

> - Adding some artificial 'cost' value that is computed while delivery
> requests are added to per-destination queues. Cost could depend on
> the number of delivery requests per sender email address, the message
> size, and so on. Then, the scheduler could choose what-to-deliver
> based on artificial cost in addition to the things that it already
> considers now.

Ideally we'd have an algorithm that could group related messages into
a set of logical "bulk" sources, and apply the current bulk message
preëmption algorithm not only to multi-recipient mail, but also to
multi-message bulk sources.  The hard part is the classification,
especially in a single-threaded queue manager that needs to do this
in O(1 millisecond).

Perhaps the best proxy for related messages is origin domain (ignoring
localpart) + approximate message size.  Related messages are likely to
carry similar content of approximately the same size to all recipients.
But completely ignoring the localpart may be too coarse.  It is not
obvious to me how to extract common elements from a per-recipient-salted
localpart...

A traditional multi-recipient message automatically qualifies as a single
source (same message size for all recipients and same origin domain).

We could attempt to group "related" messages in a fuzzy manner as above,
and then apply the existing preëmption algorithm.  Still not sure how to
do a good job of the aggregation.

-- 
-- 
Viktor.



Re: Postfix mailq priority

2020-06-16 Thread Wietse Venema
Viktor Dukhovni:
> On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote:
> 
> > Postfix has only one queue, and concurrency and process count is
> > per-destination, not based on where the mail came from.
> > 
> > Consider using a separate postfix instance for delivering mail list
> > messages to prevent them from interfering with regular mail. See
> > MULTI_INSTANCE_README
> 
> There's one active queue, but within that active queue messages are
> queued to a particular (transport,nexthop) pair, and scheduling is
> round-robin by transport, and then FIFO, subject to concurrency and
> process limits and amortised pre?mption of multi-recipient messages by
> messages with fewer recipients, so that messages with lots of recipients
> don't hog the queue too long before some other messages with fewer
> recipients that arrived later get to use a delivery slot.
> 
> If one just wants to put messages in multiple "lines" for delivery
> scheduling, but otherwise all settings are the same, then using multiple
> transports is simpler and often just as effective as multiple instances.

The more Postfix can do by itself, the better. That could be:

- Adding layer of sender-based round-robin selection. Not sure if
that would explode at large scale.

- Adding some artificial 'cost' value that is computed while delivery
requests are added to per-destination queues. Cost could depend on
the number of delivery requests per sender email address, the message
size, and so on. Then, the scheduler could choose what-to-deliver
based on artificial cost in addition to the things that it already
considers now.

Wietse