Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="
On Wednesday, June 17, 2020 12:39:06 AM EDT Bob Proulx wrote: > It resulted in this configuration entry. > > root@turmoil:~# postconf relayhost > relayhost = smtp.proulx.com:587 > > And so it is possible to use that Debian specific installation > option. Any more details here should be on a Debian specific mailing > list. Those are both good points, but it's also worth noting that typically submission requires SMTP Auth which you will have to configure manually after doing the above. Scott K
Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="
Gary Aitken wrote: > Wietse Venema wrote: > > Perhaps you're better of with > > - uninstall Postfix > > - reinstall Postfix > > > > and only after doing that edit Postfix config files. > > A simple uninstall and reinstall of postfix could not be used, as the > uninstall > would remove another package (automysqlbackup) which depended on postfix. > That package was already working, and I didn't want to disturb it. I got > around that by installing nullmailer, which satisfied the default-mta and > mail-transfer-agent dependencies from automysqlbackup; then uninstalled > postfix without removing/uninstalling anything else. Then purge postfix > to remove the config files, then reinstall postfix. For future reference it is also possible to use dpkg to remove postfix ignoring the dependency and then install it again satisfying the dependency. dpkg --purge --force-depends postfix ...verify /etc/postfix/ and other locations are clean... apt-get install postfix ...configure it as desired... That would have avoided the clever use of an alternate MTA such as nullmailer as a dependency placeholder. :-) > This is a satellite system which should receive no mail, but which needs to > post mail from root and other daemon-users. I thought I had it set up by > answering the setup questions as: > Type of system: Satellite > Domain: xbiologix.net > mail relay: aspmx3.googlemail.com ... > When attempting to send mail, /var/log/mail.log shows the following: > > Jun 16 17:18:11 xblgx-ops postfix/smtp[26224]: connect to > aspmx3.googlemail.com[142.250.10.27]:25: Connection timed out > > It is using default mail port, 25; I need port 465 or 587 because it's my > understanding google blocks everything on port 25. > During the setup, I was not (I don't think) given the option of specifying > smtps or the port. It was there. I ran through a test just now. dpkg --purge as above and the installed. Selected Satellite. I see this: Please specify a domain, host, host:port, [address] or [address]:port. Use the form [destination] to turn off MX lookups. Leave this blank for no relay host. Do not specify more than one host. The relayhost parameter specifies the default host to send mail to when no entry is matched in the optional transport(5) table. When no relay host is given, mail is routed directly to the destination. SMTP relay host (blank for none): When I specified the following to that form: smtp.proulx.com:587 It resulted in this configuration entry. root@turmoil:~# postconf relayhost relayhost = smtp.proulx.com:587 And so it is possible to use that Debian specific installation option. Any more details here should be on a Debian specific mailing list. However personally I always simply choose Internet Site and then configure everything I want on top of that template. It's a good default to start from but I always want more specific customizations on top of it. Bob
Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="
On 6/16/20 9:49 PM, Scott Kitterman wrote: On Tuesday, June 16, 2020 11:36:27 PM EDT Gary Aitken wrote: It is using default mail port, 25; I need port 465 or 587 because it's my understanding google blocks everything on port 25. During the setup, I was not (I don't think) given the option of specifying smtps or the port. There's no preconfigured option for that setup. The satellite system option does relay on port 25. If you want Postfix to do submission, you need to set that up manually. ok, I'm willing, but I haven't the faintest idea what setting it up manually means. I presume some settings in main.cf, but postconf -d shows 979 things to choose from. suggestions? Thanks, Gary
Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="
On Tuesday, June 16, 2020 11:36:27 PM EDT Gary Aitken wrote: > It is using default mail port, 25; I need port 465 or 587 because it's my > understanding google blocks everything on port 25. > During the setup, I was not (I don't think) given the option of specifying > smtps or the port. There's no preconfigured option for that setup. The satellite system option does relay on port 25. If you want Postfix to do submission, you need to set that up manually. Scott K
Re: setup issue -- debian /ubuntu 16.04.1 "bad string length 0 < 1: setgid_group ="
On 6/12/20 12:26 PM, Wietse Venema wrote: Gary Aitken: I had previously edited main.cf to set sendmail_path = /usr/sbin/sendmail setgid_group = postdrop $ sudo postfix check postfix: fatal: bad string length 0 < 1: mailq_path = Not sure what mailq_path should be set to... /var/spool/postfix/ ? http://www.postfix.org/postconf.5.html#mailq_path Sendmail compatibility feature that specifies where the Postfix mailq(1) command is installed. This command can be used to list the Postfix mail queue. Do all of these need to be set? I thought the re-configure should have taken care of this, and reasonable defaults would be applied? Do I need to remove main.cf before doing a dpkg-reconfigure? Perhaps you're better of with - uninstall Postfix - reinstall Postfix and only after doing that edit Postfix config files. A simple uninstall and reinstall of postfix could not be used, as the uninstall would remove another package (automysqlbackup) which depended on postfix. That package was already working, and I didn't want to disturb it. I got around that by installing nullmailer, which satisfied the default-mta and mail-transfer-agent dependencies from automysqlbackup; then uninstalled postfix without removing/uninstalling anything else. Then purge postfix to remove the config files, then reinstall postfix. So, now I have it installed "properly". yea! However... This is a satellite system which should receive no mail, but which needs to post mail from root and other daemon-users. I thought I had it set up by answering the setup questions as: Type of system: Satellite Domain: xbiologix.net mail relay: aspmx3.googlemail.com I have modified main.cf by adding: smtpd_tls_cert_file=/etc/letsencrypt/live/xbiologix.net/fullchain.pem smtpd_tls_key_file=/etc/letsencrypt/live/xbiologix.net/privkey.key ... smtp_tls_security_level = may smtp_tls_loglevel = 1 $ postconf -n alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases append_dot_mydomain = no biff = no html_directory = /usr/share/doc/postfix/html inet_interfaces = loopback-only inet_protocols = all mailbox_size_limit = 0 mydestination = myhostname = xblgx-ops.c.insidexblgx.internal mynetworks = 127.0.0.0/8 [:::127.0.0.0]/104 [::1]/128 myorigin = /etc/mailname readme_directory = /usr/share/doc/postfix recipient_delimiter = + relay_domains = relayhost = aspmx3.googlemail.com smtp_tls_loglevel = 1 smtp_tls_security_level = may smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu) smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination smtpd_tls_cert_file = /etc/letsencrypt/live/xbiologix.net/fullchain.pem smtpd_tls_key_file = /etc/letsencrypt/live/xbiologix.net/privkey.key smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtpd_use_tls = yes When attempting to send mail, /var/log/mail.log shows the following: Jun 16 17:18:11 xblgx-ops postfix/smtp[26224]: connect to aspmx3.googlemail.com[142.250.10.27]:25: Connection timed out It is using default mail port, 25; I need port 465 or 587 because it's my understanding google blocks everything on port 25. During the setup, I was not (I don't think) given the option of specifying smtps or the port. There is supposedly a default egress rule from google cloud vm instances, but I don't see it listed in my firewall rules. However, I think the listed firewall rules do not include the default *implied* rules imposed at a higher level; at least the google cloud docs kind of imply this: "Firewall rules must allow egress traffic from the instance. Unless overridden by a higher priority rule, the implied allow rule for egress traffic permits outbound traffic from all instances." In any case, I added an explicit rule to allow egress on port 465. tcpdump shows attempts going out, but on port 25, not 465. struggling along, any help much appreciated Gary
_restrictions
Hi, I'm cleaning up my postfix configs and am wondering if I can improve / should change my _restrictions on postfix 3.3 / 3.5: local postfix instance: smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated smtpd_helo_restrictions = permit_mynetworks permit_sasl_authenticated smtpd_sender_restrictions = permit_mynetworks permit_sasl_authenticated smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination smtpd_data_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_pipelining relaying instance inbound: smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated warn_if_reject check_client_access hash:/etc/postfix-in/client_access warn_if_reject reject_unknown_client_hostname warn_if_reject reject_unknown_reverse_client_hostname warn_if_reject reject_rbl_client ix.dnsbl.manitu.net warn_if_reject reject_rbl_client zen.spamhaus.org smtpd_helo_restrictions = permit_mynetworks permit_sasl_authenticated warn_if_reject check_helo_access hash:/etc/postfix-in/helo_access warn_if_reject reject_non_fqdn_helo_hostname warn_if_reject reject_invalid_helo_hostname warn_if_reject reject_unknown_helo_hostname smtpd_sender_restrictions = permit_mynetworks permit_sasl_authenticated warn_if_reject check_sender_access hash:/etc/postfix-in/sender_access warn_if_reject reject_non_fqdn_sender warn_if_reject reject_unknown_sender_domain smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unlisted_recipient reject_unauth_destination smtpd_data_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_pipelining smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination relaying instance outbound: smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated check_client_access cidr:/etc/postfix-out/client_access warn_if_reject reject_unknown_client_hostname warn_if_reject reject_unknown_reverse_client_hostname smtpd_helo_restrictions = permit_mynetworks permit_sasl_authenticated check_helo_access hash:/etc/postfix-out/helo_access reject_non_fqdn_helo_hostname reject_invalid_helo_hostname warn_if_reject reject_unknown_helo_hostname smtpd_sender_restrictions = permit_mynetworks permit_sasl_authenticated smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated check_client_access cidr:/etc/postfix-out/client_access reject_unauth_destination smtpd_data_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_pipelining smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated check_client_access cidr:/etc/postfix-out/client_access reject_unauth_destination Anything missing / redundant / unneccessary? Thank you!
Re: Postfix mailq priority
vi...@vheuser.com: > On 2020/06/16 13:42 PM, Wietse Venema wrote: > > vi...@vheuser.com: > >> Obviously I am above my pay grade here, > >> but can this? "Adding some artificial 'cost' value" currently be done? > > No. There is an existing solution that works for multi-recipient > > list mail. That solution does not work for single-recipient list mail, > > which we suspect is your use case. > > > > Viktor proposes to schedule single-recipient list mail from the > > same sender as if it is multi-recipient list mail (post-facto grouping). > > > > I'm proposing to just put a cost on each delivery request, initially > > based on how many other requests a sender already has in flight, > > that decays over time until the request is selected for delivery. > > This guarantees that all mail will eventually be delivered. > > > > This also would give us the option to make the scheduling dependent > > on message size, or quality-of-service indicators. > > > > Wietse > Just fyi - > Mailman domain concurrency is a yes/no/full option. > I've limited postfix to 5 and set Mailman to yes > and that sped things up dramatically > with no rejections from TWC. Perhaps you set some Postfix xxx_destination_recipient_limit=1. As documented that makes the Postfix concurrency limit a per-recipient property, which increases the total concurrency for a destinaion domain. Wietse
Re: Postfix mailq priority
On 2020/06/16 13:42 PM, Wietse Venema wrote: vi...@vheuser.com: Obviously I am above my pay grade here, but can this? "Adding some artificial 'cost' value" currently be done? No. There is an existing solution that works for multi-recipient list mail. That solution does not work for single-recipient list mail, which we suspect is your use case. Viktor proposes to schedule single-recipient list mail from the same sender as if it is multi-recipient list mail (post-facto grouping). I'm proposing to just put a cost on each delivery request, initially based on how many other requests a sender already has in flight, that decays over time until the request is selected for delivery. This guarantees that all mail will eventually be delivered. This also would give us the option to make the scheduling dependent on message size, or quality-of-service indicators. Wietse Just fyi - Mailman domain concurrency is a yes/no/full option. I've limited postfix to 5 and set Mailman to yes and that sped things up dramatically with no rejections from TWC.
Re: Postfix mailq priority
vi...@vheuser.com: > Obviously I am above my pay grade here, > but can this? "Adding some artificial 'cost' value" currently be done? No. There is an existing solution that works for multi-recipient list mail. That solution does not work for single-recipient list mail, which we suspect is your use case. Viktor proposes to schedule single-recipient list mail from the same sender as if it is multi-recipient list mail (post-facto grouping). I'm proposing to just put a cost on each delivery request, initially based on how many other requests a sender already has in flight, that decays over time until the request is selected for delivery. This guarantees that all mail will eventually be delivered. This also would give us the option to make the scheduling dependent on message size, or quality-of-service indicators. Wietse > >
Re: Postfix mailq priority
On Tue, Jun 16, 2020 at 01:25:41PM -0400, Wietse Venema wrote: > > I will look up how to create transports and assign mail to them. > > But it appears that there would be no way to redirect what is > > already in the queue. If you already have a lot of messages in the "active" queue, then those already are already resolve to a transport, but you can get the transport to be recomputed by putting all the messages "on hold", and then immediately releassing them (into the deferred queue) via "postsuper -H". > > Is that correct? > > The transport is NOT stored in the queue file. The transport map > is read every time the queue manager tries to deliver mail. More precisely, whenever the queue manager imports a message into the active queue and queues it for delivery. So recomputed each time a message is retried (from "deferred" or perhaps "hold"). -- Viktor.
Re: Postfix mailq priority
vi...@vheuser.com: > Postfix steadily delivers, but would like to de-prioritize the list emails > so regular traffic is not delayed. That's what we're talking about - making that automatic so you don't have to babysit Postfix. Otherwise, we'd be wasting everyone's time. > I will look up how to create transports and assign mail to them. > But it appears that there would be no way to redirect what is > already in the queue. > Is that correct? The transport is NOT stored in the queue file. The transport map is read every time the queue manager tries to deliver mail. Wietse
Re: Postfix mailq priority
On 2020/06/16 10:39 AM, Wietse Venema wrote: Viktor Dukhovni: On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote: Postfix has only one queue, and concurrency and process count is per-destination, not based on where the mail came from. Consider using a separate postfix instance for delivering mail list messages to prevent them from interfering with regular mail. See MULTI_INSTANCE_README There's one active queue, but within that active queue messages are queued to a particular (transport,nexthop) pair, and scheduling is round-robin by transport, and then FIFO, subject to concurrency and process limits and amortised pre?mption of multi-recipient messages by messages with fewer recipients, so that messages with lots of recipients don't hog the queue too long before some other messages with fewer recipients that arrived later get to use a delivery slot. If one just wants to put messages in multiple "lines" for delivery scheduling, but otherwise all settings are the same, then using multiple transports is simpler and often just as effective as multiple instances. The more Postfix can do by itself, the better. That could be: - Adding layer of sender-based round-robin selection. Not sure if that would explode at large scale. - Adding some artificial 'cost' value that is computed while delivery requests are added to per-destination queues. Cost could depend on the number of delivery requests per sender email address, the message size, and so on. Then, the scheduler could choose what-to-deliver based on artificial cost in addition to the things that it already considers now. Wietse Obviously I am above my pay grade here, but can this "Adding some artificial 'cost' value" currently be done? How?
Re: Postfix mailq priority
On 2020/06/16 11:32 AM, Viktor Dukhovni wrote: On Jun 16, 2020, at 12:39 PM, Wietse Venema wrote: - Adding layer of sender-based round-robin selection. Not sure if that would explode at large scale. When the input mix approximates steady-state, FIFO is essentially optimal, with each type of message getting its average share of the output in a fair and timely manner. If/when we stray from FIFO, we need to be carefuly not be simply underserving some fraction of the expected steady-state load distribution. This would mean being able to somehow detect a "burst" of traffic and characterise to distinguish the messages that are contributing to the burst from other messages. That's a tricky problem. The sender address for bulk traffic is liable to have VERP tags, the client IP of interest may be a few hops back from the edge system that encounters delays in delivering an input burst to remote organizations (ADMDs in the language of RFC5598). - Adding some artificial 'cost' value that is computed while delivery requests are added to per-destination queues. Cost could depend on the number of delivery requests per sender email address, the message size, and so on. Then, the scheduler could choose what-to-deliver based on artificial cost in addition to the things that it already considers now. Ideally we'd have an algorithm that could group related messages into a set of logical "bulk" sources, and apply the current bulk message preëmption algorithm not only to multi-recipient mail, but also to multi-message bulk sources. The hard part is the classification, especially in a single-threaded queue manager that needs to do this in O(1 millisecond). Perhaps the best proxy for related messages is origin domain (ignoring localpart) + approximate message size. Related messages are likely to carry similar content of approximately the same size to all recipients. But completely ignoring the localpart may be too coarse. It is not obvious to me how to extract common elements from a per-recipient-salted localpart... A traditional multi-recipient message automatically qualifies as a single source (same message size for all recipients and same origin domain). We could attempt to group "related" messages in a fuzzy manner as above, and then apply the existing preëmption algorithm. Still not sure how to do a good job of the aggregation. I've got a lot to learn here. Didn't expect so much detail! Ideally, one could flag the big list somehow and not worry about the little ones. I turned off concurrency because TWC.com was rejecting bundles of emails. Serious slow down. Postfix steadily delivers, but would like to de-prioritize the list emails so regular traffic is not delayed. I will look up how to create transports and assign mail to them. But it appears that there would be no way to redirect what is already in the queue. Is that correct? (Sheepish grin) Feature request? (-:
Re: Postfix mailq priority
Viktor Dukhovni: > > On Jun 16, 2020, at 12:39 PM, Wietse Venema wrote: > > > > - Adding layer of sender-based round-robin selection. Not sure if > > that would explode at large scale. > > When the input mix approximates steady-state, FIFO is essentially optimal, > with each type of message getting its average share of the output in a fair > and timely manner. If/when we stray from FIFO, we need to be carefuly not > be simply underserving some fraction of the expected steady-state load > distribution. Specifically, we don't want to forever deprioritize a request, and that is usually done over time by decreasing the thing that causes a request to be postponed. In the next approach, artificial cost, we would monotonically decrease the artificial cost over time so that a delivery request is not permanently starved. > This would mean being able to somehow detect a "burst" of traffic and > characterise to distinguish the messages that are contributing to the > burst from other messages. That's a tricky problem. The sender address > for bulk traffic is liable to have VERP tags, the client IP of interest > may be a few hops back from the edge system that encounters delays in > delivering an input burst to remote organizations (ADMDs in the language > of RFC5598). Of course any sender-based metrics would be based on sender address without the address extension (i.e. the part that VERP adds). > > - Adding some artificial 'cost' value that is computed while delivery > > requests are added to per-destination queues. Cost could depend on > > the number of delivery requests per sender email address, the message > > size, and so on. Then, the scheduler could choose what-to-deliver > > based on artificial cost in addition to the things that it already > > considers now. > > Ideally we'd have an algorithm that could group related messages into > a set of logical "bulk" sources, and apply the current bulk message > pre?mption algorithm not only to multi-recipient mail, but also to > multi-message bulk sources. The hard part is the classification, > especially in a single-threaded queue manager that needs to do this > in O(1 millisecond). With artificial cost done well, it should make little difference if a message has 1 recipients in distinct domains, or if those same recipients are enqueued in 1 single-recipient submissions. In both cases we would have 1 delivery requests, and in both cases the artificial cost would be similar. I think that would be an improvement. I'd like to avoid grouping as too diffult, and consider artificial cost (with monotonic decrease) as more feasible, provided that the 'current artificial cost' metric is easily updated (for example a hash table indexed by sender that counts the number of delivery requests with that sender). Not that the number of delivery requests measures performance impact better than number of recipients. Each new delivery request is then assigned an artificial cost that recflects the 'current cost' metrics at that point in time. Over time, the artificial cost for that delivery request is decreased monotonically so that it will not be starved. > Perhaps the best proxy for related messages is origin domain (ignoring > localpart) + approximate message size. Related messages are likely to > carry similar content of approximately the same size to all recipients. > But completely ignoring the localpart may be too coarse. It is not > obvious to me how to extract common elements from a per-recipient-salted > localpart... > > A traditional multi-recipient message automatically qualifies as a single > source (same message size for all recipients and same origin domain). See my above comment abount the equivalence of multi-recipient list mail versus single-recipient. Look, Mom! No grouping needed. Wietse > We could attempt to group "related" messages in a fuzzy manner as above, > and then apply the existing pre?mption algorithm. Still not sure how to > do a good job of the aggregation. > > -- > -- > Viktor. > >
Re: Postfix mailq priority
> On Jun 16, 2020, at 12:39 PM, Wietse Venema wrote: > > - Adding layer of sender-based round-robin selection. Not sure if > that would explode at large scale. When the input mix approximates steady-state, FIFO is essentially optimal, with each type of message getting its average share of the output in a fair and timely manner. If/when we stray from FIFO, we need to be carefuly not be simply underserving some fraction of the expected steady-state load distribution. This would mean being able to somehow detect a "burst" of traffic and characterise to distinguish the messages that are contributing to the burst from other messages. That's a tricky problem. The sender address for bulk traffic is liable to have VERP tags, the client IP of interest may be a few hops back from the edge system that encounters delays in delivering an input burst to remote organizations (ADMDs in the language of RFC5598). > - Adding some artificial 'cost' value that is computed while delivery > requests are added to per-destination queues. Cost could depend on > the number of delivery requests per sender email address, the message > size, and so on. Then, the scheduler could choose what-to-deliver > based on artificial cost in addition to the things that it already > considers now. Ideally we'd have an algorithm that could group related messages into a set of logical "bulk" sources, and apply the current bulk message preëmption algorithm not only to multi-recipient mail, but also to multi-message bulk sources. The hard part is the classification, especially in a single-threaded queue manager that needs to do this in O(1 millisecond). Perhaps the best proxy for related messages is origin domain (ignoring localpart) + approximate message size. Related messages are likely to carry similar content of approximately the same size to all recipients. But completely ignoring the localpart may be too coarse. It is not obvious to me how to extract common elements from a per-recipient-salted localpart... A traditional multi-recipient message automatically qualifies as a single source (same message size for all recipients and same origin domain). We could attempt to group "related" messages in a fuzzy manner as above, and then apply the existing preëmption algorithm. Still not sure how to do a good job of the aggregation. -- -- Viktor.
Re: Postfix mailq priority
Viktor Dukhovni: > On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote: > > > Postfix has only one queue, and concurrency and process count is > > per-destination, not based on where the mail came from. > > > > Consider using a separate postfix instance for delivering mail list > > messages to prevent them from interfering with regular mail. See > > MULTI_INSTANCE_README > > There's one active queue, but within that active queue messages are > queued to a particular (transport,nexthop) pair, and scheduling is > round-robin by transport, and then FIFO, subject to concurrency and > process limits and amortised pre?mption of multi-recipient messages by > messages with fewer recipients, so that messages with lots of recipients > don't hog the queue too long before some other messages with fewer > recipients that arrived later get to use a delivery slot. > > If one just wants to put messages in multiple "lines" for delivery > scheduling, but otherwise all settings are the same, then using multiple > transports is simpler and often just as effective as multiple instances. The more Postfix can do by itself, the better. That could be: - Adding layer of sender-based round-robin selection. Not sure if that would explode at large scale. - Adding some artificial 'cost' value that is computed while delivery requests are added to per-destination queues. Cost could depend on the number of delivery requests per sender email address, the message size, and so on. Then, the scheduler could choose what-to-deliver based on artificial cost in addition to the things that it already considers now. Wietse