Re: Dumping effective postfix configuration
Viktor Dukhovni: > > The Postfix master creates as many sockets as needed for an inet > > service, and sets the socket option SO_REUSEADDR on each socket, > > and bind()s them to the desired addresses. > > > > With that, it may be possible for the Postfix master to create new > > sockets, set SO_REUSEADDR on each socket, and bind() them to the > > new addresses; if successful, close() the old sockets, otherwise > > close() the new sockets. > > Yes, typically doable, by determining which bindings are unchanged, > which new, and which old. > > > Am I missing something? This will have to work on all supported > > systems, though. Not just Linux, not just *BSD. > > The tricky case is changing a wildcard binding to a set of non-wildcard > bindings. When a wildcard binding is active, it is AFAIK not generally > possible to bind non-wildcard listeners for the same port, and > presumably the converse. > > Changes in explicit lists of IPs should be viable as you noted. The configurations of interest are: inet_interfaces = all (wildcard binding) inet_interfaces = enumeration (non-wildcard binding) Changes of type wildcard->wildcard are noops. Changes of type enumeration->enumeration are partial noops if one only touches the sockets whose addresses have changed. That leaves wildcard->enumeration and enumeration->wildcard. Both *should* be safe on BSD-ish (and Solaris, HP-UX) systems, given how SO_REUSEADDR was originally implemented. See the "BSD SO_REUSEADDR" discussion with a nice table in: https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ This also comes with a nice test program. With Linux the changes of type wildcard->enum and enum->wildcard require code that is kernel version dependent. If we consider only kernels >= 3.9 then we can keep it simple with SO_REUSEPORT. All of this needs to be evaluated not only on an idle server where 'listen' sockets are open only in the master, but also with child process that hold a reference to a master's listen socket as well as connected sockets. And then it needs to deal with sockets that are in the TIME_WAIT state. Wietse
Re: Dumping effective postfix configuration
Demi Marie Obenour: Checking application/pgp-signature: FAILURE -- Start of PGP signed section. > On 1/6/23 07:15, Wietse Venema wrote: > > Peter Wienemann: > >> Hi, > >> > >> is there a way to dump the effective postfix configuration rather than > >> the one specified in main.cf/master.cf? It seems that changes to > >> main.cf/master.cf have an immediate impact on the output of postconf > >> regardless of whether a reload/restart of the service was performed. > > > > How would that work? Different Postfix processes start at different > > times, and some processes (qmgr, master) run forever. > > What about using inotify/kqueue to determine when a config file has > changed and reload accordingly? It is not necessarily desirable that a daemon refreshes as soon as one file has changed, especially when multiple files need to be updated. I think that it is not burdensome to "postfix reload". kqueue and inotify were introduced to BSD and Linux years after Postfix was already released. They have no equivalent on other systems, and MacOS kqueue has a problematic history with Postfix. Wietse
Re: Dumping effective postfix configuration
On Fri, Jan 06, 2023 at 03:28:23PM -0500, Wietse Venema wrote: > > The documentation of inet_interfaces says otherwise, my recollection is > > that master(8) does not attempt to rebind inet sockets on "reload": > > I see. Yes, that could be dififcult. Or maybe not. > > The Postfix master creates as many sockets as needed for an inet > service, and sets the socket option SO_REUSEADDR on each socket, > and bind()s them to the desired addresses. > > With that, it may be possible for the Postfix master to create new > sockets, set SO_REUSEADDR on each socket, and bind() them to the > new addresses; if successful, close() the old sockets, otherwise > close() the new sockets. Yes, typically doable, by determining which bindings are unchanged, which new, and which old. > Am I missing something? This will have to work on all supported > systems, though. Not just Linux, not just *BSD. The tricky case is changing a wildcard binding to a set of non-wildcard bindings. When a wildcard binding is active, it is AFAIK not generally possible to bind non-wildcard listeners for the same port, and presumably the converse. Changes in explicit lists of IPs should be viable as you noted. -- Viktor.
Re: Dumping effective postfix configuration
Le 06/01/2023 à 21:03, Demi Marie Obenour a écrit : On 1/6/23 07:15, Wietse Venema wrote: Peter Wienemann: Hi, is there a way to dump the effective postfix configuration rather than the one specified in main.cf/master.cf? It seems that changes to main.cf/master.cf have an immediate impact on the output of postconf regardless of whether a reload/restart of the service was performed. How would that work? Different Postfix processes start at different times, and some processes (qmgr, master) run forever. What about using inotify/kqueue to determine when a config file has changed and reload accordingly? Too dangerous and useless. I want to issue reload for long running processes only when I want. Periodically recycled process will take automatically the new conf when they restarts, inotify/kqueue is not needed for that. That is something you must know as a postfix user to not inadvertently crash your running config with half edited but saved config files. Same expectations and warning for maps (according to their type). For the conf files, one interesting change would be to do the opposite : Process only run with copies of validated conf files and new copies (+ process reload) are done only when explicit reload command is issued to avoid new conf taken by surprise on process recycling. Emmanuel.
Re: Dumping effective postfix configuration
Viktor Dukhovni: > On Fri, Jan 06, 2023 at 01:45:48PM -0500, Wietse Venema wrote: > > > > Most changes in main.cf require a reload to become effective but there > > > are some exceptions requiring a restart, e. g. a change of > > > inet_interfaces. > > > > That is incorrect. If inet_interfaces is changed, then a "postfix > > reload" (not restart) is required because the master daemon is a > > long-running process. > > The documentation of inet_interfaces says otherwise, my recollection is > that master(8) does not attempt to rebind inet sockets on "reload": I see. Yes, that could be dififcult. Or maybe not. The Postfix master creates as many sockets as needed for an inet service, and sets the socket option SO_REUSEADDR on each socket, and bind()s them to the desired addresses. With that, it may be possible for the Postfix master to create new sockets, set SO_REUSEADDR on each socket, and bind() them to the new addresses; if successful, close() the old sockets, otherwise close() the new sockets. Am I missing something? This will have to work on all supported systems, though. Not just Linux, not just *BSD. Wietse
Re: Dumping effective postfix configuration
On 1/6/23 07:15, Wietse Venema wrote: > Peter Wienemann: >> Hi, >> >> is there a way to dump the effective postfix configuration rather than >> the one specified in main.cf/master.cf? It seems that changes to >> main.cf/master.cf have an immediate impact on the output of postconf >> regardless of whether a reload/restart of the service was performed. > > How would that work? Different Postfix processes start at different > times, and some processes (qmgr, master) run forever. What about using inotify/kqueue to determine when a config file has changed and reload accordingly? -- Sincerely, Demi Marie Obenour (she/her/hers) OpenPGP_0xB288B55FFF9C22C1.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: Dumping effective postfix configuration
On 06.01.23 20:02, Viktor Dukhovni wrote: On Fri, Jan 06, 2023 at 01:45:48PM -0500, Wietse Venema wrote: Most changes in main.cf require a reload to become effective but there are some exceptions requiring a restart, e. g. a change of inet_interfaces. That is incorrect. If inet_interfaces is changed, then a "postfix reload" (not restart) is required because the master daemon is a long-running process. The documentation of inet_interfaces says otherwise, my recollection is that master(8) does not attempt to rebind inet sockets on "reload": src/master/master_ent.c: /* src/master/master_ent.c: * XXX We cannot change the inet_interfaces setting for a running master src/master/master_ent.c: * process. Listening sockets are inherited by child processes so that src/master/master_ent.c: * closing and reopening those sockets in the master does not work. src/master/master_ent.c: * src/master/master_ent.c: * Another problem is that library routines still cache results that are src/master/master_ent.c: * based on the old inet_interfaces setting. It is too much trouble to src/master/master_ent.c: * recompute everything. src/master/master_ent.c: * src/master/master_ent.c: * In order to keep our data structures consistent we ignore changes in src/master/master_ent.c: * inet_interfaces settings, and issue a warning instead. src/master/master_ent.c: */ If one really wants to change the IP address and/or number of listen endpoints, a restart is required. This also matches the warnings in the logs. After a reload following a change of inet_interfaces the logs contain the following warning: --- postfix/master[58283]: reload -- version 3.5.17, configuration /etc/postfix postfix/master[58283]: warning: service smtp: ignoring inet_interfaces change postfix/master[58283]: warning: to change inet_interfaces, stop and start Postfix --- Peter
Re: Dumping effective postfix configuration
On Fri, Jan 06, 2023 at 01:45:48PM -0500, Wietse Venema wrote: > > Most changes in main.cf require a reload to become effective but there > > are some exceptions requiring a restart, e. g. a change of > > inet_interfaces. > > That is incorrect. If inet_interfaces is changed, then a "postfix > reload" (not restart) is required because the master daemon is a > long-running process. The documentation of inet_interfaces says otherwise, my recollection is that master(8) does not attempt to rebind inet sockets on "reload": src/master/master_ent.c: /* src/master/master_ent.c: * XXX We cannot change the inet_interfaces setting for a running master src/master/master_ent.c: * process. Listening sockets are inherited by child processes so that src/master/master_ent.c: * closing and reopening those sockets in the master does not work. src/master/master_ent.c: * src/master/master_ent.c: * Another problem is that library routines still cache results that are src/master/master_ent.c: * based on the old inet_interfaces setting. It is too much trouble to src/master/master_ent.c: * recompute everything. src/master/master_ent.c: * src/master/master_ent.c: * In order to keep our data structures consistent we ignore changes in src/master/master_ent.c: * inet_interfaces settings, and issue a warning instead. src/master/master_ent.c: */ If one really wants to change the IP address and/or number of listen endpoints, a restart is required. -- Viktor.
Re: Dumping effective postfix configuration
Peter Wienemann: > Most changes in main.cf require a reload to become effective but there > are some exceptions requiring a restart, e. g. a change of > inet_interfaces. That is incorrect. If inet_interfaces is changed, then a "postfix reload" (not restart) is required because the master daemon is a long-running process. Similar considerations apply when changes are made that affect the queue mamager process (for example, changes that affect concurrency management). Other Postfix processes have a finite life time, and will eventually pick up configuration changes. If you can't wait, then use "postfix reload". Wietse
Re: Dumping effective postfix configuration
On Fri, Jan 06, 2023 at 06:42:18PM +0100, Peter Wienemann wrote: > Most changes in main.cf require a reload to become effective No. In most cases they merely require the passage of time, and typical changes are not that time-sensitive. > I am trying to put main.cf under configuration > management and have to trigger an action each time main.cf changes such > that the changes become effective. If you make changes to main.cf that are time-sensitive, push a separate file whose update will cause a reload. If your changes require a restart, push a separate file whose update triggers a restart. Most of the time leave both sentinel files alone. > At the moment I always perform a restart after a change of main.cf > (just to be on the safe side). This is not necessary in most cases, and should be avoided if possible. > But on a busy mail server this is not ideal. I am looking for a way to > do better. One idea that came to my mind is: Reloads are also not ideal. > 1. Initially try a reload and check whether the effective configuration > is different from the target configuration as specified in main.cf. > > 2. If both match, we are done. If there is a discrepancy, launch a restart. You're over-engineering the problem. Let the operator who authors the change decide whether a restart or reload is required, or whether just normal Postfix process recycling is sufficient for your needs. On a busy server, you can reduce max_idle and or max_use to recycle processes more often: max_idle = 3s max_use = 20 Most smtpd(8) processes and delivery agents would last not much longer than a minute before they exit and are replaced. The additional cost of process creation will be modest. -- Viktor.
Re: Dumping effective postfix configuration
On 06.01.23 17:33, Wietse Venema wrote: Peter Wienemann: On 06.01.23 13:15, Wietse Venema wrote: is there a way to dump the effective postfix configuration rather than the one specified in main.cf/master.cf? It seems that changes to main.cf/master.cf have an immediate impact on the output of postconf regardless of whether a reload/restart of the service was performed. How would that work? Different Postfix processes start at different times, and some processes (qmgr, master) run forever. I could think of something like this: Instead of sending a HUP signal like it is done to trigger a reload, one sends a different signal (e. g. USR1) to the master. The further processing of the signal is similar to the HUP case except that configuration dumps are triggered rather than configuration reloads. How could that be useful? On a busy server, there will be be 100+ smtpd (SMTP server) processes, 100+ cleanup processes, 100+ smtp (SMTP client) processes and so on. Those processes were started an different times and they will produce different dumps if you change Postfix configuration without "postfix reload". If you want an accurate picture of what configuration is in effect. then I think that is not too much trouble to run "postfix reload" after a configuration change. Then "postconf" output reflects the effective configuration. Instead of dumping configurations, there are other ways. For example, each daemon could log configuration file stamps when the process is started (or communicate that info in some other manner). Given a history of (time stamp, config file content) you could use the logged time stamps to find out what configuration is (or was) in effect for each daemon process. I am trying to solve the following problem: Most changes in main.cf require a reload to become effective but there are some exceptions requiring a restart, e. g. a change of inet_interfaces. I am trying to put main.cf under configuration management and have to trigger an action each time main.cf changes such that the changes become effective. At the moment I always perform a restart after a change of main.cf (just to be on the safe side). But on a busy mail server this is not ideal. I am looking for a way to do better. One idea that came to my mind is: 1. Initially try a reload and check whether the effective configuration is different from the target configuration as specified in main.cf. 2. If both match, we are done. If there is a discrepancy, launch a restart. That is how I ended up with my initial question. Maybe there are better/simpler ways to accomplish the desired result. Suggestions are welcome. Peter
Re: Dumping effective postfix configuration
On Fri, Jan 06, 2023 at 11:33:28AM -0500, Wietse Venema wrote:
> > Instead of sending a HUP signal like it is done to trigger a reload, one
> > sends a different signal (e. g. USR1) to the master. The further
> > processing of the signal is similar to the HUP case except that
> > configuration dumps are triggered rather than configuration reloads.
>
> How could that be useful? On a busy server, there will be be 100+
> smtpd (SMTP server) processes, 100+ cleanup processes, 100+ smtp
> (SMTP client) processes and so on. Those processes were started an
> different times and they will produce different dumps if you change
> Postfix configuration without "postfix reload".
+1. It makes little sense to ask a distributed (over multiple
dynamically spawned processes) system about its "current" state.
The age in seconds of master and its descendants can be obtained via,
for example:
$ ps -c -o etimes= -o pid= -o command= -p $(pgrep -g $(ps -o pgid= $(pgrep
-x master))) | sort -nr
2466115 87674 master
384351 70993 qmgr
384315 71008 tlsmgr
2402 39744 pickup
368 39967 anvil
368 39965 smtpd
52 40089 proxymap
52 40088 smtpd
14 40098 smtpd
12 40099 trivial-rewrite
Even fancier, splice in the age the main.cf and master.cf files:
$ (
ps -c -o etimes= -o pid= -o command= -p $(pgrep -g $(ps -o pgid= $(pgrep -x
master)))
perl -le '
use File::stat;
for (@ARGV) {printf "%d - %s\n", time - stat($_)->mtime, $_}
' /usr/local/etc/postfix/ma*.cf
) | awk '{printf "%10s %10s %s\n", $1, $2, $3}' | sort -nr
28755246 - /usr/local/etc/postfix/master.cf
2467086 87674 master
385324 - /usr/local/etc/postfix/main.cf
385322 70993 qmgr
385286 71008 tlsmgr
3373 39744 pickup
1 40259 anvil
1 40258 proxymap
1 40257 smtpd
0 40260 trivial-rewrite
The master(8) process is younger than master.cf and the rest are younger
than main.cf, so all is well, assuming that no critical changes
affecting master(8) were made in main.cf (inet_interfaces, ...).
--
Viktor.
Re: Dumping effective postfix configuration
Peter Wienemann: > On 06.01.23 13:15, Wietse Venema wrote: > >> is there a way to dump the effective postfix configuration rather than > >> the one specified in main.cf/master.cf? It seems that changes to > >> main.cf/master.cf have an immediate impact on the output of postconf > >> regardless of whether a reload/restart of the service was performed. > > > > How would that work? Different Postfix processes start at different > > times, and some processes (qmgr, master) run forever. > > I could think of something like this: > > Instead of sending a HUP signal like it is done to trigger a reload, one > sends a different signal (e. g. USR1) to the master. The further > processing of the signal is similar to the HUP case except that > configuration dumps are triggered rather than configuration reloads. How could that be useful? On a busy server, there will be be 100+ smtpd (SMTP server) processes, 100+ cleanup processes, 100+ smtp (SMTP client) processes and so on. Those processes were started an different times and they will produce different dumps if you change Postfix configuration without "postfix reload". If you want an accurate picture of what configuration is in effect. then I think that is not too much trouble to run "postfix reload" after a configuration change. Then "postconf" output reflects the effective configuration. Instead of dumping configurations, there are other ways. For example, each daemon could log configuration file stamps when the process is started (or communicate that info in some other manner). Given a history of (time stamp, config file content) you could use the logged time stamps to find out what configuration is (or was) in effect for each daemon process. Wietse
Re: Dumping effective postfix configuration
On 06.01.23 13:15, Wietse Venema wrote: is there a way to dump the effective postfix configuration rather than the one specified in main.cf/master.cf? It seems that changes to main.cf/master.cf have an immediate impact on the output of postconf regardless of whether a reload/restart of the service was performed. How would that work? Different Postfix processes start at different times, and some processes (qmgr, master) run forever. I could think of something like this: Instead of sending a HUP signal like it is done to trigger a reload, one sends a different signal (e. g. USR1) to the master. The further processing of the signal is similar to the HUP case except that configuration dumps are triggered rather than configuration reloads. What I have in mind would be a similar user experience to what one gets when running "ceph config show [...]" [0] in the Ceph case or "condor_config_val [daemon]" [1] in case of HTCondor. Peter [0] https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#commands [1] https://htcondor.readthedocs.io/en/latest/man-pages/condor_config_val.html
Re: Dumping effective postfix configuration
Peter Wienemann: > Hi, > > is there a way to dump the effective postfix configuration rather than > the one specified in main.cf/master.cf? It seems that changes to > main.cf/master.cf have an immediate impact on the output of postconf > regardless of whether a reload/restart of the service was performed. How would that work? Different Postfix processes start at different times, and some processes (qmgr, master) run forever. Wietse
Dumping effective postfix configuration
Hi, is there a way to dump the effective postfix configuration rather than the one specified in main.cf/master.cf? It seems that changes to main.cf/master.cf have an immediate impact on the output of postconf regardless of whether a reload/restart of the service was performed. Peter
