Re: PATCH: Keep Postfix running in the foreground

[Wietse Venema]

Eray Aslan:
> On Tue, Dec 19, 2017 at 10:53:38AM -0500, Wietse Venema wrote:
> > Postfix will bail out if it knows that the queue or data directory
> > are shared, because that can result in data corruption.
> > 
> > How do I enforce that constraint when directories are imported into
> > a container from the host?
> 
> I don't think you can.  But how is that different from the current
> status quo?  We can only continue doing what we have been doing so far.

The status quo (prior to containers) is that Postfix prevents
unnecessary requests for support from people who do things like
sharing a queue_directory or data_directory.

The Postfix master relies on opening a 'lock' file below the queue
and data directories, and holding an exclusive lock until the process
terminates. If that works across container boundaries, then that
is one less thing to worry about. But all this is speculation until
it is proven to work.

Wietse

Re: PATCH: Keep Postfix running in the foreground

[Eray Aslan]

On Tue, Dec 19, 2017 at 10:53:38AM -0500, Wietse Venema wrote:
> Postfix will bail out if it knows that the queue or data directory
> are shared, because that can result in data corruption.
> 
> How do I enforce that constraint when directories are imported into
> a container from the host?

I don't think you can.  But how is that different from the current
status quo?  We can only continue doing what we have been doing so far.

For the record, while running postfix in a container is certainly doable
and should probably be given as an option, I am not convinced that it is
such a good idea for any serious MTA work.

-- 
Eray

Re: PATCH: Keep Postfix running in the foreground

[Wietse Venema]

Viktor Dukhovni:
> 
> 
> > On Dec 19, 2017, at 10:19 AM, Eray Aslan  wrote:
> > 
> > Usually sending log output to console is the preferred approach.  If we
> > cannot do it natively, having syslog daemon write to console (in
> > addition to local log file?) looks like a better option than importing
> > sockets from the host.
> 
> Not an option for Postfix.  Multiple processes generate log messages
> concurrently, a shared output stream would result in interleaving of
> message fragments.  The syslogd service receives discrete messages
> and serializes output.  It is important to use a datagram syslog
> service, to preserve chronological message order.

It should be possible to run an in-container process that receives
syslog datagrams from all Postfix daemons, and that sends complete
text records one-by-one, as long as there is only one thread that
writes to stdout/stderr.

However, I have concerns about the performance cost of multiple
layers of forwarding that handles the logging from hundreds of
concurrent Postfix daemon processes, on top of a Linux logging
infrastructure that is already insanely expensive.

Wietse

Re: PATCH: Keep Postfix running in the foreground

[Viktor Dukhovni]

> On Dec 19, 2017, at 10:19 AM, Eray Aslan  wrote:
> 
> Usually sending log output to console is the preferred approach.  If we
> cannot do it natively, having syslog daemon write to console (in
> addition to local log file?) looks like a better option than importing
> sockets from the host.

Not an option for Postfix.  Multiple processes generate log messages
concurrently, a shared output stream would result in interleaving of
message fragments.  The syslogd service receives discrete messages
and serializes output.  It is important to use a datagram syslog
service, to preserve chronological message order.

-- 
Viktor.

Re: PATCH: Keep Postfix running in the foreground

[Wietse Venema]

Eray Aslan:
> On Tue, Dec 19, 2017 at 10:01:53AM -0500, Wietse Venema wrote:
> > I suppose one approach is to make a Postfix container disposable,
> > i.e. a container is never updated with a new Postfix version, but
> > it is replaced with a newer one
> 
> That is the common Docker approach.  Images are immutable.
> 
> > and it imports its queue and data directories from the host. These
> > directories must of course not be imported into multiple containers. I
> > don't know how to prevent that.
> 
> That is a problem for a different layer of the stack.  The sysadmin is
> supposed to provide persistent storage and make sure that multiple
> containers do not write to the same directory.  It should not be our job
> to babysit the infra.  Inform and bail out if the deal is broken?

Postfix will bail out if it knows that the queue or data directory
are shared, because that can result in data corruption.

How do I enforce that constraint when directories are imported into
a container from the host?

> > Also, a Postfix container would import the logging sockets from the
> > host (www.projectatomic.io/blog/2016/10/playing-with-docker-logging)
> > and would set 'syslog_name = $myhostname/postfix' in the container's
> > main.cf file to make logging from different containers distinct.
> > Of course the logging sockets may be imported into as many containers
> > as needed.
> 
> Uhm, systemd (or any other init system) as pid 1 is not the "docker
> way".  It is better for docker to know when the service has stopped /
> crashed etc so it can take appropriate action.  So consider ditching
> seperate pid 1 daemon option.

THIS ARTICLE IS NOT ABOUT RUNNING SYSTEMD IN THE CONTAINER. 
Or any other non-Postfix daemon, for that matter.

Wietse

Re: PATCH: Keep Postfix running in the foreground

[Eray Aslan]

On Tue, Dec 19, 2017 at 10:01:53AM -0500, Wietse Venema wrote:
> I suppose one approach is to make a Postfix container disposable,
> i.e. a container is never updated with a new Postfix version, but
> it is replaced with a newer one

That is the common Docker approach.  Images are immutable.

> and it imports its queue and data directories from the host. These
> directories must of course not be imported into multiple containers. I
> don't know how to prevent that.

That is a problem for a different layer of the stack.  The sysadmin is
supposed to provide persistent storage and make sure that multiple
containers do not write to the same directory.  It should not be our job
to babysit the infra.  Inform and bail out if the deal is broken?

> Also, a Postfix container would import the logging sockets from the
> host (www.projectatomic.io/blog/2016/10/playing-with-docker-logging)
> and would set 'syslog_name = $myhostname/postfix' in the container's
> main.cf file to make logging from different containers distinct.
> Of course the logging sockets may be imported into as many containers
> as needed.

Uhm, systemd (or any other init system) as pid 1 is not the "docker
way".  It is better for docker to know when the service has stopped /
crashed etc so it can take appropriate action.  So consider ditching
seperate pid 1 daemon option.

Usually sending log output to console is the preferred approach.  If we
cannot do it natively, having syslog daemon write to console (in
addition to local log file?) looks like a better option than importing
sockets from the host.

> If one wants multiple Postfix instances in a single container, then
> that will require a 'minder' program that runs in the foreground and
> that plays nice with higher-level orchestration systems. I won't
> sabotage that approach.

Do we really need that?  Too many layers all trying to do a similar job?

-- 
Eray

PATCH: Keep Postfix running in the foreground

[Wietse Venema]

Wietse Venema:
> I think that Docker fundamentally wants one service instance per
> container. On Postfix service instance translates into one queue,
> for example submission+smtp sharing one queue, similar to http+https
> sharing one website. Let's not fight the Docker approach, and leave
> orchestration to a different layer in the stack.
...
> As for forgrounding, this must happen only after the 'postfix
> check' sanity checks and repairs complete sucessfully. Running a
> 'bare' master daemon would violate design assumptions. So this
> will require a new 'postfix' subcommand that starts exactly one
> instance in the foreground.

Attached is a proof-of-concept implementation.
Manpage fragment:

POSTFIX(1)  POSTFIX(1)

NAME
   postfix - Postfix control program
...
DESCRIPTION
...
   The following commands are implemented:

   check  Warn about bad directory/file ownership or permissions, and cre-
  ate missing directories.

   start  Start  the Postfix mail system. This also runs the configuration
  check described above.

   start-fg
  Like start, but keep the master  daemon  running  in  the  fore-
  ground.  This  requires  that multi-instance support is disabled
  (i.e. the multi_instance_directories parameter value is  empty).
...

I suppose one approach is to make a Postfix container disposable,
i.e. a container is never updated with a new Postfix version, but
it is replaced with a newer one, and it imports its queue and data
directories from the host. These directories must of course not be
imported into multiple containers. I don't know how to prevent that.

Also, a Postfix container would import the logging sockets from the
host (www.projectatomic.io/blog/2016/10/playing-with-docker-logging)
and would set 'syslog_name = $myhostname/postfix' in the container's
main.cf file to make logging from different containers distinct.
Of course the logging sockets may be imported into as many containers
as needed.

If one wants multiple Postfix instances in a single container, then
that will require a 'minder' program that runs in the foreground and
that plays nice with higher-level orchestration systems. I won't
sabotage that approach.

Wietse

diff --exclude=man --exclude=html --exclude=README_FILES --exclude=INSTALL 
--exclude=.indent.pro --exclude=Makefile.in -r -ur 
/var/tmp/postfix-3.3-20171218/conf/postfix-script ./conf/postfix-script
--- /var/tmp/postfix-3.3-20171218/conf/postfix-script   2017-09-12 
14:16:54.0 -0400
+++ ./conf/postfix-script   2017-12-19 09:37:18.0 -0500
@@ -117,7 +117,7 @@
echo "Stop postfix"
;;
 
-start)
+start|start-fg)
 
$daemon_directory/master -t 2>/dev/null || {
$FATAL the Postfix mail system is already running
@@ -135,11 +135,28 @@
$daemon_directory/postfix-script check-warn
fi
$INFO starting the Postfix mail system
-   # NOTE: wait in foreground process to get the initialization status.
-   $daemon_directory/master -w || {
-   $FATAL "mail system startup failed"
-   exit 1
-   }
+   case $1 in
+   start)
+   # NOTE: wait in foreground process to get the initialization status.
+   $daemon_directory/master -w || {
+   $FATAL "mail system startup failed"
+   exit 1
+   }
+   ;;
+   start-fg)
+   # Foreground start-up is incompatible with multi-instance mode.
+   # We can't use "exec $daemon_directory/master" here: that would
+   # break process group management, and "postfix stop" would kill
+   # too many processes.
+   case $instances in
+   "") $daemon_directory/master
+   ;;
+*) $FATAL "start-fg does not support multi_instance_directories"
+   exit 1
+   ;;
+   esac
+   ;;
+   esac
;;
 
 drain)
diff --exclude=man --exclude=html --exclude=README_FILES --exclude=INSTALL 
--exclude=.indent.pro --exclude=Makefile.in -r -ur 
/var/tmp/postfix-3.3-20171218/src/postfix/postfix.c ./src/postfix/postfix.c
--- /var/tmp/postfix-3.3-20171218/src/postfix/postfix.c 2016-09-17 
10:50:56.0 -0400
+++ ./src/postfix/postfix.c 2017-12-19 09:19:59.0 -0500
@@ -31,6 +31,11 @@
 /* .IP \fBstart\fR
 /* Start the Postfix mail system. This also runs the configuration
 /* check described above.
+/* .IP \fBstart-fg\fR
+/* Like \fBstart\fR, but keep the master daemon running in the
+/* foreground. This requires that multi-instance support is
+/* disabled (i.e. the multi_instance_directories parameter
+/* value is empty).
 /* .IP \fBstop\fR
 /* Stop the Postfix mail system in an orderly fashion. If
 /* possible, running processes are allowed to terminate at