We recently switched our Postfix mail servers to Ubuntu Server 18, which
uses journald for logging. Since we have monitoring systems that parse
/var/log/maillog, we enabled rsyslog with imuxsock so we still can parse
the log like we did before journald. But, it's unreliable.
Our monitoring systems are reporting failed deliveries of messages
because of missing log lines in /var/log/maillog. When using journalctl
to query the journal, the missing lines can be found, but these queries
are too CPU intensive.
We also see that journald is occasionally logging messages such as this:
Jan 08 20:55:16 host123 systemd-journald[11136]: Forwarding to syslog
missed 2 messages.
Since this message doesn't provide any information as to why the
messages were missed, I have to wonder if it's related to this warning
message on the rsyslog site:
"Note: It must be noted, however, that the journal tends to drop
messages when it becomes busy instead of forwarding them to the system
log socket. This is because the journal uses an async log socket
interface for forwarding instead of the traditional synchronous one."
See:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#imuxsock-systemd-details-label
I'm aware we could switch to using imjournal, which might solve the
issue since it reads the journal directly (which does seem to contain
the missing messages), but I have to imagine that it would come at a
very high CPU cost.
See:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html
So, I'm trying to figure out if it would be possible to get Postfix to
use an alternate logging mechanism that would completely bypass journald
so that we can have reliable loggging in a manner that is less CPU
intensive than journald/imjournal.
Ideas?
Thanks,
Curtis