Re: It is possible for Postfix logging to bypass journald?
Curtis: > On 1/9/2019 5:55 PM, Robert L Mathews wrote: > > On 1/9/19 4:05 PM, Curtis wrote: > >> We recently switched our Postfix mail servers to Ubuntu Server 18, which > >> uses journald for logging. Since we have monitoring systems that parse > >> /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse > >> the log like we did before journald.? But, it's unreliable. Postfix 3.4 will have support for logging to file and to stdout. Otherwise, Postfix uses the syslog function, part of the system library; Postfix has no control over where that library sends its data. You may be able to tell systemd to keep its hands off the syslog socket, in which case rsyslogd can do it job. Wietse
Re: It is possible for Postfix logging to bypass journald?
On 1/9/2019 5:55 PM, Robert L Mathews wrote: On 1/9/19 4:05 PM, Curtis wrote: We recently switched our Postfix mail servers to Ubuntu Server 18, which uses journald for logging. Since we have monitoring systems that parse /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse the log like we did before journald. But, it's unreliable. Our monitoring systems are reporting failed deliveries of messages because of missing log lines in /var/log/maillog. We had this problem. It was fixed by putting this in /etc/systemd/journald.conf: # allow for busy mail logs; allows 1000 per second RateLimitInterval=5s RateLimitBurst=5000 And/or by putting this into /etc/rsyslog.conf: $SystemLogRateLimitInterval 0 (The latter is supposedly no longer necessary, but it used to be, and does not appear to be harmful.) Thanks for your input. Unfortunately, even after playing with these settings, we see no improvement. When you run "systemctl status systemd-journald" do you see any messages like this? Jan 28 18:16:01 [somehost] systemd-journald[25662]: Forwarding to syslog missed 6 messages. If others are not seeing this issue, then I am wondering if it has something to do our setup being inside of an LXC container. For now, we ended up fixing our log parsing script to make journalctl calls so that no lines are missed. Overall, journald seems like a huge downgrade for us... I get the impression it was designed for desktop users, not for servers. That said, I noticed in another thread that Wietse announced that Postfix has an option to log to a file now... ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/postfix-3.4-20190127-nonprod-logger.RELEASE_NOTES We're not ready to upgrade right now, but I'm looking forward to trying this option out in the future.
Re: It is possible for Postfix logging to bypass journald?
Curtis: We recently switched our Postfix mail servers to Ubuntu Server 18, which uses journald for logging. Since we have monitoring systems that parse /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse the log like we did before journald. But, it's unreliable. On 09.01.19 19:38, Wietse Venema wrote: I recall that system-effing-d has a rare-limiting feature that very helpfully drops Postfix logging. Here's one search result with suggestions for systemd. https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/ Another search result: systemd and rsyslog both have rate limits. https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disable-rsyslog-rate-limiting It is time to update the Postfix page on LINUX logging brain damage. oh, please... systemd and rsyslog. I use sysvinit+syslog-ng wherever possible, on linux -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. The 3 biggets disasters: Hiroshima 45, Tschernobyl 86, Windows 95
Re: It is possible for Postfix logging to bypass journald?
On 1/9/19 4:05 PM, Curtis wrote: > We recently switched our Postfix mail servers to Ubuntu Server 18, which > uses journald for logging. Since we have monitoring systems that parse > /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse > the log like we did before journald. But, it's unreliable. > > Our monitoring systems are reporting failed deliveries of messages > because of missing log lines in /var/log/maillog. We had this problem. It was fixed by putting this in /etc/systemd/journald.conf: # allow for busy mail logs; allows 1000 per second RateLimitInterval=5s RateLimitBurst=5000 And/or by putting this into /etc/rsyslog.conf: $SystemLogRateLimitInterval 0 (The latter is supposedly no longer necessary, but it used to be, and does not appear to be harmful.) -- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
Re: It is possible for Postfix logging to bypass journald?
Curtis: > We recently switched our Postfix mail servers to Ubuntu Server 18, which > uses journald for logging. Since we have monitoring systems that parse > /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse > the log like we did before journald. But, it's unreliable. I recall that system-effing-d has a rare-limiting feature that very helpfully drops Postfix logging. Here's one search result with suggestions for systemd. https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/ Another search result: systemd and rsyslog both have rate limits. https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disable-rsyslog-rate-limiting It is time to update the Postfix page on LINUX logging brain damage. Wietse
It is possible for Postfix logging to bypass journald?
We recently switched our Postfix mail servers to Ubuntu Server 18, which uses journald for logging. Since we have monitoring systems that parse /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse the log like we did before journald. But, it's unreliable. Our monitoring systems are reporting failed deliveries of messages because of missing log lines in /var/log/maillog. When using journalctl to query the journal, the missing lines can be found, but these queries are too CPU intensive. We also see that journald is occasionally logging messages such as this: Jan 08 20:55:16 host123 systemd-journald[11136]: Forwarding to syslog missed 2 messages. Since this message doesn't provide any information as to why the messages were missed, I have to wonder if it's related to this warning message on the rsyslog site: "Note: It must be noted, however, that the journal tends to drop messages when it becomes busy instead of forwarding them to the system log socket. This is because the journal uses an async log socket interface for forwarding instead of the traditional synchronous one." See: https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#imuxsock-systemd-details-label I'm aware we could switch to using imjournal, which might solve the issue since it reads the journal directly (which does seem to contain the missing messages), but I have to imagine that it would come at a very high CPU cost. See: https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html So, I'm trying to figure out if it would be possible to get Postfix to use an alternate logging mechanism that would completely bypass journald so that we can have reliable loggging in a manner that is less CPU intensive than journald/imjournal. Ideas? Thanks, Curtis
