Re: [systemd-devel] Native Journal source vs syslog forwarding
On Wed, 26.11.14 11:04, Gergely Nagy (alger...@madhouse-project.org) wrote: Hi! I have an interesting situation here, which I'm trying to wrap my head around and solve. The problem is that I have a syslog daemon (syslog-ng 3.6.1) that has a native Journal source, meaning it can pull entries from the Journal directly, and does not need the syslog forwarding socket - and this is the default when running on a systemd-enabled machine. This works beautifully, except there's one problem: Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages. On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. Hmmm, we only increase the counter when we get EAGAIN from the socket. Normally, if nobody is listening on the other side we shouldn't get EAGAIN, we should get ECONNREFUSED or so. Most likely for some reason syslog.socket gets started (which establishes the socket), but nothing is then started behind it? That sounds wrong. Note that in systemd upstream we don't do forward-to-syslog anymore, simply because it turned out to be really slow when nobody is listening on the socket, and rsyslog didn't use it anymore. Does Debian turn this on after all? 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) We really should figure out why journald gets EAGAIN when writing to that socket. Is syslog.socket started? If so, why? Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On Wed, 26.11.14 13:08, Gergely Nagy (alger...@madhouse-project.org) wrote: Colin == Colin Guthrie gm...@colin.guthr.ie writes: Colin Gergely Nagy wrote on 26/11/14 10:04: 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) Colin I think there were some patches recently that talked about added .d/ Colin dir support for such config files. Thus syslog-ng could ship a Colin /etc/systemd/journal.conf.d/syslog-ng.conf file which did this (or Colin preferably a /usr/lib/systemd/journal.conf.d/syslog-ng.conf file so it Colin follows best practice guidelines for stateless systems) Thanks, this sounds like a good start! Please don't tape over things. Let's figure out the cause for the message first. You really shouldn't get the message if everything was working right... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On Wed, 26.11.14 13:07, Gergely Nagy (alger...@madhouse-project.org) wrote: Jóhann == Jóhann B Guðmundsson johan...@gmail.com writes: Jóhann On 11/26/2014 10:04 AM, Gergely Nagy wrote: On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. [...] 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) Jóhann As of systemd 216 forwarding to another syslogd has been disabled by Jóhann default so no need to fiddle with this setting. Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. There are other syslogds in Debian (including the default one) that do not read from the Journal, but rely on forwarding. If forwarding was turned off, they'd stop working. Older versions of syslog-ng would, too. other syslogds? Which ones just out of curiosity? If both syslog-ng and rsyslog can now read directly from the journal I wonder what other syslog implementation debian wants to support there... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On Fri, 28.11.14 14:53, Martin Pitt (martin.p...@ubuntu.com) wrote: Hey Gergely, Gergely Nagy [2014-11-26 13:07 +0100]: Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. I'm just packaging systemd 217, and will revert the disabled forwarding by default (i. e. retain the behaviour of 215). As a systemd package maintainer I'm also not in a position to unilaterally changing the default and breaking rsyslog and friends. So this requires some coordination indeed (and either way, none of this is a matter for the frozen jessie). So we either need the journal.conf.d/ feature and have journal-pulling sysloggers disable forwarding along the way, or we need to wait until all packaged sysloggers can read from the journal before we turn off forwarding by default. On Fedora all loggers apparently read from the journal directly now, to my knowledge. So the code at least is there. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On Fri, 28.11.14 15:13, Gergely Nagy (alger...@madhouse-project.org) wrote: Martin == Martin Pitt martin.p...@ubuntu.com writes: Martin So we either need the journal.conf.d/ feature and have journal-pulling Martin sysloggers disable forwarding along the way, or we need to wait until Martin all packaged sysloggers can read from the journal before we turn off Martin forwarding by default. For what its worth, syslog-ng upstream suggested another solution, that appears to fix the issue (but I've yet to test it myself): if the syslog-ng.service file does *not* Require=syslog.socket, then the syslog.socket does not get started, and if it doesn't get started, the journal will not do forwarding, either, and thus, the error messages disappear too. Ah, yeah! It shouldn't require it. If it pulls it in, that's what it gets... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
Lennart == Lennart Poettering lenn...@poettering.net writes: Lennart On Wed, 26.11.14 11:04, Gergely Nagy (alger...@madhouse-project.org) wrote: This works beautifully, except there's one problem: Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages. On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. Lennart Hmmm, we only increase the counter when we get EAGAIN from the Lennart socket. Normally, if nobody is listening on the other side we Lennart shouldn't get EAGAIN, we should get ECONNREFUSED or so. Lennart Most likely for some reason syslog.socket gets started (which Lennart establishes the socket), but nothing is then started behind it? That Lennart sounds wrong. Yep, it turns out that was the issue. syslog-ng.service had a Requires=syslog.socket, a left over from an older version which did not support reading from the journal. It appears that removing that line disabled starting the socket, and the forwarding errors stopped too. Lennart Note that in systemd upstream we don't do forward-to-syslog anymore, Lennart simply because it turned out to be really slow when nobody is Lennart listening on the socket, and rsyslog didn't use it anymore. Does Lennart Debian turn this on after all? Yes, because rsyslog (the default syslogd in debian) does not have the journald reader enabled (or at least, not as a default), as far as I remember. -- |8] signature.asc Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
Lennart == Lennart Poettering lenn...@poettering.net writes: Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. There are other syslogds in Debian (including the default one) that do not read from the Journal, but rely on forwarding. If forwarding was turned off, they'd stop working. Older versions of syslog-ng would, too. Lennart other syslogds? Which ones just out of curiosity? If both syslog-ng Lennart and rsyslog can now read directly from the journal I wonder what other Lennart syslog implementation debian wants to support there... There's a handful of syslogds, including busybox-syslogd, and the ancient sysklogd, among other things. Furthermore, we want to support upgrades that continue running sysvinit, so we can't make the journal reader default (unless the syslogd can - like syslog-ng - figure out which one to use at run time). (iirc the default rsyslog.conf uses /dev/log, and rsyslog translates that to /run/systemd/journal/syslog if it detects systemd, thus preserving compatibility.) -- |8] signature.asc Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On Mon, 01.12.14 00:30, Gergely Nagy (alger...@madhouse-project.org) wrote: Lennart == Lennart Poettering lenn...@poettering.net writes: Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. There are other syslogds in Debian (including the default one) that do not read from the Journal, but rely on forwarding. If forwarding was turned off, they'd stop working. Older versions of syslog-ng would, too. Lennart other syslogds? Which ones just out of curiosity? If both syslog-ng Lennart and rsyslog can now read directly from the journal I wonder what other Lennart syslog implementation debian wants to support there... There's a handful of syslogds, including busybox-syslogd, and the ancient sysklogd, among other things. Furthermore, we want to support upgrades that continue running sysvinit, so we can't make the journal reader default (unless the syslogd can - like syslog-ng - figure out which one to use at run time). (iirc the default rsyslog.conf uses /dev/log, and rsyslog translates that to /run/systemd/journal/syslog if it detects systemd, thus preserving compatibility.) Note that in very recent systemd /dev/log is actually a symlink to the real socket in /run/systemd/. This concept allows compatibility with those ancient syslog implementations by simply making them replace the symlink and own /dev/log directly. That way the the journal is bypassed for syslog messages, but I figure this might be a better compat model for those implementations, after all they'd need patching anyway to read from the forwarder socket or the journal... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
Hey Gergely, Gergely Nagy [2014-11-26 13:07 +0100]: Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. I'm just packaging systemd 217, and will revert the disabled forwarding by default (i. e. retain the behaviour of 215). As a systemd package maintainer I'm also not in a position to unilaterally changing the default and breaking rsyslog and friends. So this requires some coordination indeed (and either way, none of this is a matter for the frozen jessie). So we either need the journal.conf.d/ feature and have journal-pulling sysloggers disable forwarding along the way, or we need to wait until all packaged sysloggers can read from the journal before we turn off forwarding by default. Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Native Journal source vs syslog forwarding
Hi! I have an interesting situation here, which I'm trying to wrap my head around and solve. The problem is that I have a syslog daemon (syslog-ng 3.6.1) that has a native Journal source, meaning it can pull entries from the Journal directly, and does not need the syslog forwarding socket - and this is the default when running on a systemd-enabled machine. This works beautifully, except there's one problem: Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages. On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. I'm not sure how to solve this problem. As far as I see, I have the following options: 1) Drop the native journal source and use syslog forwarding. This is trivial to do, but I loose the extra fields and info the Journal collects. I'd rather not do this. 2) Have a dummy listener on /run/systemd/journal/syslog, that just reads everything and drops it on the floor. This sounds fishy, and is a bit awkward to implement in the config. This would also be an ugly hack, not a real solution. 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) I'd appreciate any hints. (Disabling syslog forwarding by default is not an option.) -- |8] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
Gergely Nagy wrote on 26/11/14 10:04: 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) I think there were some patches recently that talked about added .d/ dir support for such config files. Thus syslog-ng could ship a /etc/systemd/journal.conf.d/syslog-ng.conf file which did this (or preferably a /usr/lib/systemd/journal.conf.d/syslog-ng.conf file so it follows best practice guidelines for stateless systems) See the patches by Josh Triplett: [PATCH 1/2] Introduce CONF_DIRS_NULSTR helper to define standard conf dirs [PATCH 2/2] logind: Support logind.conf.d directories in the usual search paths The latter patch just adds support in logind, but mentions the likes of journald too as further work if approved. I think this approach would solve your problem fairly cleanly. Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On 11/26/2014 10:04 AM, Gergely Nagy wrote: Hi! I have an interesting situation here, which I'm trying to wrap my head around and solve. The problem is that I have a syslog daemon (syslog-ng 3.6.1) that has a native Journal source, meaning it can pull entries from the Journal directly, and does not need the syslog forwarding socket - and this is the default when running on a systemd-enabled machine. This works beautifully, except there's one problem: Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages. This happens when the socket buffer is full for the syslogd which is being forwarding to. ( try tweaking those settings ) On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. I'm not sure how to solve this problem. As far as I see, I have the following options: 1) Drop the native journal source and use syslog forwarding. This is trivial to do, but I loose the extra fields and info the Journal collects. I'd rather not do this. 2) Have a dummy listener on /run/systemd/journal/syslog, that just reads everything and drops it on the floor. This sounds fishy, and is a bit awkward to implement in the config. This would also be an ugly hack, not a real solution. 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) As of systemd 216 forwarding to another syslogd has been disabled by default so no need to fiddle with this setting. I'd appreciate any hints. (Disabling syslog forwarding by default is not an option.) Why is that not an option since there is nothing by your own account listening to /run/systemd/journal/syslog? JBG ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
Jóhann == Jóhann B Guðmundsson johan...@gmail.com writes: Jóhann On 11/26/2014 10:04 AM, Gergely Nagy wrote: On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. [...] 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) Jóhann As of systemd 216 forwarding to another syslogd has been disabled by Jóhann default so no need to fiddle with this setting. Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. There are other syslogds in Debian (including the default one) that do not read from the Journal, but rely on forwarding. If forwarding was turned off, they'd stop working. Older versions of syslog-ng would, too. -- |8] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
Colin == Colin Guthrie gm...@colin.guthr.ie writes: Colin Gergely Nagy wrote on 26/11/14 10:04: 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) Colin I think there were some patches recently that talked about added .d/ Colin dir support for such config files. Thus syslog-ng could ship a Colin /etc/systemd/journal.conf.d/syslog-ng.conf file which did this (or Colin preferably a /usr/lib/systemd/journal.conf.d/syslog-ng.conf file so it Colin follows best practice guidelines for stateless systems) Thanks, this sounds like a good start! -- |8] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Native Journal source vs syslog forwarding
On 11/26/2014 12:07 PM, Gergely Nagy wrote: Jóhann == Jóhann B Guðmundsson johan...@gmail.com writes: Jóhann On 11/26/2014 10:04 AM, Gergely Nagy wrote: On Debian, syslog forwarding is enabled by default, and since syslog-ng reads from the journal, there's nothing listening on /run/systemd/journal/syslog, and I get spammed with messages like the one above. [...] 3) Disable syslog forwarding if syslog-ng is installed Not sure how this could be achieved, because journald.conf does not belong to the syslog-ng package, therefore I can't fiddle its settings from there. (Technically, I could, but I won't, that'd be extremely rude.) Jóhann As of systemd 216 forwarding to another syslogd has been disabled by Jóhann default so no need to fiddle with this setting. Forwarding is enabled by default on Debian, as I wrote in my original mail. I have no control over the default, and I have no desire to argue for changing it. There are other syslogds in Debian (including the default one) that do not read from the Journal, but rely on forwarding. If forwarding was turned off, they'd stop working. Older versions of syslog-ng would, too. Right but you are going against downstream distribution policy and unwilling yourself ( or have the consumer of the syslog-ng package do that ) to change the default manually ( via journald.conf ) or advocate for that change ( which would be simply advocating for following upstream ). If you are the syslog-ng maintainer in Debian then I guess you will have to introduce conflicts with other sysloggers in other components and have them remove the default syslogger ( assuming the journal has not been made the default yet in Debian and something like rsyslog is ) because if you go the snipped route you would overwrite the default thus break rsyslog ( and others if they exist ) in the process. What is Jessie using 215? if so you are probably stuck with 215 for the reminder of Jessie which requires you to convince the Debian systemd maintainers to backport the relevant patch(es) to make that work. Does Debian have two syslog-ng components one tweak for journal as in uses systemd-journal() along with filters and default journal tweaked send declaration and another one that is tweaked for $other systemd init systems since there are more tweaks than just systemd-journal() which should be made as an default in that process? Or is there a single component that detects which init system is installed and install configuration files based on that? ( if so could he not simply s/ForwardToSyslog=/ from yes to no in the process? ) JBG ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
