Re: [systemd-devel] Journald stops logging when lots of output
'Twas brillig, and Lennart Poettering at 26/09/13 19:35 did gyre and gimble: On Thu, 26.09.13 15:46, Olav Vitters (o...@vitters.nl) wrote: On Wed, Sep 11, 2013 at 08:35:49PM +0200, Lennart Poettering wrote: (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Though it might hide problems, shouldn't journald be configured to automatically be restarted by systemd? We do that, but not indefinitely. See StartLimitInterval=/StartLimitBurst= in systemd.service(5). Quick question, I presume logging is still lost for services stderr/out etc. when journald is restarted? Or has this been worked-around these days? Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Fri, 27.09.13 09:47, Colin Guthrie (gm...@colin.guthr.ie) wrote: 'Twas brillig, and Lennart Poettering at 26/09/13 19:35 did gyre and gimble: On Thu, 26.09.13 15:46, Olav Vitters (o...@vitters.nl) wrote: On Wed, Sep 11, 2013 at 08:35:49PM +0200, Lennart Poettering wrote: (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Though it might hide problems, shouldn't journald be configured to automatically be restarted by systemd? We do that, but not indefinitely. See StartLimitInterval=/StartLimitBurst= in systemd.service(5). Quick question, I presume logging is still lost for services stderr/out etc. when journald is restarted? Or has this been worked-around these days? Still doesn't work. I do have an idea now however we could make this work by using a single AF_UNIX/SOCK_DGRAM socket instead of individual AF_UNIX/SOCK_STREAM sockets. For services we start it shouldn't matter if stdout/stderr is SOCK_STREAM or SOCK_DGRAM really, and if we use SOCK_DGRAM we even get per-client SCM_CREDENTIALS which would allow us to properly attribute every single write to the right process which we currently can't. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Wed, Sep 11, 2013 at 08:35:49PM +0200, Lennart Poettering wrote: (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Though it might hide problems, shouldn't journald be configured to automatically be restarted by systemd? -- Regards, Olav ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Thu, Sep 26, 2013 at 2:46 PM, Olav Vitters o...@vitters.nl wrote: On Wed, Sep 11, 2013 at 08:35:49PM +0200, Lennart Poettering wrote: (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Though it might hide problems, shouldn't journald be configured to automatically be restarted by systemd? I'm not sure but I believe it crashes three times in a row, which is how far systemd will go trying to revive it. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Thu, 26.09.13 15:46, Olav Vitters (o...@vitters.nl) wrote: On Wed, Sep 11, 2013 at 08:35:49PM +0200, Lennart Poettering wrote: (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Though it might hide problems, shouldn't journald be configured to automatically be restarted by systemd? We do that, but not indefinitely. See StartLimitInterval=/StartLimitBurst= in systemd.service(5). Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Sat, 03.08.13 23:14, Pedro Francisco (pedrogfranci...@gmail.com) wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? Is this still a problem? It seems journald dies for you for some reason. Is there a way how you can reproduce the issue? If so, can you strace the journal daemon, then reproduce it, and paste the last 100 strace lines somwhere before the daemon exited? This should give us an indication what might have failed that caused it to exit. (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Also, if you log using a tool like logger, does that get lost too after the issue happens? Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Sun, Aug 4, 2013 at 12:14 AM, Pedro Francisco pedrogfranci...@gmail.com wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? Probably fixed by: http://cgit.freedesktop.org/systemd/systemd/commit/?id=8b18fdc19531ba56d0bdfe34c62870997a9bcc96 Kay ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Sun, Aug 04, 2013 at 01:22:21PM +0200, Kay Sievers wrote: On Sun, Aug 4, 2013 at 12:14 AM, Pedro Francisco pedrogfranci...@gmail.com wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? Probably fixed by: http://cgit.freedesktop.org/systemd/systemd/commit/?id=8b18fdc19531ba56d0bdfe34c62870997a9bcc96 Kay, are you saying that this commit could influence the starting and stopping of journald, not just lost messages? Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
How should I see journald status/log? sudo systemctl status systemd-journald.service ? P.S.: /var/lib/systemd/coredump is empty. On Sun, Aug 4, 2013 at 5:15 AM, Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl wrote: On Sat, Aug 03, 2013 at 11:14:28PM +0100, Pedro Francisco wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Some similar bugs appeared with jorunald apparently dying when the kernel logs a lot. [1] is recent, but I think I saw some other instances. journald should throttle kernel messages, and could ligitimately loose some when the kernel is really verbose, but of course it shouldn't exit. I would start by investigating why journald exited. Zbyszek [1] https://bugzilla.redhat.com/show_bug.cgi?id=990323 Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Sun, Aug 4, 2013 at 2:43 PM, Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl wrote: On Sun, Aug 04, 2013 at 01:22:21PM +0200, Kay Sievers wrote: On Sun, Aug 4, 2013 at 12:14 AM, Pedro Francisco pedrogfranci...@gmail.com wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? Probably fixed by: http://cgit.freedesktop.org/systemd/systemd/commit/?id=8b18fdc19531ba56d0bdfe34c62870997a9bcc96 Kay, are you saying that this commit could influence the starting and stopping of journald, not just lost messages? Seems I was wrong. It's should be some other problem that returns the failure leading to the exit. Kay ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
On Sat, Aug 03, 2013 at 11:14:28PM +0100, Pedro Francisco wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Some similar bugs appeared with jorunald apparently dying when the kernel logs a lot. [1] is recent, but I think I saw some other instances. journald should throttle kernel messages, and could ligitimately loose some when the kernel is really verbose, but of course it shouldn't exit. I would start by investigating why journald exited. Zbyszek [1] https://bugzilla.redhat.com/show_bug.cgi?id=990323 Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? -- Pedro ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
