[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
This bug was fixed in the package systemd - 237-3ubuntu6 --- systemd (237-3ubuntu6) bionic; urgency=medium * Adjust the new dropin test, for v237 systemd. * Refresh the keyring patch, to the one merged. -- Dimitri John Ledkov Tue, 27 Mar 2018 13:40:09 +0100 ** Changed in: systemd (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Artful: Opinion Status in systemd source package in Bionic: Fix Released Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
** Tags added: id-5ab94cf1ab21d3c7e6eee4ec -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Fix Committed Status in systemd source package in Artful: Opinion Status in systemd source package in Bionic: Fix Committed Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
@colin-king I'm thinking to make "journald" more important by setting Nice=-1, to hopefully make it prioritized more often. There are also CPUSchedulingPolicy=, CPUSchedulingPriority=, IOSchedulingClass=, IOSchedulingPriority= -> i wonder if I should be adjusting these settings too, to make journald more "real-time" processed. ** Changed in: systemd (Ubuntu Bionic) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Fix Committed Status in systemd source package in Artful: Opinion Status in systemd source package in Bionic: Fix Committed Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
> and/or failing. Imho, one shouldn't be killing journald, when it is otherwise > obviously > operating fine (aka waiting to be run). @xnox: How do you tell if there is no live lock and it is operating fine despite it timed out on a 3 minute timer? > I'm concerned as to why there is a watchdog on journald now. It should be > rocksolid, > and either work or crash, there is no need to crash it on a fixed schedule > just because. If we drop the watchdog we won't get any new journal entries if it enters/tricked into an infinite loop. I don't think that would be wise. There are upstream bugs with too little information for similar issues: https://github.com/systemd/systemd/issues/2899 https://github.com/systemd/systemd/issues/2924 @xnox Do you have links with enough info for debugging? ** Bug watch added: github.com/systemd/systemd/issues #2899 https://github.com/systemd/systemd/issues/2899 ** Bug watch added: github.com/systemd/systemd/issues #2924 https://github.com/systemd/systemd/issues/2924 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Artful: Opinion Status in systemd source package in Bionic: Confirmed Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
I see more chatter about journald aborting upstream and on the mailing lists. Imho, just because journald was not scheduled to run kind of means that maybe it has too low of a priority, or the system is overloaded. It does not show that journald is actually at fault here, and/or failing. Imho, one shouldn't be killing journald, when it is otherwise obviously operating fine (aka waiting to be run). I'm concerned as to why there is a watchdog on journald now. It should be rocksolid, and either work or crash, there is no need to crash it on a fixed schedule just because. ** Also affects: systemd (Ubuntu Bionic) Importance: High Assignee: Balint Reczey (rbalint) Status: Opinion ** Changed in: systemd (Ubuntu Bionic) Status: Opinion => Confirmed ** Changed in: systemd (Ubuntu Bionic) Assignee: Balint Reczey (rbalint) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Artful: Opinion Status in systemd source package in Bionic: Confirmed Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
@colin-king In this case the journal was not broken. Do you have many examples of really broken journals? If so, do you have any solution in mind that would work better in scenarios you care about but don't cause regressions? Proper infrastructure planning with (CPU) quotas in place where it matters seems to be a good practice instead of running highly loaded servers. The 3 min default timer for systemd-journald seems appropriate or even generous. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Opinion Status in systemd source package in Artful: Opinion Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
** Tags added: id-597a8343fc8685288751db7a -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Opinion Status in systemd source package in Artful: Opinion Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
Hrm, softlockups that break the journal aren't particularly rare on highly loaded servers. I'm a little concerned that a potential user space DoS attack could cause the log to discard important information that the system admin may use to help track down misbehaving system behaviour. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Opinion Status in systemd source package in Artful: Opinion Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
I suggest setting this bug Won't Fix. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Opinion Status in systemd source package in Artful: Opinion Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
@xnox IMO watchdogs should not care if it is a "soft" or other kind of lockup. I'm closing the bug because the workaround for recovering from a potentially broken journal is working and there is no indication of a journal which is actually broken. ** Changed in: systemd (Ubuntu Artful) Status: Incomplete => Opinion -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Opinion Status in systemd source package in Artful: Opinion Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
Note that journal too has watchdogsec limit. I do wonder if softlockup should or should not be counted towards the watchdog counter =/ -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Artful: Incomplete Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
The journal restart can be observed on zesty as well. However, the journal is not corrupted, the renaming and replacing took place as a safety measure due to unclean shutdown. $ sudo journalctl --verify --file=/run/log/journal/*/* PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system.journal PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@00055289b166aebb-17bfde05076f84d2.journal~ PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00013de4-0005528690181932.journal PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00013569-000552843559e7e6.journal PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00012d03-00055282a0e0b31b.journal PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-000124a8-000552806e21938c.journal PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00011ca8-0005527efc9ec6de.journal PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-000114a4-0005527d2438261c.journal Was the journal corrupted in your case? ** Changed in: systemd (Ubuntu Artful) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Artful: Incomplete Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
** Changed in: systemd (Ubuntu Artful) Assignee: Canonical Foundations Team (canonical-foundations) => Balint Reczey (rbalint) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: New Status in systemd source package in Artful: New Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
The watchdogs are by design. The unclean journal may not be. OTOH if this only happens with the ephemeral journal in /run, the impact is minor. We should check whether the unclean journal problem is reproducible with /var/log/journal. ** Also affects: systemd (Ubuntu Artful) Importance: High Status: New ** Tags removed: rls-aa-incoming ** Changed in: systemd (Ubuntu Artful) Assignee: (unassigned) => Canonical Foundations Team (canonical-foundations) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: New Status in systemd source package in Artful: New Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT
** Tags added: rls-aa-incoming -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1696970 Title: softlockup DoS causes systemd-journald.service to abort with SIGABORT Status in systemd package in Ubuntu: New Bug description: I was running the new stress-ng softlockup stressor and observed that systemd-journald gets killed with an abort and this corrupts the systemd journal. How to reproduce: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng make clean; make sudo ./stress-ng --softlockup 0 -t 360 -v ..and wait for 360 seconds. dmesg shows the following, 100% reproduceable: [ 875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)! [ 875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 (systemd-timesyn) with signal SIGABRT. [ 875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=6/ABRT [ 875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed state. [ 875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 'watchdog'. [ 875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart. [ 875.327954] systemd[1]: Stopped Network Time Synchronization. [ 875.328845] systemd[1]: Starting Network Time Synchronization... [ 875.525071] systemd[1]: Started Network Time Synchronization. [ 875.539619] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 875.544257] systemd-journald[5214]: File /run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or uncleanly shut down, renaming and replacing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp