[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Launchpad Bug Tracker]

This bug was fixed in the package systemd - 237-3ubuntu6

---
systemd (237-3ubuntu6) bionic; urgency=medium

  * Adjust the new dropin test, for v237 systemd.
  * Refresh the keyring patch, to the one merged.

 -- Dimitri John Ledkov   Tue, 27 Mar 2018 13:40:09
+0100

** Changed in: systemd (Ubuntu Bionic)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Artful:
  Opinion
Status in systemd source package in Bionic:
  Fix Released

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Francis Ginther]

** Tags added: id-5ab94cf1ab21d3c7e6eee4ec

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Fix Committed
Status in systemd source package in Artful:
  Opinion
Status in systemd source package in Bionic:
  Fix Committed

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Dimitri John Ledkov]

@colin-king

I'm thinking to make "journald" more important by setting Nice=-1, to
hopefully make it prioritized more often.

There are also CPUSchedulingPolicy=, CPUSchedulingPriority=,
IOSchedulingClass=, IOSchedulingPriority= -> i wonder if I should be
adjusting these settings too, to make journald more "real-time"
processed.

** Changed in: systemd (Ubuntu Bionic)
   Status: Confirmed => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Fix Committed
Status in systemd source package in Artful:
  Opinion
Status in systemd source package in Bionic:
  Fix Committed

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Balint Reczey]

> and/or failing. Imho, one shouldn't be killing journald, when it is otherwise 
> obviously
> operating fine (aka waiting to be run).

@xnox: How do you tell if there is no live lock and it is operating fine
despite it timed out on a 3 minute timer?

> I'm concerned as to why there is a watchdog on journald now. It should be 
> rocksolid,
> and either work or crash, there is no need to crash it on a fixed schedule 
> just because.

If we drop the watchdog we won't get any new journal entries if it
enters/tricked into an infinite loop. I don't think that would be wise.

There are upstream bugs with too little information for similar issues:
https://github.com/systemd/systemd/issues/2899
https://github.com/systemd/systemd/issues/2924

@xnox Do you have links with enough info for debugging?


** Bug watch added: github.com/systemd/systemd/issues #2899
   https://github.com/systemd/systemd/issues/2899

** Bug watch added: github.com/systemd/systemd/issues #2924
   https://github.com/systemd/systemd/issues/2924

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Confirmed
Status in systemd source package in Artful:
  Opinion
Status in systemd source package in Bionic:
  Confirmed

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Dimitri John Ledkov]

I see more chatter about journald aborting upstream and on the mailing
lists. Imho, just because journald was not scheduled to run kind of
means that maybe it has too low of a priority, or the system is
overloaded. It does not show that journald is actually at fault here,
and/or failing. Imho, one shouldn't be killing journald, when it is
otherwise obviously operating fine (aka waiting to be run).

I'm concerned as to why there is a watchdog on journald now. It should
be rocksolid, and either work or crash, there is no need to crash it on
a fixed schedule just because.

** Also affects: systemd (Ubuntu Bionic)
   Importance: High
 Assignee: Balint Reczey (rbalint)
   Status: Opinion

** Changed in: systemd (Ubuntu Bionic)
   Status: Opinion => Confirmed

** Changed in: systemd (Ubuntu Bionic)
 Assignee: Balint Reczey (rbalint) => (unassigned)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Confirmed
Status in systemd source package in Artful:
  Opinion
Status in systemd source package in Bionic:
  Confirmed

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Balint Reczey]

@colin-king In this case the journal was not broken. Do you have many
examples of really broken journals?

If so, do you have any solution in mind that would work better in
scenarios you care about but don't cause regressions?

Proper infrastructure planning with (CPU) quotas in place where it
matters seems to be a good practice instead of running highly loaded
servers.

The 3 min default timer for systemd-journald seems appropriate or even
generous.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Opinion
Status in systemd source package in Artful:
  Opinion

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Francis Ginther]

** Tags added: id-597a8343fc8685288751db7a

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Opinion
Status in systemd source package in Artful:
  Opinion

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Colin Ian King]

Hrm, softlockups that break the journal aren't particularly rare on
highly loaded servers. I'm a little concerned that a potential user
space DoS attack could cause the log to discard important information
that the system admin may use to help track down misbehaving system
behaviour.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Opinion
Status in systemd source package in Artful:
  Opinion

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Balint Reczey]

I suggest setting this bug Won't Fix.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Opinion
Status in systemd source package in Artful:
  Opinion

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Balint Reczey]

@xnox IMO watchdogs should not care if it is a "soft" or other kind of
lockup.

I'm closing the bug because the workaround for recovering from a
potentially broken journal is working and there is no indication of a
journal which is actually broken.


** Changed in: systemd (Ubuntu Artful)
   Status: Incomplete => Opinion

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Opinion
Status in systemd source package in Artful:
  Opinion

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Dimitri John Ledkov]

Note that journal too has watchdogsec limit.

I do wonder if softlockup should or should not be counted towards the
watchdog counter =/

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Incomplete
Status in systemd source package in Artful:
  Incomplete

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Balint Reczey]

The journal restart can be observed on zesty as well.

However, the journal is not corrupted, the renaming and replacing took place as 
a safety measure due to unclean shutdown.
$ sudo journalctl --verify --file=/run/log/journal/*/*
PASS: /run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system.journal 

PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@00055289b166aebb-17bfde05076f84d2.journal~
  
PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00013de4-0005528690181932.journal
PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00013569-000552843559e7e6.journal
PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00012d03-00055282a0e0b31b.journal
PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-000124a8-000552806e21938c.journal
PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-00011ca8-0005527efc9ec6de.journal
PASS: 
/run/log/journal//6ff4e09ca827435bbe1b6efc2236a105/system@b7f19f66c640442d854033592fdedbb6-000114a4-0005527d2438261c.journal

Was the journal corrupted in your case?


** Changed in: systemd (Ubuntu Artful)
   Status: New => Incomplete

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  Incomplete
Status in systemd source package in Artful:
  Incomplete

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Steve Langasek]

** Changed in: systemd (Ubuntu Artful)
 Assignee: Canonical Foundations Team (canonical-foundations) => Balint 
Reczey (rbalint)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  New
Status in systemd source package in Artful:
  New

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Steve Langasek]

The watchdogs are by design.  The unclean journal may not be.  OTOH if
this only happens with the ephemeral journal in /run, the impact is
minor.  We should check whether the unclean journal problem is
reproducible with /var/log/journal.

** Also affects: systemd (Ubuntu Artful)
   Importance: High
   Status: New

** Tags removed: rls-aa-incoming

** Changed in: systemd (Ubuntu Artful)
 Assignee: (unassigned) => Canonical Foundations Team 
(canonical-foundations)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  New
Status in systemd source package in Artful:
  New

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1696970] Re: softlockup DoS causes systemd-journald.service to abort with SIGABORT

[Brian Murray]

** Tags added: rls-aa-incoming

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1696970

Title:
  softlockup DoS causes systemd-journald.service to abort with SIGABORT

Status in systemd package in Ubuntu:
  New

Bug description:
  I was running the new stress-ng softlockup stressor and observed that
  systemd-journald gets killed with an abort and this corrupts the
  systemd journal.

  How to reproduce:

  git clone git://kernel.ubuntu.com/cking/stress-ng
  cd stress-ng
  make clean; make

  sudo ./stress-ng --softlockup 0 -t 360 -v

  ..and wait for 360 seconds.  dmesg shows the following, 100%
  reproduceable:

  
  [  875.310331] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 
3min)!
  [  875.310740] systemd[1]: systemd-timesyncd.service: Killing process 574 
(systemd-timesyn) with signal SIGABRT.
  [  875.327289] systemd[1]: systemd-timesyncd.service: Main process exited, 
code=killed, status=6/ABRT
  [  875.327666] systemd[1]: systemd-timesyncd.service: Unit entered failed 
state.
  [  875.327686] systemd[1]: systemd-timesyncd.service: Failed with result 
'watchdog'.
  [  875.327917] systemd[1]: systemd-timesyncd.service: Service has no hold-off 
time, scheduling restart.
  [  875.327954] systemd[1]: Stopped Network Time Synchronization.
  [  875.328845] systemd[1]: Starting Network Time Synchronization...
  [  875.525071] systemd[1]: Started Network Time Synchronization.
  [  875.539619] systemd[1]: systemd-journald.service: Main process exited, 
code=dumped, status=6/ABRT
  [  875.544257] systemd-journald[5214]: File 
/run/log/journal/440e485e550040e3b93b66b2faae8525/system.journal corrupted or 
uncleanly shut down, renaming and replacing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1696970/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp