[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
This bug was fixed in the package systemd - 245.4-4ubuntu3.16 --- systemd (245.4-4ubuntu3.16) focal; urgency=medium [ Dan Streetman ] * d/p/lp1946388-sd-journal-don-t-check-namespaces-if-we-have-no-name.patch: Avoid journalctl segfault (LP: #1946388) [ Jeremy Szu ] * Add a allowlist to unblock intel-hid on new HP machines (LP: #1955997) Author: Jeremy Szu File: debian/patches/lp1955997-add-a-allowlist-to-unblock-intel-hid-on-HP-mach.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=88a859eaddb6c9a611fcbc44edab441aef4c4355 [ Nick Rosbrook ] * Prevent arguments from being overwritten with defaults at shutdown (LP: #1958284) File: debian/patches/lp1958284-core-move-reset_arguments-to-the-end-of-main-s-finish.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e61052bd1f20bcc54e7417542c6d445cf5040f56 [ Lukas Märdian ] * Fix deadlock between pid1 and dbus-daemon (LP: #1871538) Author: Lukas Märdian File: debian/patches/pid1-set-SYSTEMD_NSS_DYNAMIC_BYPASS-1-env-var-for-dbus-da.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e3aacfa26e3fc6df369e6f28e740389ae0020907 -- Nick Rosbrook Wed, 23 Mar 2022 09:29:33 -0400 ** Changed in: systemd (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Released Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
The autopkgtest regressions blocking systemd 245.4-4ubuntu3.16 in focal- proposed have been resolved. The regressions appear to have been related to recent autopkgtest infrastructure issues, and retrying the tests resolved the issues. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages M
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
I tested systemd 245.4-4ubuntu3.16 from focal-proposed using the test plan above. I observed that the loop-ignore-sigterm.service processes were killed after ~20s on shutdown, which is what I configured in /etc/systemd/system.conf. ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-pa
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
Hello Jean, or anyone else affected, Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.16 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: systemd (Ubuntu Focal) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enou
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Changed in: systemd (Ubuntu Focal) Status: Confirmed => In Progress ** Changed in: systemd (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: In Progress Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Merge proposal linked: https://code.launchpad.net/~enr0n/ubuntu/+source/systemd/+git/systemd/+merge/417577 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: Confirmed Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setting of DefaultTimeoutStopSec. In such cases, this impacts users by significantly increasing the time required for shutdown/reboot. [Test Plan] * Create a new script, /usr/local/bin/loop-ignore-sigterm: ``` #!/bin/bash loop_forever() { while true; do sleep 1; done } ( trap 'echo Ignoring SIGTERM...' SIGTERM loop_forever ) loop_forever ``` This script will spawn a subshell which will loop forever and ignore SIGTERM. This will force systemd to wait for the subprocess at reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec (DefaultTimeoutStopSec in this case). * Make the script executable: $ chmod +x /usr/local/bin/loop-ignore-sigterm * Create a systemd service for this script. Add the following to /etc/systemd/system/loop-ignore-sigterm.service: ``` [Service] KillMode=process ExecStart=/usr/local/bin/loop-ignore-sigterm ``` * Start the service: $ systemctl start loop-ignore-sigterm.service * Edit /etc/systemd/system.conf, and uncomment the 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, e.g. 20s. * Re-exec the daemon so this new default takes effect: $ systemctl daemon-reexec * Reboot, and monitor the logs. Observe that systemd-shutdown will wait for the loop-ignore-sigterm process for 90s, instead of the 20s configured earlier. [Where problems could occur] The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec. [ Original Description ] With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Description changed: + [Impact] + + The systemd shutdown sequence does not honor systemd-system.conf + settings when waiting for remaining processes. This means that, for + example, if a systemd service specifies KillMode=process and a process + remaining from that service does not properly handle SIGTERM, then the + remaining process will not be killed until after the compiled-in default + value of DefaultTimeoutStopSec (90s), even if the user has changed the + setting of DefaultTimeoutStopSec. In such cases, this impacts users by + significantly increasing the time required for shutdown/reboot. + + [Test Plan] + + * Create a new script, /usr/local/bin/loop-ignore-sigterm: + ``` + #!/bin/bash + loop_forever() { + while true; do sleep 1; done + } + + ( + trap 'echo Ignoring SIGTERM...' SIGTERM + loop_forever + ) + + loop_forever + ``` + + This script will spawn a subshell which will loop forever and ignore + SIGTERM. This will force systemd to wait for the subprocess at + reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec + (DefaultTimeoutStopSec in this case). + + * Make the script executable: + $ chmod +x /usr/local/bin/loop-ignore-sigterm + + * Create a systemd service for this script. Add the following to + /etc/systemd/system/loop-ignore-sigterm.service: + ``` + [Service] + KillMode=process + ExecStart=/usr/local/bin/loop-ignore-sigterm + ``` + + * Start the service: + $ systemctl start loop-ignore-sigterm.service + + * Edit /etc/systemd/system.conf, and uncomment the + 'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter, + e.g. 20s. + + * Re-exec the daemon so this new default takes effect: + $ systemctl daemon-reexec + + * Reboot, and monitor the logs. Observe that systemd-shutdown will wait + for the loop-ignore-sigterm process for 90s, instead of the 20s + configured earlier. + + [Where problems could occur] + + The patch moves the reset_arguments() call to the end of main, which + means reset_arguments() is no longer called before daemon re-execution + (if that branch is taken). If anything in that code path relied on + reset_arguments() being called before re-executing, those assumptions + could be broken. Any such problems would potentially be seen during + daemon re-execution, e.g. when calling systemctl daemon-reexec. + + [ Original Description ] + With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: Confirmed Bug description: [Impact] The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutStopSec (90s), even if the user has changed the setti
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
It has recently been picked up by Foundations, and we should have the capacity to start working on this next week. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: Confirmed Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
Any updates on this? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: Confirmed Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Changed in: systemd (Ubuntu Focal) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: Confirmed Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Tags added: fr-1987 ** Tags removed: fr-1987 rls-ff-incoming ** Changed in: systemd (Ubuntu Focal) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: Confirmed Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Tags added: rls-ff-incoming ** Also affects: systemd (Ubuntu Focal) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Focal: New Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: systemd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: Confirmed Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1958284] Re: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
** Description changed: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - - http://localhost:8000/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html + - https://github.com/systemd/systemd/issues/16991 + - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1958284 Title: shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec Status in systemd package in Ubuntu: New Bug description: With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeoutStopSec` to wait for remaining processes, it instead uses the compiled in default of 90s. This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it. For example: ``` [ OK ] Finished Reboot. [ OK ] Reached target Reboot. [ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit --- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower value --- ``` The bug has been fixed upstream here: https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22 Marc was kind enough to package the patch for 20.04 so I could test it (https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing- archive-extra) and with that package, I can confirm that it indeed fixes the issue. Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using: - https://github.com/moby/moby/issues/41831 - https://github.com/k3s-io/k3s/issues/2400 - https://github.com/systemd/systemd/issues/16991 - https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround. Given that the patch is relatively simple, would it be possible to add it the package for 20.04? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp