[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** No longer affects: snapd (Ubuntu Bionic) ** No longer affects: snapd (Ubuntu Xenial) ** No longer affects: snapd (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2 --- unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium * Don't check blacklist too early and report updates from not allowed origins as kept back. (LP: #1781176) * test/test_blacklisted_wrong_origin.py: Fix and enable test * Filter out progress indicator from dpkg log (LP: #1599646) * Clear cache when autoremoval fails (LP: #1779157) * Find autoremovable kernel packages using the patterns in APT's way (LP: #1815494) unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium * Start service after systemd-logind.service to be able to take inhibition lock (LP: #1806487) * Handle gracefully when logind is down (LP: #1806487) unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium * Backport to Xenial (LP: #1702793) * Revert to build-depending on debhelper (>= 9~) and dh-systemd * Revert configuration example changes to avoid triggering a debconf question * debian/postinst: Update recovery to be triggered on Xenial's package versions unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown() Performing upgrades in service's ExecStop did not work when the upgrades involved restarting services because systemd blocked other stop/start actions making maintainer scripts time out and be killed leaving a broken system behind. Running unattended-upgrades.service before shutdown.target as a oneshot service made it run after unmounting filesystems and scheduling services properly on shutdown is a complex problem and adding more services to the mix make it even more fragile. The solution of monitoring PrepareForShutdown() signal from DBus allows Unattended Upgrade to run _before_ the jobs related to shutdown are queued thus package upgrades can safely restart services without risking causing deadlocks or breaking part of the shutdown actions. Also ask running unattended-upgrades to stop when shutdown starts even in InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to UnattendedUpgradesShutdown class. (LP: #1778219) * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219) This allows more time for unattended-upgrades to shut down gracefully or even install a few packages in InstallOnShutdown mode, but is still a big step back from the 30 minutes allowed for InstallOnShutdown previously. Users enabling InstallOnShutdown node are advised to increase InhibitDelayMaxSec even further possibly to 30 minutes. - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown changes * Ignore "W503 line break before binary operator" because it will become the best practice and breaks the build * Stop using ActionGroups, they interfere with apt.Cache.clear() causing all autoremovable packages to be handled as newly autoremovable ones and be removed by default. Dropping ActionGroup usage does not slow down the most frequent case of not having anything to upgrade and when there are packages to upgrade the gain is small compared to the actual package installation. Also collect autoremovable packages before adjusting candidates because that also changed .is_auto_removable attribute of some of them. (LP: #1803749) (Closes: #910874) unattended-upgrades (1.1ubuntu1.18.04.6) bionic; urgency=medium * Unlock for dpkg operations with apt_pkg.pkgsystem_unlock_inner() when it is available. Also stop running when reacquiring the lock fails. Thanks to Julian Andres Klode for original partial patch (LP: #1789637) * Skip rebuilding python-apt in upgrade autopkgtests. Python-apt has a new build dependency making the rebuilding as is failing and the reference handling issue is worked around in unattended-upgrades already. (LP: #1781586) * Stop trying when no adjustment could be made and adjust package candidates only to lower versions (LP: #1785093) * Skip already adjusted packages from being checked for readjusting. This makes it clearer that the recursion ends and can also be a bit quicker. (LP: #1785093) unattended-upgrades (1.1ubuntu1.18.04.5) bionic; urgency=medium * Stop updating the system when reacquiring the dpkg system lock fails. (LP: #1260041) unattended-upgrades (1.1ubuntu1.18.04.4) bionic; urgency=medium * Redirect stderr output in upgrade-between-snapshots, too, otherwise it breaks the test sometimes (LP: #1781446) unattended-upgrades (1.1ubuntu1.18.04.3) bionic; urgency=medium * Redirect stderr output in upgrade-all-security, otherwise it breaks the test (LP: #1781446) unattended-upgrades (1.1ubuntu1.18.04.2) bionic; urgency=medium [ Balint Reczey ] * Clear cache when autoremoval is invalid for a package set marked for removal and
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
@rbalint I removed the init-system-helpers task because it was marked as invalid. I think we should also remove snapd from this bug and open another one if there needs to be more work done. Otherwise, this bug will become confusing -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Description changed:
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades
that stat/stop services on shutdown, leaving the system in a broken
state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking
that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1
/org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting
after a few seconds with all updates installed
-
- * Since this fix is partially reverting the fix for LP: #1654600 please
- test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly
transition from coupling u-u.service with multi-user.target to coupling it with
shutdown.target again which can make u-u started during normal boot when there
is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and
targets u-u may fail to run on shutdown in case of an unexpected regression.
* There is a rarely occurring new crash caused by the (second) fix tracked in
LP: #1806487.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the
package installation on various packages hangs until the systemd
ShutdownTimeout (30min) is expired and systemd kills all processes and
powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least
sometimes this cannot be fixed with a "dpkg --configure -a", but instead
requires the user to manually reinstall the package that caused the
hang.
This appears to be a deadlock, because the hanging commands are always
"systemctl stop ..." or "systemctl restart ...", etc.. If I understand
this correctly, those systemctl commands block because systemd tries to
shutdown the system and tries to satisfy all dependencies for the
shutdown targets before those systemctl commands could get executed,
which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package.
The hanging command is "systemctl stop apport-forward.socket". The
system hangs until the systemd ShutdownTimeout expires and systemd
forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and
needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the
default configuration unattended-updates does not use bionic-updates,
and seems to have skipped installation of apport from bionic-security
(supposedly because an already newer version of apport was in bionic-
updates). If my understanding of why apport initially did not get
installed is correct, then this would be another
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Verified 1.1ubuntu1.18.04.7~16.04.2 on Xenial: root@x-uu:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false method return time=1554982311.759518 sender=:1.1 -> destination=:1.7 serial=19 reply_serial=2 root@x-uu:~# $ $ lxc shell x-uu root@x-uu:~# dpkg -l unattended-upgrades | cat Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ NameVersionArchitecture Description +++-===-==--=== ii unattended-upgrades 1.1ubuntu1.18.04.7~16.04.2 all automatic installation of security upgrades root@x-uu:~# cat /var/log/unattended-upgrades/unattended-upgrades.log 2019-04-11 11:31:44,714 INFO Initial blacklisted packages: 2019-04-11 11:31:44,715 INFO Initial whitelisted packages: 2019-04-11 11:31:44,715 INFO Starting unattended upgrades script 2019-04-11 11:31:44,715 INFO Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial, o=Ubuntu,a=xenial-updates 2019-04-11 11:31:52,034 INFO Initial blacklisted packages: 2019-04-11 11:31:52,034 INFO Initial whitelisted packages: 2019-04-11 11:31:52,035 INFO Starting unattended upgrades script 2019-04-11 11:31:52,035 INFO Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial, o=Ubuntu,a=xenial-updates 2019-04-11 11:31:59,193 INFO Packages that will be upgraded: snapd ubuntu-core-launcher 2019-04-11 11:31:59,194 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log 2019-04-11 11:32:06,043 INFO All upgrades installed ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
@mathew-hodson Could you please add a comment why you think init-system- helpers is not affected? It could handle service operations better when systemd already queued the shutdown, but since it would not have solved the graceful shutdown of u-u i did not fix it yet. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** No longer affects: init-system-helpers (Ubuntu) ** No longer affects: init-system-helpers (Ubuntu Xenial) ** No longer affects: init-system-helpers (Ubuntu Bionic) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
@andre-jutisz From the log it looks like unattended-upgrades stops gracefully during shutdown and there are not broken packages upon next invocation. Do you have other logs that suggest otherwise? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
I am still affected by this bug (Bionic server installation with the latest updates - 24.01.2019) Attached: unattended-upgrades.log ** Attachment added: "unattended-upgrades.log" https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1778219/+attachment/5232300/+files/unattended-upgrades.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Changed in: unattended-upgrades (Ubuntu Cosmic) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** No longer affects: apt (Ubuntu) ** No longer affects: apt (Ubuntu Xenial) ** No longer affects: apt (Ubuntu Bionic) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Description changed:
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades
that stat/stop services on shutdown, leaving the system in a broken
state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking
that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
- * Dowload packages for u-u:
-# unattended-upgrade --download-only
- * Reboot using logind to let inhibitors hold up shutdown:
+ * Dowload packages for u-u:
+ # unattended-upgrade --download-only
+ * Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1
/org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting
after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please
test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly
transition from coupling u-u.service with multi-user.target to coupling it with
shutdown.target again which can make u-u started during normal boot when there
is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and
targets u-u may fail to run on shutdown in case of an unexpected regression.
+ * There is a rarely occurring new crash caused by the (second) fix tracked in
LP: #1806487.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the
package installation on various packages hangs until the systemd
ShutdownTimeout (30min) is expired and systemd kills all processes and
powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least
sometimes this cannot be fixed with a "dpkg --configure -a", but instead
requires the user to manually reinstall the package that caused the
hang.
This appears to be a deadlock, because the hanging commands are always
"systemctl stop ..." or "systemctl restart ...", etc.. If I understand
this correctly, those systemctl commands block because systemd tries to
shutdown the system and tries to satisfy all dependencies for the
shutdown targets before those systemctl commands could get executed,
which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package.
The hanging command is "systemctl stop apport-forward.socket". The
system hangs until the systemd ShutdownTimeout expires and systemd
forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and
needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the
default configuration unattended-updates does not use bionic-updates,
and seems to have skipped installation of apport from bionic-security
(supposedly because an already newer version of apport
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Hello Cs-gon, or anyone else affected, Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags removed: verification-done ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7 --- unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown() Performing upgrades in service's ExecStop did not work when the upgrades involved restarting services because systemd blocked other stop/start actions making maintainer scripts time out and be killed leaving a broken system behind. Running unattended-upgrades.service before shutdown.target as a oneshot service made it run after unmounting filesystems and scheduling services properly on shutdown is a complex problem and adding more services to the mix make it even more fragile. The solution of monitoring PrepareForShutdown() signal from DBus allows Unattended Upgrade to run _before_ the jobs related to shutdown are queued thus package upgrades can safely restart services without risking causing deadlocks or breaking part of the shutdown actions. Also ask running unattended-upgrades to stop when shutdown starts even in InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to UnattendedUpgradesShutdown class. (LP: #1778219) * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219) This allows more time for unattended-upgrades to shut down gracefully or even install a few packages in InstallOnShutdown mode, but is still a big step back from the 30 minutes allowed for InstallOnShutdown previously. Users enabling InstallOnShutdown node are advised to increase InhibitDelayMaxSec even further possibly to 30 minutes. - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown changes * Ignore "W503 line break before binary operator" because it will become the best practice and breaks the build * Stop using ActionGroups, they interfere with apt.Cache.clear() causing all autoremovable packages to be handled as newly autoremovable ones and be removed by default. Dropping ActionGroup usage does not slow down the most frequent case of not having anything to upgrade and when there are packages to upgrade the gain is small compared to the actual package installation. Also collect autoremovable packages before adjusting candidates because that also changed .is_auto_removable attribute of some of them. (LP: #1803749) (Closes: #910874) -- Balint Reczey Mon, 26 Nov 2018 13:37:47 +0100 ** Changed in: unattended-upgrades (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
This bug was fixed in the package unattended-upgrades - 1.5ubuntu3.18.10.0 --- unattended-upgrades (1.5ubuntu3.18.10.0) cosmic; urgency=medium * Revert to running unattended-upgrades.service in multi-user.target * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown() Performing upgrades in service's ExecStop did not work when the upgrades involved restarting services because systemd blocked other stop/start actions making maintainer scripts time out and be killed leaving a broken system behind. Running unattended-upgrades.service before shutdown.target as a oneshot service made it run after unmounting filesystems and scheduling services properly on shutdown is a complex problem and adding more services to the mix make it even more fragile. The solution of monitoring PrepareForShutdown() signal from DBus allows Unattended Upgrade to run _before_ the jobs related to shutdown are queued thus package upgrades can safely restart services without risking causing deadlocks or breaking part of the shutdown actions. Also ask running unattended-upgrades to stop when shutdown starts even in InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to UnattendedUpgradesShutdown class. (LP: #1778219, LP: #1803137) * Handle reverting to WantedBy=multi-user.target * Increase logind's InhibitDelayMaxSec to 30s. This allows more time for unattended-upgrades to shut down gracefully or even install a few packages in InstallOnShutdown mode, but is still a big step back from the 30 minutes allowed for InstallOnShutdown previously. Users enabling InstallOnShutdown mode are advised to increase InhibitDelayMaxSec even further possibly to 30 minutes. * Cache polling result for PreparingForShutdown after it becomes true * debian/tests/test-systemd.py: Reboot system with dbus call to honor inhibitor locks * Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown changes * Stop using ActionGroups, they interfere with apt.Cache.clear() causing all autoremovable packages to be handled as newly autoremovable ones and be removed by default. Dropping ActionGroup usage does not slow down the most frequent case of not having anything to upgrade and when ther are packages to upgrade the gain is small compared to the actual package installation. Also collect autoremovable packages before adjusting candidates because that also changed .is_auto_removable attribute of some of them. (LP: #1803749) (Closes: #910874) -- Balint Reczey Mon, 26 Nov 2018 12:28:55 +0100 ** Changed in: unattended-upgrades (Ubuntu Cosmic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Verified 1.5ubuntu3.18.10.0 on cosmic with installing apport, rather
than snapd:
...
root@uu-shutdown-test:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root@uu-shutdown-test:~# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
root@uu-shutdown-test:~# unattended-upgrade --download-only
root@uu-shutdown-test:~# dbus-send --system --print-reply
--dest=org.freedesktop.login1 /org/freedesktop/login1
"org.freedesktop.login1.Manager.Reboot" boolean:false
method return time=1543423811.418244 sender=:1.4 -> destination=:1.27 serial=81
reply_serial=2
root@uu-shutdown-test:~#
Session terminated, terminating shell... ...terminated.
rbalint@yogi:~$ lxc shell uu-shutdown-test
mesg: ttyname failed: No such device
root@uu-shutdown-test:~# cat
/var/log/unattended-upgrades/unattended-upgrades-dpkg.log
Log started: 2018-11-28 16:50:13
(Reading database ... 28278 files and directories currently installed.)
Preparing to unpack .../apport_2.20.10-0ubuntu13.1_all.deb ...
Unpacking apport (2.20.10-0ubuntu13.1) over (2.20.10-0ubuntu13) ...
Processing triggers for systemd (239-7ubuntu10.4) ...
Setting up apport (2.20.10-0ubuntu13.1) ...
apport-autoreport.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.8.4-2) ...
Log ended: 2018-11-28 16:50:16
Installing apport locked up during shutdown when downgrading u-u to 1.4ubuntu2.
Latest unfixed u-u in cosmic does not lock up because the system shuts down
earlier with u-u still running (LP: #1803137), but this is fixed, too.
** Tags removed: verification-needed verification-needed-cosmic
** Tags added: verification-done verification-done-cosmic
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1778219
Title:
unattended-upgrades hangs on shutdown, leaves system in a broken state
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Verified 1.1ubuntu1.18.04.7 on Bionic:
...
root@uu-shutdown-test:~# service unattended-upgrades status
● unattended-upgrades.service - Unattended Upgrades Shutdown
Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled;
vendor preset: enabled)
Active: active (running) since Wed 2018-11-28 16:06:14 UTC; 1s ago
Docs: man:unattended-upgrade(8)
Main PID: 230 (unattended-upgr)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/unattended-upgrades.service
└─230 /usr/bin/python3
/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
Nov 28 16:06:14 uu-shutdown-test systemd[1]: unattended-upgrades.service:
Failed to reset devices.list: Operation not permitted
Nov 28 16:06:14 uu-shutdown-test systemd[1]: Started Unattended Upgrades
Shutdown.
root@uu-shutdown-test:~# vi /etc/apt/sources.list
root@uu-shutdown-test:~# apt update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 247 kB in 0s (505 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@uu-shutdown-test:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root@uu-shutdown-test:~# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
root@uu-shutdown-test:~#
root@uu-shutdown-test:~# apt install snapd=2.32.5+18.04
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
libfreetype6
Use 'apt autoremove' to remove it.
The following packages will be DOWNGRADED:
snapd
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 6 not upgraded.
Need to get 12.2 MB of archives.
After this operation, 860 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 snapd amd64
2.32.5+18.04 [12.2 MB]
Fetched 12.2 MB in 0s (75.4 MB/s)
dpkg: warning: downgrading snapd from 2.34.2+18.04 to 2.32.5+18.04
(Reading database ... 28538 files and directories currently installed.)
Preparing to unpack .../snapd_2.32.5+18.04_amd64.deb ...
Unpacking snapd (2.32.5+18.04) over (2.34.2+18.04) ...
Setting up snapd (2.32.5+18.04) ...
Installing new version of config file
/etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Installing new version of config file /etc/profile.d/apps-bin-path.sh ...
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
root@uu-shutdown-test:~# unattended-upgrade --download-only
root@uu-shutdown-test:~# dbus-send --system --print-reply
--dest=org.freedesktop.login1 /org/freedesktop/login1
"org.freedesktop.login1.Manager.Reboot" boolean:false
method return time=1543421249.487619 sender=:1.4 -> destination=:1.10 serial=18
reply_serial=2
root@uu-shutdown-test:~# tail -f
/var/log/unattended-upgrades/unattended-upgrades-dpkg.log
Log started: 2018-11-28 16:07:34
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.4_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.4) over (1:2.17.1-1ubuntu0.3) ...
Setting up git (1:2.17.1-1ubuntu0.4) ...
Log ended: 2018-11-28 16:07:39
Log started: 2018-11-28 16:07:40
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../git-man_1%3a2.17.1-1ubuntu0.4_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.4) over (1:2.17.1-1ubuntu0.3) ...
Setting up git-man (1:2.17.1-1ubuntu0.4) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Log ended: 2018-11-28 16:07:43
Log started: 2018-11-28 16:07:43
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../snapd_2.34.2+18.04_amd64.deb ...
Unpacking snapd (2.34.2+18.04) over (2.32.5+18.04) ...
Setting up snapd (2.34.2+18.04) ...
Installing new version of config file
/etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Installing new version of config file /etc/profile.d/apps-bin-path.sh ...
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Log ended: 2018-11-28 16:07:48
Log started: 2018-11-28 16:07:49
(Reading database ... 28538 files and directories currently installed.)
Preparing to unpack .../sosreport_3.6-1ubuntu0.18.04.2_amd64.deb ...
Unpacking sosreport (3.6-1ubuntu0.18.04.2) over (3.5-1ubuntu3.18.04.1) ...
Setting up sosreport (3.6-1ubuntu0.18.04.2) ...
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Description changed:
[Impact]
- * Unattended-upgrades hangs and gets killed when installing upgrades
+ * Unattended-upgrades hangs and gets killed when installing upgrades
that stat/stop services on shutdown, leaving the system in a broken
state
[Test Case]
- * Install an updated bionic system:
-$ lxc launch ubuntu:18.04 uu-shutdown-test
-# apt update
-...
+ * Install an updated bionic system:
+ $ lxc launch ubuntu:18.04 uu-shutdown-test
+ # apt update
+ ...
- * When testing the fixed version, install upgrade u-u at this point checking
that u-u.service is set up before and is wanted by shutdown.target:
+ * When testing the fixed version, install upgrade u-u at this point checking
that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
- "unattended-upgrades.service"->"-.mount" [color="green"];
- "unattended-upgrades.service"->"system.slice" [color="green"];
- "unattended-upgrades.service"->"network.target" [color="green"];
- "unattended-upgrades.service"->"systemd-journald.socket"
[color="green"];
- "unattended-upgrades.service"->"local-fs.target" [color="green"];
- "unattended-upgrades.service"->"-.mount" [color="black"];
- "unattended-upgrades.service"->"system.slice" [color="black"];
- "shutdown.target"->"unattended-upgrades.service" [color="green"];
- "shutdown.target"->"unattended-upgrades.service" [color="grey66"];
-Color legend: black = Requires
- dark blue = Requisite
- dark grey = Wants
- red = Conflicts
- green = After
+ "unattended-upgrades.service"->"-.mount" [color="green"];
+ "unattended-upgrades.service"->"system.slice" [color="green"];
+ "unattended-upgrades.service"->"network.target" [color="green"];
+ "unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
+ "unattended-upgrades.service"->"local-fs.target" [color="green"];
+ "unattended-upgrades.service"->"-.mount" [color="black"];
+ "unattended-upgrades.service"->"system.slice" [color="black"];
+ "shutdown.target"->"unattended-upgrades.service" [color="green"];
+ "shutdown.target"->"unattended-upgrades.service" [color="grey66"];
+ Color legend: black = Requires
+ dark blue = Requisite
+ dark grey = Wants
+ red = Conflicts
+ green = After
- * Configure u-u to run on shutdown and install -updates:
-# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
-# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
+ * Configure u-u to run on shutdown and install -updates:
+ # echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
+ # echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
- * Downgrade snapd:
-# apt install snapd=2.32.5+18.04
-
- * # reboot
+ * Downgrade snapd:
+ # apt install snapd=2.32.5+18.04
+ * Dowload packages for u-u:
+# unattended-upgrade --download-only
+ * Reboot using logind to let inhibitors hold up shutdown:
+ # dbus-send --system --print-reply --dest=org.freedesktop.login1
/org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
- * With not fixed u-u observe the upgrade process being stuck:
- # pstree| grep unatt
-
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
- `-{unattended-upgr}
+ * With not fixed u-u observe the upgrade process being stuck:
+ # pstree| grep unatt
+
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
+ `-{unattended-upgr}
- * With fixed u-u observe snapd update taking place and system rebooting
+ * With fixed u-u observe snapd update taking place and system rebooting
after a few seconds with all updates installed
- * Since this fix is partially reverting the fix for LP: #1654600 please
+ * Since this fix is partially reverting the fix for LP: #1654600 please
test LP: #1654600 as well to avoid regressions.
[Regression Potential]
- * As part of the fix manual changes were made to postinst to properly
transition from coupling u-u.service with multi-user.target to coupling it with
shutdown.target again which can make u-u started during normal boot when there
is a bug in the implementation.
+ * As part of the fix manual changes were made to postinst to properly
transition from coupling u-u.service with multi-user.target to coupling it with
shutdown.target again which can make u-u
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Hello Cs-gon, or anyone else affected, Accepted unattended-upgrades into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/1.5ubuntu3.18.10.0 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: unattended-upgrades (Ubuntu Cosmic) Status: New => Fix Committed ** Tags added: verification-needed verification-needed-cosmic ** Changed in: unattended-upgrades (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: snapd (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: apt (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: snapd (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: apt (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: init-system-helpers (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: init-system-helpers (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: snapd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Also affects: apt (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: unattended-upgrades (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: init-system-helpers (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: snapd (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: unattended-upgrades (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: New => In Progress ** Changed in: unattended-upgrades (Ubuntu Xenial) Assignee: (unassigned) => Balint Reczey (rbalint) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Changed in: unattended-upgrades (Ubuntu Bionic) Assignee: Eric Desrochers (slashd) => (unassigned) ** Changed in: unattended-upgrades (Ubuntu Bionic) Assignee: (unassigned) => Balint Reczey (rbalint) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
@slashd (and all) the fix caused regressions elsewhere and it is being partially reverted and the issue is fixed in a different way. Please don't backport the current fix. The candidate for fixing the issue without the regressions introduced: https://github.com/mvo5/unattended-upgrades/pull/148 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Changed in: unattended-upgrades (Ubuntu Bionic) Status: Confirmed => In Progress ** Changed in: unattended-upgrades (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: unattended-upgrades (Ubuntu Bionic) Assignee: (unassigned) => Eric Desrochers (slashd) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
This affects Bionic reported on (LP: #1796376). It was only fixed for Cosmic. I have tested w/ a test package of 'u-u' and it fixes the problem so far, but I have asked impacted user to give it a try as well for double- confirmation. If it fixes the problem, I'll SRU it for Bionic. - Eric ** Also affects: apt (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: unattended-upgrades (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: init-system-helpers (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: snapd (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: unattended-upgrades (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
@juliank nevermind, i'm adding the inhibition support to u-u, thus apt does not need patching. ** Changed in: apt (Ubuntu) Status: New => Won't Fix ** Changed in: apt (Ubuntu) Status: Won't Fix => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
The hang can also occur when unattended-upgrades is started by apt's
timer (u-u is modified to help hitting it at the right time):
$ cat reproduce-apt-ran-uu.sh
#!/bin/sh
set -e
apt update
apt upgrade -y
echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
case $(lsb_release -c -s) in
bionic)
apt install --allow-downgrades -y apport=2.20.9-0ubuntu7
snapd=2.32.5+18.04
;;
esac
grep -q 'sleep' /usr/bin/unattended-upgrade || sed -i 's/import
atexit/import time\nimport atexit/;s/res = cache\.commit/time.sleep(5)\n
res = cache.commit/' /usr/bin/unattended-upgrade
rm -f /var/lib/apt/periodic/u*
apt upgrade -d -y
service apt-daily-upgrade start &
sleep 7
reboot
@juliank In https://bugs.launchpad.net/ubuntu/+source/unattended-
upgrades/+bug/1690980/comments/50 you suggested inhibiting shutdown for
the commit operations only in u-u, but IMO that would still leave the
race condition open where the user can start shutdown between commits
thus possibly hanging the system with the issue reported in this bug.
One option (1) would be u-u starting the inhibition at the beginning of its run
via dbus, the other (2) would be apt.systemd.daily calling u-u with the
systemd-inhibit wrapper.
2 would be cleaner and u-u would not have to grow dependency on Python3 dbus
modules.
Apt.systemd.daily calling u-u with systemd-inhibit would not back
shutdown for long on average since it is already unlikely to hit an
upgrade with a shutdown, but in the worst case installing all bionic
security updates can take 10-s of minutes. Again, this is a highly
unlikely scenario, but if users are hitting that u-u could still get
support for monitoring inhibited shutdown requests on dbus and
gracefully stop without applying all updates.
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac
/autopkgtest-bionic/bionic/amd64/u/unattended-
upgrades/20181010_011153_2b8c7@/log.gz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1778219
Title:
unattended-upgrades hangs on shutdown, leaves system in a broken state
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Also affects: apt (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Changed in: init-system-helpers (Ubuntu) Assignee: Balint Reczey (rbalint) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
This bug was fixed in the package unattended-upgrades - 1.5ubuntu2 --- unattended-upgrades (1.5ubuntu2) cosmic; urgency=medium * Reopen Cache after commit() even when frontend locking is supported. This fixes build and operation with latest python-apt. -- Balint Reczey Tue, 28 Aug 2018 15:46:25 +0200 ** Changed in: unattended-upgrades (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Description changed:
+ [Impact]
+
+ * Unattended-upgrades hangs and gets killed when installing upgrades
+ that stat/stop services on shutdown, leaving the system in a broken
+ state
+
+ [Test Case]
+
+ * Install an updated bionic system:
+$ lxc launch ubuntu:18.04 uu-shutdown-test
+# apt update
+...
+
+ * When testing the fixed version, install upgrade u-u at this point checking
that u-u.service is set up before and is wanted by shutdown.target:
+ # systemd-analyze dot | grep unatt
+ "unattended-upgrades.service"->"-.mount" [color="green"];
+ "unattended-upgrades.service"->"system.slice" [color="green"];
+ "unattended-upgrades.service"->"network.target" [color="green"];
+ "unattended-upgrades.service"->"systemd-journald.socket"
[color="green"];
+ "unattended-upgrades.service"->"local-fs.target" [color="green"];
+ "unattended-upgrades.service"->"-.mount" [color="black"];
+ "unattended-upgrades.service"->"system.slice" [color="black"];
+ "shutdown.target"->"unattended-upgrades.service" [color="green"];
+ "shutdown.target"->"unattended-upgrades.service" [color="grey66"];
+Color legend: black = Requires
+ dark blue = Requisite
+ dark grey = Wants
+ red = Conflicts
+ green = After
+
+ * Configure u-u to run on shutdown and install -updates:
+# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
+# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
+
+ * Downgrade snapd:
+# apt install snapd=2.32.5+18.04
+
+ * # reboot
+
+ * With not fixed u-u observe the upgrade process being stuck:
+ # pstree| grep unatt
+
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
+ `-{unattended-upgr}
+
+ * With fixed u-u observe snapd update taking place and system rebooting
+ after a few seconds with all updates installed
+
+ * Since this fix is partially reverting the fix for LP: #1654600 please
+ test LP: #1654600 as well to avoid regressions.
+
+ [Regression Potential]
+
+ * As part of the fix manual changes were made to postinst to properly
transition from coupling u-u.service with multi-user.target to coupling it with
shutdown.target again which can make u-u started during normal boot when there
is a bug in the implementation.
+ on-testing the SRU.
+ * Due to relationship changes between u-u.service, other services and
targets u-u may fail to run on shutdown in case of an unexpected regression.
+
+ [Original Bug Text]
+
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the
package installation on various packages hangs until the systemd
ShutdownTimeout (30min) is expired and systemd kills all processes and
powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least
sometimes this cannot be fixed with a "dpkg --configure -a", but instead
requires the user to manually reinstall the package that caused the
hang.
This appears to be a deadlock, because the hanging commands are always
"systemctl stop ..." or "systemctl restart ...", etc.. If I understand
this correctly, those systemctl commands block because systemd tries to
shutdown the system and tries to satisfy all dependencies for the
shutdown targets before those systemctl commands could get executed,
which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package.
The hanging command is "systemctl stop apport-forward.socket". The
system hangs until the systemd ShutdownTimeout expires and systemd
forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and
needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the
default configuration unattended-updates does not use bionic-updates,
and seems to have skipped installation of apport from bionic-security
(supposedly because an already newer version of apport was in bionic-
updates). If my understanding of why apport initially did not get
installed is correct, then this would be another problem, because it
would mean that unattended-upgrades potentially does not install all
available security updates when bionic-updates is enabled in
sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in
sources.list, but instead enabling
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
The fix for LP: #1654600 in addition to making u-u keep /var mounted also made u-u perform the upgrades in ExecStop that causes hanging when an upgrade would start/stop services. ** Changed in: unattended-upgrades (Ubuntu) Importance: Undecided => High ** Changed in: snapd (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Changed in: init-system-helpers (Ubuntu) Status: In Progress => Invalid ** Changed in: unattended-upgrades (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Changed in: init-system-helpers (Ubuntu) Status: New => In Progress ** Changed in: init-system-helpers (Ubuntu) Assignee: (unassigned) => Balint Reczey (rbalint) ** Changed in: init-system-helpers (Ubuntu) Importance: Undecided => Critical ** Changed in: init-system-helpers (Ubuntu) Importance: Critical => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Tags added: id-5b3568d38c766fbe4a6b1710 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
An approximation of the packages that need to be fixed individually are listed here: https://lintian.debian.org/tags/maintainer-script-calls-systemctl.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Tags added: rls-cc-incoming -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
This is not a bug in unattended-upgrades, but u-u triggers the problem in many packages which are not able to install/upgrade during a system shutdown. Helper scripts in init-system-helpers must not start services when maintainer scripts are executed during shutdown and maintainer scripts must use helper scripts and not call systemct and friends directly to start services. Snapd for example calls systemctl start "$unit" in postinst. Thank you for the bug report. I marked it "Confirmed for u-u" to not let it disappear, but u-u is just triggering the issue and it can't be fixed in u-u. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Also affects: init-system-helpers (Ubuntu) Importance: Undecided Status: New ** Also affects: snapd (Ubuntu) Importance: Undecided Status: New ** Changed in: unattended-upgrades (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/init-system-helpers/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
** Description changed: When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system. This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang. This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock. - Steps to reproduce: - Install 18.04 - activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades - disable bionic-updates in /etc/apt/sources.list (more on that later) - execute "unattended-upgrade --download-only" - reboot the system -> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The - system hangs until it the systemd ShutdownTimeout expires and systemd + system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system. - After the system is rebooted the apport package is in "iHR" state, and + After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this. - I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic- updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default). - The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above. - --- + --- ApportVersion: 2.20.9-0ubuntu7.1 Architecture: amd64 CurrentDesktop: ubuntu:GNOME DistroRelease: Ubuntu 18.04 InstallationDate: Installed on 2018-06-22 (0 days ago) InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426) Package: unattended-upgrades 1.1ubuntu1 PackageArchitecture: all ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17 Tags: bionic Uname: Linux 4.15.0-20-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo _MarkForUpload: True -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state
apport information ** Tags added: apport-collected ** Description changed: When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system. This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang. This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock. Steps to reproduce: - Install 18.04 - activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades - disable bionic-updates in /etc/apt/sources.list (more on that later) - execute "unattended-upgrade --download-only" - reboot the system -> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until it the systemd ShutdownTimeout expires and systemd forcefully reboots the system. After the system is rebooted the apport package is in "iHR" state, and needs to be reinstalled to fix this. I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic- updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default). The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above. + --- + ApportVersion: 2.20.9-0ubuntu7.1 + Architecture: amd64 + CurrentDesktop: ubuntu:GNOME + DistroRelease: Ubuntu 18.04 + InstallationDate: Installed on 2018-06-22 (0 days ago) + InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426) + Package: unattended-upgrades 1.1ubuntu1 + PackageArchitecture: all + ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17 + Tags: bionic + Uname: Linux 4.15.0-20-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo + _MarkForUpload: True ** Attachment added: ".var.log.apt.history.log.txt" https://bugs.launchpad.net/bugs/1778219/+attachment/5155525/+files/.var.log.apt.history.log.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778219 Title: unattended-upgrades hangs on shutdown, leaves system in a broken state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1778219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
