[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2 --- unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium * Don't check blacklist too early and report updates from not allowed origins as kept back. (LP: #1781176) * test/test_blacklisted_wrong_origin.py: Fix and enable test * Filter out progress indicator from dpkg log (LP: #1599646) * Clear cache when autoremoval fails (LP: #1779157) * Find autoremovable kernel packages using the patterns in APT's way (LP: #1815494) unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium * Start service after systemd-logind.service to be able to take inhibition lock (LP: #1806487) * Handle gracefully when logind is down (LP: #1806487) unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium * Backport to Xenial (LP: #1702793) * Revert to build-depending on debhelper (>= 9~) and dh-systemd * Revert configuration example changes to avoid triggering a debconf question * debian/postinst: Update recovery to be triggered on Xenial's package versions unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown() Performing upgrades in service's ExecStop did not work when the upgrades involved restarting services because systemd blocked other stop/start actions making maintainer scripts time out and be killed leaving a broken system behind. Running unattended-upgrades.service before shutdown.target as a oneshot service made it run after unmounting filesystems and scheduling services properly on shutdown is a complex problem and adding more services to the mix make it even more fragile. The solution of monitoring PrepareForShutdown() signal from DBus allows Unattended Upgrade to run _before_ the jobs related to shutdown are queued thus package upgrades can safely restart services without risking causing deadlocks or breaking part of the shutdown actions. Also ask running unattended-upgrades to stop when shutdown starts even in InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to UnattendedUpgradesShutdown class. (LP: #1778219) * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219) This allows more time for unattended-upgrades to shut down gracefully or even install a few packages in InstallOnShutdown mode, but is still a big step back from the 30 minutes allowed for InstallOnShutdown previously. Users enabling InstallOnShutdown node are advised to increase InhibitDelayMaxSec even further possibly to 30 minutes. - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown changes * Ignore "W503 line break before binary operator" because it will become the best practice and breaks the build * Stop using ActionGroups, they interfere with apt.Cache.clear() causing all autoremovable packages to be handled as newly autoremovable ones and be removed by default. Dropping ActionGroup usage does not slow down the most frequent case of not having anything to upgrade and when there are packages to upgrade the gain is small compared to the actual package installation. Also collect autoremovable packages before adjusting candidates because that also changed .is_auto_removable attribute of some of them. (LP: #1803749) (Closes: #910874) unattended-upgrades (1.1ubuntu1.18.04.6) bionic; urgency=medium * Unlock for dpkg operations with apt_pkg.pkgsystem_unlock_inner() when it is available. Also stop running when reacquiring the lock fails. Thanks to Julian Andres Klode for original partial patch (LP: #1789637) * Skip rebuilding python-apt in upgrade autopkgtests. Python-apt has a new build dependency making the rebuilding as is failing and the reference handling issue is worked around in unattended-upgrades already. (LP: #1781586) * Stop trying when no adjustment could be made and adjust package candidates only to lower versions (LP: #1785093) * Skip already adjusted packages from being checked for readjusting. This makes it clearer that the recursion ends and can also be a bit quicker. (LP: #1785093) unattended-upgrades (1.1ubuntu1.18.04.5) bionic; urgency=medium * Stop updating the system when reacquiring the dpkg system lock fails. (LP: #1260041) unattended-upgrades (1.1ubuntu1.18.04.4) bionic; urgency=medium * Redirect stderr output in upgrade-between-snapshots, too, otherwise it breaks the test sometimes (LP: #1781446) unattended-upgrades (1.1ubuntu1.18.04.3) bionic; urgency=medium * Redirect stderr output in upgrade-all-security, otherwise it breaks the test (LP: #1781446) unattended-upgrades (1.1ubuntu1.18.04.2) bionic; urgency=medium [ Balint Reczey ] * Clear cache when autoremoval is invalid for a package set marked for removal and
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Already verified in 0.90ubuntu0.5 for Xenial, not additional test is needed. ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello Erik, or anyone else affected, Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/1.1ubuntu1.18.04.7~16.04.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Fix Released => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.10 --- unattended-upgrades (0.90ubuntu0.10) xenial-security; urgency=medium * No change rebuild in the -security pocket (See LP #1686470) -- Marc Deslauriers Fri, 18 Jan 2019 13:34:27 -0500 ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Bug watch removed: Debian Bug tracker #797108 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797108 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello Erik, or anyone else affected, Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Fix Released => Fix Committed ** Tags removed: verification-done ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Changed in: unattended-upgrades (Ubuntu Xenial) Assignee: Louis Bouchard (louis) => (unassigned) ** Changed in: unattended-upgrades (Ubuntu Yakkety) Assignee: Louis Bouchard (louis) => (unassigned) ** Changed in: unattended-upgrades (Ubuntu Zesty) Assignee: Louis Bouchard (louis) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu2.1 --- unattended-upgrades (0.93.1ubuntu2.1) zesty; urgency=medium * Complete the solution for the unattended-upgrades.service unit not correctly working (LP: #1654600): - d/rules : Remove the override_dh_installinit. The stop option is no longer available so the command falls back to default. This is the normal behavior so the override is not required - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the unattended-upgrades.service unit cannot be enabled on boot - d/postinst : Cleanup the stop symlinks created by the wrong override_dh_installinit. Without that, the systemd unit cannot be enabled correctly. Force disable the service before deb-systemd-helper runs so the old symlink is not left dangling (workaround for Debian Bug #797108). Force enable and start of the systemd unit to work around Debian Bug #797108 which fails to enable systemd units correctly when WantedBy= statement is changed which is the case here. - d/unattended-upgrades.service : Fix the service so it runs correctly on shutdown : Remove DefaultDependencies=no : Breaks normal shutdown dependencies Set After= to network.target and local-fs.target. Since our service is now ExecStop, it will run before network and local-fs become unavailable. Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if /var is a separate file system. Set WantedBy= to multi-user.target - Add DEP8 tests to verify the following : Verify that the unattended-upgrades.service unit is enabled and started. Verify that InstallOnShutdown works when configured. -- Louis BouchardFri, 21 Apr 2017 12:40:29 +0200 ** Changed in: unattended-upgrades (Ubuntu Zesty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 0.92ubuntu1.4 --- unattended-upgrades (0.92ubuntu1.4) yakkety; urgency=medium * Complete the solution for the unattended-upgrades.service unit not correctly working (LP: #1654600): - d/rules : Remove the override_dh_installinit. The stop option is no longer available so the command falls back to default. This is the normal behavior so the override is not required - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the unattended-upgrades.service unit cannot be enabled on boot - d/postinst : Cleanup the stop symlinks created by the wrong override_dh_installinit. Without that, the systemd unit cannot be enabled correctly. Force disable the service before deb-systemd-helper runs so the old symlink is not left dangling (workaround for Debian Bug #797108). Force enable and start of the systemd unit to work around Debian Bug #797108 which fails to enable systemd units correctly when WantedBy= statement is changed which is the case here. - d/unattended-upgrades.service : Fix the service so it runs correctly on shutdown : Remove DefaultDependencies=no : Breaks normal shutdown dependencies Set After= to network.target and local-fs.target. Since our service is now ExecStop, it will run before network and local-fs become unavailable. Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if /var is a separate file system. Set WantedBy= to multi-user.target - Add DEP8 tests to verify the following : Verify that the unattended-upgrades.service unit is enabled and started. Verify that InstallOnShutdown works when configured. -- Louis BouchardTue, 04 Apr 2017 18:31:42 +0200 ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: Fix Committed => Fix Released ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.5 --- unattended-upgrades (0.90ubuntu0.5) xenial; urgency=medium * Complete the solution for the unattended-upgrades.service unit not correctly working (LP: #1654600): - d/rules : Remove the override_dh_installinit. The stop option is no longer available so the command falls back to default. This is the normal behavior so the override is not required - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the unattended-upgrades.service unit cannot be enabled on boot - d/postinst : Cleanup the stop symlinks created by the wrong override_dh_installinit. Without that, the systemd unit cannot be enabled correctly. Force disable the service before deb-systemd-helper runs so the old symlink is not left dangling (workaround for Debian Bug #797108). Force enable and start of the systemd unit to work around Debian Bug #797108 which fails to enable systemd units correctly when WantedBy= statement is changed which is the case here. Both systemctl enable AND start are needed as enable --now fails on Xenial. - d/unattended-upgrades.service : Fix the service so it runs correctly on shutdown : Remove DefaultDependencies=no : Breaks normal shutdown dependencies Set After= to network.target and local-fs.target. Since our service is now ExecStop, it will run before network and local-fs become unavailable. Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if /var is a separate file system. Set WantedBy= to multi-user.target - Add DEP8 tests to verify the following : Verify that the unattended-upgrades.service unit is enabled and started. Verify that InstallOnShutdown works when configured. -- Louis BouchardFri, 21 Apr 2017 16:47:52 +0200 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Changed in: unattended-upgrades (Debian) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This resolved the issue. Version of the package testes: unattended-upgrades 0.90ubuntu0.5 Ubuntu version - 16.04 xenial verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Test performed for verification-done for : - Xenial - Yakkety - Zesty 1) Installed unattended-upgrades from -proposed 2) Verified the status of the unattended-upgrades service. All displayed : # systemctl status unattended-upgrades ● unattended-upgrades.service - Unattended Upgrades Shutdown Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2017-04-28 14:02:35 CEST; 47s ago Docs: man:unattended-upgrade(8) 3) Enabled Unattended-Upgrade::InstallOnShutdown "true"; in /etc/apt/apt.conf.d/50unattended-upgrades and rebooted. All releases applied the upgradeable packages upon reboot (some had too many and the timeout ran out before end of update) 4) Ran apt -y dist-upgrade on all releases. Rebooted. systems rebooted normally. 5) Enabled trusty-proposed on a Trusty system and ran do-release-upgrade to go to Xenial. After completion of the full upgrade, unattended- upgrades service was active (as seen in #2) 6) Enabled zesty-proposed on a system with /var as a separate file system. Upgraded unattended-upgrades. After upgrade, unattended-upgrades service was active (as seen in #2) 7) Verified that the workaround for Debian Bug #809669 left the upgraded system in an adequate configuration : cat /var/lib/systemd/deb-systemd-helper-enabled/unattended-upgrades.service.dsh-also /etc/systemd/system/multi-user.target.wants/unattended-upgrades.service # find /etc/systemd | grep unatt /etc/systemd/system/multi-user.target.wants/unattended-upgrades.service -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Tags removed: sts-sru-needed verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello Erik, or anyone else affected, Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/0.90ubuntu0.5 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello Erik, or anyone else affected, Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/0.92ubuntu1.4 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello Erik, or anyone else affected, Accepted unattended-upgrades into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended- upgrades/0.93.1ubuntu2.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: unattended-upgrades (Ubuntu Zesty) Status: In Progress => Fix Committed ** Tags removed: verification-failed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Description changed: [SRU justification] This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system [Impact] System can hang up to 10 minutes if not fixed. [Fix] Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced. + Change WantedBy to multi-user.target. This requires working around Debian Bug #797108 which causes the new unit not to be enabled. + Remove the unneeded override_dh_isntallinit + Add Default-Start levels to the SysV initscript + Improve DEP8 testing [Test Case] In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes [Regression] - None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously. + Upgrade has been tested on Xenial, Yakkety, Zesty. do-release-upgrade has been tested from Trusty to Xenial. All behave as expected. + + A change of behavior may occur now that the systemd unit is correctly + enabled, which would make functionalities like InstallOnShutdown to work + as expected whereas it could have been broken previously. This cannot be + considered as a regression as it is expected behavior. + + Shutdown may be slowed down as it now correctly depends on /var and + /boot to be available so the unit will run earlier than previously. [Original description of the problem] The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade process during shutdown. This unit file is running together with all filesystem unmount services. The unattended-upgrades service checks if the lockfile for unattended-upgrade (in /var/run) exists, and if it does, there is an unattended-upgrade in progress and the service will wait until it finishes (and therefore automatically wait at shutdown). However, if /var is a separate filesystem, it will get unmounted even though /var/run is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem. The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and tries to check the lockfile again. After 10 minutes (the default timeout), it will finally exit and the system will continue shutdown. The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown where it tries to lock itself: while True: res = apt_pkg.get_lock(options.lock_file) logging.debug("get_lock returned %i" % res) # exit here if there is no lock if res > 0: logging.debug("lock not taken") break lock_was_taken = True The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error. File descriptors are just C file descriptors, so they are always positive integers. The code should check the result to be negative, not positive. I have attached a patch to reverse the logic. Additional information: 1) Description: Ubuntu 16.04.1 LTS Release: 16.04 2) unattended-upgrades: Installed: 0.90ubuntu0.3 Candidate: 0.90ubuntu0.3 Version table: *** 0.90ubuntu0.3 500 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages 100 /var/lib/dpkg/status 0.90 500 500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages 500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages 3) Fast reboot 4) Very slow reboot (after a 10 minutes timeout) ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: Triaged => In Progress ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Triaged => In Progress ** Changed in: unattended-upgrades (Ubuntu) Assignee: Louis Bouchard (louis) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu3 --- unattended-upgrades (0.93.1ubuntu3) artful; urgency=medium * Complete the solution for the unattended-upgrades.service unit not correctly working (LP: #1654600): - d/rules : Remove the override_dh_installinit. The stop option is no longer available so the command falls back to default. This is the normal behavior so the override is not required - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the unattended-upgrades.service unit cannot be enabled on boot - d/postinst : Cleanup the stop symlinks created by the wrong override_dh_installinit. Without that, the systemd unit cannot be enabled correctly. Force disable the service before deb-systemd-helper runs so the old symlink is not left dangling (workaround for Debian Bug #797108). Force enable and start of the systemd unit to work around Debian Bug #797108 which fails to enable systemd units correctly when WantedBy= statement is changed which is the case here. - d/unattended-upgrades.service : Fix the service so it runs correctly on shutdown : Remove DefaultDependencies=no : Breaks normal shutdown dependencies Set After= to network.target and local-fs.target. Since our service is now ExecStop, it will run before network and local-fs become unavailable. Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if /var is a separate file system. Set WantedBy= to multi-user.target - Add DEP8 tests to verify the following : Verify that the unattended-upgrades.service unit is enabled and started. Verify that InstallOnShutdown works when configured. -- Louis BouchardMon, 24 Apr 2017 14:07:19 +0200 ** Changed in: unattended-upgrades (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Brief update to tell everyone that I have a fix ready for all stable releases. I will update it to Artful soon and the SRU will follow soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Targetting Zesty as the fix will come after release so as a SRU. ** Also affects: unattended-upgrades (Ubuntu Zesty) Importance: High Assignee: Louis Bouchard (louis-bouchard) Status: In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Tags added: sts-sru-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
A quick update on this issue. I have a solution ready to fix this. Unfortunately it requires changing the WantedBy= statement, which triggers a deb-systemd-helper bug : https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=797108 This has the effect of leaving the service disabled after upgrade. I'm working on fixing that. ** Bug watch added: Debian Bug tracker #797108 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797108 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Follow-up to #34, on a system where unattended-upgrades happens to be running when shutdown is called, the shutdown does not wait, therefore any running dpkg is killed and the package left in an inconsistent state. - Xenial with single partition, InstallOnShutdown not defined. This may be a separate issue, but I'll investigate further before raising it as such. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
The Unit file in #33 works for me, ie stops the 10 minute hang, when unattended-upgrades is not running. - Xenial with /var as a separate FS, InstallOnShutdown="false" I will try to create a test case where unattended-upgrades is running when shutdown is called. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Here is a recap of my work and current status : #1) when a system has a separated /var, the unit will hang since it is looking for /var/run to be present and it has been unmounted. #2) when using Unattended-Upgrade::InstallOnShutdown "true"; the upgrade never completes as the query to the online archive fails since the network is no longer available. #3) it is impossible to enable the unattended-upgrades.service unit. Here is an example : > $ systemctl status unattended-upgrades.service > ● unattended-upgrades.service - Unattended Upgrades Shutdown >Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; > vendor preset: enabled) >Active: inactive (dead) > Docs: man:unattended-upgrade(8) According to the doc[2], during shutdown, even if Before= or After= is used, the unit being started will only start after the Shutdown of its dependencies. Prior to the recent addition of local-fs.target network.target, the unit ran quickly enough for the /var to be still mounted. But even without these dependencies, InstallOnShutdown would fail with the following : > 2017-03-10 13:40:42,803 INFO Starting unattended upgrades script > 2017-03-10 13:40:42,803 INFO Allowed origins are: ['o=Ubuntu,a=zesty', > 'o=Ubuntu,a=zesty-security'] > 2017-03-10 13:41:40,554 ERROR An error occurred: 'Cannot initiate the > connection to 192.168.200.3:8000 (192.168.200.3). - connect (101: Network is > unreachable)' > 2017-03-10 13:41:40,555 ERROR The URI > 'http://fr.archive.ubuntu.com/ubuntu/pool/main/i/init-system-helpers/init-system-helpers_1.47_all.deb' > failed to download, aborting When trying to switch the unit to an ExecStop=, we find that the Stop never runs. This is caused by the fact that the unit is disabled (#3). Trying to enable the unit leads to : > # systemctl enable unattended-upgrades > Synchronizing state of unattended-upgrades.service with SysV service script > with /lib/systemd/systemd-sysv-install. > Executing: /lib/systemd/systemd-sysv-install enable unattended-upgrades > update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, > aborting. Adding runlevels 2 3 4 5 fixes this then the unit can be enabled. So we get to a unit that looks like this : > [Unit] > Description=Unattended Upgrades Shutdown > DefaultDependencies=no > After=network.target local-fs.target > RequiresMountsFor=/var/run /var/log > Documentation=man:unattended-upgrade(8) > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > TimeoutStopSec=900 > > [Install] > WantedBy=multi-user.target Before= is replaced by After= as, during shutdown otherwise the unit does not run. RequiresMountsFor= is added as both /var/log and /var/run are needed in order to run correctly. RemainAfterExit=yes is added so the unit appears as started. There is no longer a requirement to have an ExecStart present. WantedBy is switched to multi-user.target as on the way up, we do nothing and we are no longer depending on anything related to shutdown. Now this only works IF /var is a separate FS. The reason for that is the presence of DefaultDependencies=no. I don't think that it is required but was there in the initial unit. Removing it in the final unit fixes the only remaining issue. The unit is now : > [Unit] > Description=Unattended Upgrades Shutdown > After=network.target local-fs.target > RequiresMountsFor=/var/run /var/log > Documentation=man:unattended-upgrade(8) > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > TimeoutStopSec=900 > > [Install] > WantedBy=multi-user.target This works correctly and has been tested on : - Xenial with and without /var as a separate FS - Zesty with and without /var as a separate FS InstallOnShutdown now also works as advertized. I am now getting confirmation on that change since it is a rather sensible modification. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Still doesn't work for me. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Here is a recap of my day of work : The unattended-upgrades.service unit never runs the ExecStop because it is not enabled as we see here : # systemctl status unattended-upgrades.service ● unattended-upgrades.service - Unattended Upgrades Shutdown Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled) Active: inactive (dead) <<< Docs: man:unattended-upgrade(8) This is caused by a missing "Default-Start" header in /etc/init.d /unattended-upgrades. Trying to enable the unit leads to : # systemctl enable unattended-upgrades.service Synchronizing state of unattended-upgrades.service with SysV init with /lib/systemd/systemd-sysv-install... Executing /lib/systemd/systemd-sysv-install enable unattended-upgrades update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting. I'm currently working on fixing all this in the packaging so we get a correctly working unit after the upgrade of the package. I have also reworked the unit : [Unit] Description=Unattended Upgrades Shutdown DefaultDependencies=no After=network.target local-fs.target RequiresMountsFor=/var/log /var/run Documentation=man:unattended-upgrade(8) [Service] Type=oneshot RemainAfterExit=yes ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown TimeoutStopSec=900 [Install] WantedBy=multi-user.target With this configuration, the unit runs the ExecStop as expected. I also tested using Unattended-Upgrade::InstallOnShutdown "true" and it works as expected, which is to block the shutdown for upgrade with the network being available. I should have that available for testing tomorrow. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Changed in: unattended-upgrades (Ubuntu) Status: Fix Released => In Progress ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Fix Committed => Triaged ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: Fix Committed => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
@sergei-franco: >From what I can tell, it works for you because changing the unit the way you did simply disable execution of the unit. When running with such a configuration either on Xenial or Zesty, I do not see any mention of "Unattended Upgrades Shutdown" in the console output. When limiting to "After=network.target local-fs.target", I see the unit being started (which is basically a NOOP) but not stopping which is what we want. I'm continuing my tests -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello! I have changed Before to After and it is working Here is what the unit looks like now: [Unit] Description=Unattended Upgrades Shutdown DefaultDependencies=no After=shutdown.target reboot.target halt.target network.target local-fs.target Documentation=man:unattended-upgrade(8) [Service] Type=oneshot RemainAfterExit=yes ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown TimeoutStopSec=900 [Install] WantedBy=shutdown.target So the proper fix would be changing the Service to ExecStop= and Unit to After= -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello, Yes, I am fully aware that my /var/run->/run replacement in that script is a hack. To be clear I am not testing in Zesty but Ubuntu 16.04.2 LTS Xenial. The setup is following: /dev/sda1 on / type xfs (rw,relatime,attr2,inode64,noquota) /dev/sdc on /var type xfs (rw,relatime,attr2,inode64,noquota) Every single machine we (my and my colleague) tested with the updated unit still hangs. As var as I can tell they never reboot. I have posted a question regarding this issue to systemd mailing list, and one of the responses was the following: """ > [Unit] > Description=Unattended Upgrades Shutdown > DefaultDependencies=no > Before=shutdown.target reboot.target halt.target network.target > local-fs.target Well, you order if before local-fs.target, which means it is stopped after local-fs.target. You get exactly what you ask for. If this is wrong, fix unit definition, but we do not really know what is appropriate for this service. > Documentation=man:unattended-upgrade(8) > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug > TimeoutStopSec=900 > > [Install] > WantedBy=shutdown.target > > It appears no one who is involved in this fix understands systemd (which is > very worrying!). > > How does one would fix this unit so it is ran before the file systems get > unmounted? > Order it After=local-fs.target """ "Obviously" according to systemd mailing list if I want to things to run Before I need to use After! So I believe the confusion is with what exactly Before and After mean in systemd units. Today I will be testing various combinations with After/Before to see if I can make sense of it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
@louis-bouchard I'd love to help you, but I'm in rush on my current job. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
@kay-diam: I get what you mean. I also use KVM to debug the issue. I'm going to switch to verification-failed to this one does not get released and push the investigation further as there is another issue that side-tracked me : If Unattended-Upgrade::InstallOnShutdown is set to "true", it will still fail as the network will no longer be available when it gets to access the archive so this must also be fixed. ...Louis ** Tags removed: verification-needed ** Tags added: verification-failed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
BTW, KVM allows you to use serial console you can use it for debugging. But you have explicitly specify the order of defined "console" kernel parameters, i.e.: this will specify VGA console as default output: console=ttyS0 console=tty1 this will specify serial console as default output: console=tty1 console=ttyS0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
@louis-bouchard "Started Unattended Upgrades Shutdown." means that the unit was started. Since the patch assumes that we don't use "ExecStart" this means nothing. Upgrade procedure should be executed on "ExecStop". But it is not happening. I tried to set debug info inside the "ExecStop" instead of executing "unattended-upgrade-shutdown", but it doesn't work. I can provide you with the detailed step-by-step instructions on how to 100% reproduce the issue, but it will require KVM host, and running my scripts which will deploy VM with LVM. Basic steps: * follow https://github.com/kayrus/deploy-vm docs * git clone https://github.com/kayrus/deploy-vm * cd deploy-vm * ./deploy_vms_cluster.sh -o ubuntu -c xenial * follow https://github.com/kayrus/rescue-initramfs docs * and reinstall VM from initramfs using "deploy_os_lvm.sh" script Then your Xenial VM will stuck at reboot -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
@kay-diam I don't understand your statement. The unit clearly get started. This is from your log : [ OK ] Stopped target Local File Systems. [ OK ] Started Unattended Upgrades Shutdown. <<< Unmounting /run/user/1000... Unmounting /tmp... [ OK ] Stopped Apply Kernel Variables. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
@arges, I already noticed that upgrade procedure doesn't start at all with the new change. But actual shutdown/reboot hang has disappeared. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello Erik, or anyone else affected, Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /unattended-upgrades/0.90ubuntu0.4 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags added: verification-needed ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Before=shutdown.target reboot.target halt.target network.target local- fs.target should be Before=shutdown.target reboot.target halt.target network.target After=local-fs.target because, AIUI, at shutdown, "Before" means "After" and "After" means "Before". So unattended-upgrades.service's ExecStop'll start before local- fs.target, so before "/var" is unmounted and "/var/log" and "/var/run" are gone. [Why is "/var/run" used? This is a native Ubuntu package; are there any versions of Ubuntu where "/var/run" isn't a symlink to "/run"?] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello, I am curious to hear about the context of your tests : my Zesty system with a separated /var does show the problem prior to upgrade to unattended-upgrades-0.93.1ubuntu2 and no longer hangs when the new package is applied. There might be situations that I have not tested where the issue could still exists. Your suggestion to replace /var/run by /var is incorrect : /var/run /unattended-upgrades.lock is placed by the unattended-upgrades script; Looking in /run for a lock file that has been put in /var/run is simply inadequate. While replacing /var/run by /run across the board is a valid proposition, it requires more inspection than simply replacing its occurrences in one script. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
I have tested this and it does NOT work. I fixed the issue in couple of different ways: One way is completely remove the unit (who in right mind does install updates during reboot on production servers?). I will be actively doing that on all machines I encounter. Another way is this: sed -i 's#/var/run/#/run/#g' /usr/share/unattended-upgrades/unattended-upgrade-shutdown So the problem with that sed is that the logging part will fail. So not really a solution. Systemd has proven to be inconsistent pile of crap so far... I could not make systemd to run this unit before local-fs.target. It appears that the documentation is misleading... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Description changed: + [SRU justification] + This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system + + [Impact] + System can hang up to 10 minutes if not fixed. + + [Fix] + Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced. + + [Test Case] + In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes + + [Regression] + None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously. + + [Original description of the problem] The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade process during shutdown. This unit file is running together with all filesystem unmount services. The unattended-upgrades service checks if the lockfile for unattended-upgrade (in /var/run) exists, and if it does, there is an unattended-upgrade in progress and the service will wait until it finishes (and therefore automatically wait at shutdown). However, if /var is a separate filesystem, it will get unmounted even though /var/run is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem. The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and tries to check the lockfile again. After 10 minutes (the default timeout), it will finally exit and the system will continue shutdown. The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown where it tries to lock itself: - while True: - res = apt_pkg.get_lock(options.lock_file) - logging.debug("get_lock returned %i" % res) - # exit here if there is no lock - if res > 0: - logging.debug("lock not taken") - break - lock_was_taken = True + while True: + res = apt_pkg.get_lock(options.lock_file) + logging.debug("get_lock returned %i" % res) + # exit here if there is no lock + if res > 0: + logging.debug("lock not taken") + break + lock_was_taken = True The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error. File descriptors are just C file descriptors, so they are always positive integers. The code should check the result to be negative, not positive. I have attached a patch to reverse the logic. Additional information: 1) Description: Ubuntu 16.04.1 LTS Release: 16.04 2) unattended-upgrades: - Installed: 0.90ubuntu0.3 - Candidate: 0.90ubuntu0.3 - Version table: - *** 0.90ubuntu0.3 500 - 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages - 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages - 100 /var/lib/dpkg/status - 0.90 500 - 500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages - 500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages + Installed: 0.90ubuntu0.3 + Candidate: 0.90ubuntu0.3 + Version table: + *** 0.90ubuntu0.3 500 + 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages + 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages + 100 /var/lib/dpkg/status + 0.90 500 + 500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages + 500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages 3) Fast reboot 4) Very slow reboot (after a 10 minutes timeout) ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: Confirmed => In Progress ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Tested the fix. It works. Thanks! quick fix: sed -i "s#ExecStart=#RemainAfterExit=yes\nExecStop=#;" /lib/systemd/system/unattended-upgrades.service But now it seems that it doesn't run it on shutdown. I used the following debug service: ExecStop=/bin/bash -c 'echo -e "\nhello##\n" > /dev/ttyS0' And it doesn't print message in serial console. But when you start and stop this unit manually - it prints. [ OK ] Stopped target Graphical Interface. Stopping Accounts Service... Stopping User Manager for UID 1000... Stopping Session 1 of user ubuntu. Stopping ACPI event daemon... [ OK ] Stopped target Cloud-init target. [ OK ] Stopped Execute cloud user/final scripts. [ OK ] Stopped target Multi-User System. Stopping Deferred execution scheduler... Stopping LXD - container startup/shutdown... Stopping D-Bus System Message Bus... Stopping LSB: daemon to balance interrupts for SMP systems... Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"... Stopping LSB: MD monitoring daemon... Stopping Regular background program processing daemon... Stopping FUSE filesystem for LXC... Stopping OpenBSD Secure Shell server... Stopping LSB: Record successful boot for GRUB... [ OK ] Stopped target Timers. [ OK ] Stopped Timer to automatically refresh installed snaps. [ OK ] Stopped Daily apt activities. [ OK ] Stopped Daily Cleanup of Temporary Directories. [ OK ] Stopped target Login Prompts. Stopping Getty on tty1... Stopping Serial Getty on ttyS0... [ OK ] Stopped Apply the settings specified in cloud-config. [ OK ] Stopped target Cloud-config availability. Stopping Snappy daemon... Stopping System Logging Service... [ OK ] Stopped target System Time Synchronized. Stopping LSB: automatic crash report generation... [ OK ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch. Stopping Authenticate and Authorize Users to Run Privileged Tasks... [ OK ] Unmounted /var/lib/lxcfs. [ OK ] Stopped System Logging Service. [ OK ] Stopped Deferred execution scheduler. [ OK ] Stopped OpenBSD Secure Shell server. [ OK ] Stopped Accounts Service. [ OK ] Stopped Snappy daemon. [ OK ] Stopped ACPI event daemon. [ OK ] Stopped Authenticate and Authorize Users to Run Privileged Tasks. [ OK ] Stopped Serial Getty on ttyS0. [ OK ] Stopped Regular background program processing daemon. [ OK ] Stopped Getty on tty1. [ OK ] Stopped User Manager for UID 1000. [ OK ] Stopped Session 1 of user ubuntu. [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped FUSE filesystem for LXC. [ OK ] Stopped LXD - container startup/shutdown. [ OK ] Stopped LSB: MD monitoring daemon. [ OK ] Stopped LSB: Record successful boot for GRUB. [ OK ] Stopped LSB: daemon to balance interrupts for SMP systems. [ OK ] Stopped LSB: automatic crash report generation. [ OK ] Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand". [ OK ] Stopped User Manager for UID 1000. [ OK ] Removed slice User Slice of ubuntu. Stopping Login Service... [ OK ] Removed slice system-getty.slice. [ OK ] Removed slice system-serial\x2dgetty.slice. [ OK ] Stopped /etc/rc.local Compatibility. Stopping Permit User Sessions... [ OK ] Stopped Login Service. [ OK ] Stopped Permit User Sessions. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Login to default iSCSI targets... [ OK ] Stopped target User and Group Name Lookups. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped Forward Password Requests to Wall Directory Watch. [ OK ] Stopped Trigger resolvconf update for networkd DNS. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped ACPI Events Check. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed ACPID Listen Socket. [ OK ] Closed UUID daemon activation socket. [ OK ] Closed LXD - unix socket. [ OK ] Closed Syslog Socket. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Closed Socket activation for snappy daemon. [ OK ] Stopped target System Initialization. Stopping Network Time Synchronization... [ OK ] Stopped target Encrypted Volumes. Stopping Load/Save Random Seed... [ OK ] Stopped target Swap. [ OK ] Stopped Network Time Synchronization. [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Unmounted /home. [ OK ] Unmounted /boot. [ OK ] Stopped Login to default iSCSI targets. Stopping iSCSI initiator daemon (iscsid)... [ OK ] Stopped iSCSI initiator daemon (iscsid). [ OK ] Stopped target Network is Online. [ OK ] Stopped target Network. [
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu2 --- unattended-upgrades (0.93.1ubuntu2) zesty; urgency=medium * The systemd unit needs to be an ExecStop since it is is activated on shutdown. Otherwise, it will get scheduled after completion of the local-fs.target. In the case where /var is a separate filesystem, unattended-upgrade-shutdown will hang until timeout since /var/run is expected but no longer there (LP: #1654600) -- Louis BouchardThu, 02 Mar 2017 16:55:26 +0100 ** Changed in: unattended-upgrades (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Changed in: unattended-upgrades (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Bug #1661611 seems to be identical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
As a workaround I created a symlink to /run inside the mounted /var : mount -o bind / /mnt ln -s /run /mnt/var/ umount /mnt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Is there a good reason not to simply change the path to the lock file from /var/run to /run? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Changed in: unattended-upgrades (Debian) Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Bug watch added: Debian Bug tracker #809669 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809669 ** Also affects: unattended-upgrades (Debian) via http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809669 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: unattended-upgrades (Ubuntu Yakkety) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Ok, here is how I understand the situation. According to the systemd documentation about [Unit] Before= : "Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up. It doesn't matter if the ordering dependency is After= or Before=." To me this means that network.service & local-fs.service will be shutdown _BEFORE_ unattended-upgrades-shutdown runs, hence /var var will be unmounted when it runs. My current solution is to turn the unattended-upgrades.service ExecStart= into an ExecStop= so the unit will run as a shutdown instead of a start when the system shuts down : [Unit] Description=Unattended Upgrades Shutdown DefaultDependencies=no Before=shutdown.target reboot.target halt.target network.target local-fs.target Documentation=man:unattended-upgrade(8) [Service] Type=oneshot RemainAfterExit=yes ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug TimeoutStopSec=900 [Install] WantedBy=shutdown.target Preliminary tests seem to run fine but I want to get confirmation on such a change by someone more expert with systemd that I am. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
You're right, I thought it was the lockfile for the shutdown helper only. Somewhere else in this thread I already mentioned that a dependency on the local_fs and remote_fs targets fixes the systemd order: Before=remote-fs.target local-fs.target That makes sure the service is run before all file systems are unmounted. If you add the shutdown.target to that rule, systemd somehow thinks it should be run just before shutdown which is obviously too late. I'm not sure that is a systemd bug or a systemd feature, but either way it doesn't do what we want. Regards, Erik -- Erik Mouw > Op 26 jan. 2017 om 18:00 heeft Louis Bouchard> het volgende geschreven: > > Hello, > > This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to > acquire a lockfile (/var/run/unattended-upgrades.lock) which is only > possible if no unattended-upgrade is already running. If it can acquire > the lock, it means that NO unattended-upgrade is running, hence shutdown > may proceed. > > while True: > >res = apt_pkg.get_lock(options.lock_file) > >logging.debug("get_lock returned %i" % res) > ># exit here if there is no lock > >if res > 0: > >logging.debug("lock not taken") > >break > >lock_was_taken = True > > The problem is that apt_pkg.get_lock(options.lock_file) will also return > -1 if the path to the lock file doesn't exist, which is the case when > /var is unmounted, hence /var/run is no longer present (/var/run is a > symlink to /run). > > I'm working on identifying the proper systemd order to make sure that > /var/run is still accessible. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1654600 > > Title: > unattended-upgrade-shutdown hangs when /var is a separate filesystem > > Status in unattended-upgrades package in Ubuntu: > Confirmed > Status in unattended-upgrades source package in Xenial: > Confirmed > Status in unattended-upgrades source package in Yakkety: > New > > Bug description: > The systemd unit file unattended-upgrades.service is used to stop a running > unattended-upgrade > process during shutdown. This unit file is running together with all > filesystem > unmount services. > > The unattended-upgrades service checks if the lockfile for unattended-upgrade > (in /var/run) exists, and if it does, there is an unattended-upgrade in > progress > and the service will wait until it finishes (and therefore automatically > wait at > shutdown). > > However, if /var is a separate filesystem, it will get unmounted even though > /var/run > is a tmpfs that's still mounted on top of the /var/run directory in the /var > filesystem. > The unattended-upgrade script will fail to find lockfile, sleeps for 5 > seconds, and > tries to check the lockfile again. After 10 minutes (the default timeout), > it will finally > exit and the system will continue shutdown. > > The problem is the error handling in > /usr/share/unattended-upgrades/unattended-upgrade-shutdown > where it tries to lock itself: > > while True: > res = apt_pkg.get_lock(options.lock_file) > logging.debug("get_lock returned %i" % res) > # exit here if there is no lock > if res > 0: > logging.debug("lock not taken") > break > lock_was_taken = True > > The function apt_pkg.get_lock() either returns a file descriptor, or -1 on > an error. > File descriptors are just C file descriptors, so they are always positive > integers. > The code should check the result to be negative, not positive. I have > attached a patch > to reverse the logic. > > Additional information: > > 1) > Description:Ubuntu 16.04.1 LTS > Release:16.04 > > 2) > unattended-upgrades: >Installed: 0.90ubuntu0.3 >Candidate: 0.90ubuntu0.3 >Version table: > *** 0.90ubuntu0.3 500 > 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 > Packages > 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 > Packages > 100 /var/lib/dpkg/status > 0.90 500 > 500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages > 500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages > 3) > Fast reboot > 4) > Very slow reboot
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello, This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to acquire a lockfile (/var/run/unattended-upgrades.lock) which is only possible if no unattended-upgrade is already running. If it can acquire the lock, it means that NO unattended-upgrade is running, hence shutdown may proceed. while True: res = apt_pkg.get_lock(options.lock_file) logging.debug("get_lock returned %i" % res) # exit here if there is no lock if res > 0: logging.debug("lock not taken") break lock_was_taken = True The problem is that apt_pkg.get_lock(options.lock_file) will also return -1 if the path to the lock file doesn't exist, which is the case when /var is unmounted, hence /var/run is no longer present (/var/run is a symlink to /run). I'm working on identifying the proper systemd order to make sure that /var/run is still accessible. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Also affects: unattended-upgrades (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: unattended-upgrades (Ubuntu Yakkety) Importance: Undecided Status: New ** Changed in: unattended-upgrades (Ubuntu) Assignee: (unassigned) => Louis Bouchard (louis-bouchard) ** Changed in: unattended-upgrades (Ubuntu Xenial) Assignee: (unassigned) => Louis Bouchard (louis-bouchard) ** Changed in: unattended-upgrades (Ubuntu Yakkety) Assignee: (unassigned) => Louis Bouchard (louis-bouchard) ** Changed in: unattended-upgrades (Ubuntu Xenial) Importance: Undecided => High ** Changed in: unattended-upgrades (Ubuntu Yakkety) Importance: Undecided => High ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
I get this sometimes on shutdown but not always. Erik, until the fix reaches users, should I: systemctl disable unattended-upgrades.service ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Changed in: unattended-upgrades (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
FWIW, the init script /etc/init.d/unattended-upgrades should be removed from systemd systems (Xenial, Yaketty). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
I have narrowed down. This bug was introduced by the fix in #1618900. That tries to fix the problem of the the unattended-upgrades.service by letting it depend on network.target and local-fs.target. However, that doesn't change any of the dependencies, it only makes the race window larger. After some experiments, the correct dependencies in the unit file appear to be: Before=remote-fs.target local-fs.target Which make it behave just like the SysV init script. This change together with the patch I attached to this bug report together should fix the issue and also fix #1618900 . Regards, Erik -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
I have same problem. I need to kill unattended-upgrades service manually to shutdown machine. This problem comes in 2017 year. System is Ubuntu 16.04 x86_64. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: unattended-upgrades (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
I have also tried to let the systemd unit file for this service WantedBy the umount.target. According to the systemd documentation that should make it run before the filesystems are unmounted, but apparently that doesn't work. systemctl list-dependencies still shows that the unattended-upgrade.service reverse depends on the shutdown.target. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs