Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hello, The debdiff that I provided contains two typos in a comment : > 64 +# Explicitely enable and start the service.i Debian Bug #797108 for It should read Explicitly and "start the service. Debian" Just let me know if you want an updated debdiff or if you will be fixing it up yourself. Kind regards, ...Louis -- Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu developer Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61 signature.asc Description: OpenPGP digital signature
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hello, Here is the patch for this issue that I uploaded to Ubuntu. It is slightly adapted to take into account the possibility of being install on a non-systemd installation. I tried to make the changelog entry as explicit as possible but here are some details. As outlined previously, the systemd unit is changed to be an ExecStart instead of an ExecStop. RequiresMountsFor= are added for /var/log, /var/run, /var/lib & /boot. Disabling DefaultDependencies is removed so, as outlined previously, local-fs.target might be superfluous. RemainAfterExit is set to Yes so the unit is seen as started. WantedBy is set to multi-user.target This introduces a problem since Bug #797108[1] is causing the unit not to be enabled upon upgrade. The following is done to work around this issue: 1) On systemd enabled system, postinst forces the unattended-upgrades service to be disabled before deb-systemd-helper is executed so the previous shutdown.target symlink does not remain. 2) At the end of the postinst script, after the deb-systemd-helper has been run, manually enable and start the service. This will leave the service correctly configured, as if the deb-systemd-helper had no bug. 3) systemctl enable requires the SysV init script to have a Default-Start statement in the header otherwise it fails. Add the header in the script. 4) Remove the override_dh_installinit since it uses the 'stop' option which is no longer available hence switching to 'default' which is the normal installinit behavior. The postinst script also needs to cleanup the faulty stop symlink created previously otherwise the systemclt enable fails. 5) Add DEP8 tests to verify that the unit is correctly started and that InstallOnShutdown works as expected. I have only tested the upgrade from 0.93.1 to 0.93.2 on Debian/Sid but I have done extensive testing on Ubuntu which includes : * do-release-upgrade from Trusty(upstart) to Xenial(systemd) * upgrade on Xenial, Yakkety, Zesty & Artful Please let me know if I can help with any more testing. Kind regards, ...Louis [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797108 -- Louis Bouchard Software engineer, Ubuntu Developer / Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61 @@ -1,3 +1,34 @@ +unattended-upgrades (0.93.2) unstable; urgency=medium + + [ Louis Bouchard ] + * Fix the unattended-upgrades.service unit not correctly working: +- d/rules : Remove the override_dh_installinit. The stop option is no longer + available so the command falls back to default. This is the normal + behavior so the override is not required +- d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the + unattended-upgrades.service unit cannot be enabled on boot +- d/postinst : Cleanup the stop symlinks created by the wrong + override_dh_installinit. Without that, the systemd unit cannot be + enabled correctly. + Force disable the service before deb-systemd-helper runs so the old + symlink is not left dangling (workaround for Debian Bug #797108). + Force enable and start of the systemd unit to work around Debian Bug #797108 + which fails to enable systemd units correctly when WantedBy= statement + is changed which is the case here. +- d/unattended-upgrades.service : Fix the service so it runs correctly on + shutdown : +Remove DefaultDependencies=no : Breaks normal shutdown dependencies +Set After= to network.target and local-fs.target. Since our service is +now ExecStop, it will run before network and local-fs become unavailable. +Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if +/var is a separate file system. Set WantedBy= to multi-user.target +- Add DEP8 tests to verify the following : + Verify that the unattended-upgrades.service unit is enabled and started. + Verify that InstallOnShutdown works when configured. +(Closes: #809669) + + -- Louis BouchardMon, 24 Apr 2017 14:42:01 +0200 + unattended-upgrades (0.93.1) unstable; urgency=medium [ Brian Murray ] diff -Nru unattended-upgrades-0.93.1/debian/postinst unattended-upgrades-0.93.2/debian/postinst --- unattended-upgrades-0.93.1/debian/postinst 2016-12-11 11:31:26.0 +0100 +++ unattended-upgrades-0.93.2/debian/postinst 2017-04-24 14:42:01.0 +0200 @@ -61,6 +61,20 @@ && [ -f /etc/rc6.d/S[0-9][0-9]unattended-upgrades ] ; then update-rc.d -f unattended-upgrades remove fi +# Recover from broken dh_installinit override in versions < 0.93.2 +if dpkg --compare-versions "$2" lt "0.93.2"; then +if [ -f /etc/rc0.d/K[0-9][0-9]unattended-upgrades ] \ +&& [ -f /etc/rc6.d/K[0-9][0-9]unattended-upgrades ] ; then + update-rc.d -f unattended-upgrades remove + fi + # If running systemd, explicitely disable the service otherwise +
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
On Thu, 20 Apr 2017 14:05:39 +0200 Louis Bouchardwrote: > Hello, > > Le 20/04/2017 à 12:27, Simon McVittie a écrit : > > [...] > > First of all, I am sorry for not keeping up with my work in both bug reports. > Recent events at my employer have made things a bit more difficult, including > the recent release of Ubuntu Zesty. > Hi, Congrats on the Zesty release, btw. :) > [...] > > I am also preparing a .debdiff of a proposed debian fix that I will create and > test today and tomorrow. In the meantime, here is the final unit that will be > shiped : > > Any news from the test and about the debdiff? :) Thanks, ~Niels
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hi Louis, thanks for your work on this. On 2017-04-20.14:05, Louis Bouchard wrote: > > [Unit] > > > > Description=Unattended Upgrades Shutdown > > > > After=network.target local-fs.target > > > > RequiresMountsFor=/var/log /var/run /var/lib /boot > > > > Documentation=man:unattended-upgrade(8) > > > > > > > > [Service] > > > > Type=oneshot > > > > RemainAfterExit=yes > > > > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > > > > TimeoutStopSec=900 > > > > > > > > [Install] > > > > WantedBy=multi-user.target > > > > DefaultDependencies=no Needs to be removed as switching from shutdown.target > to > multi-user.target requires to have the DefaultDependencies I think the After=local-fs.target is unnecessary, as DefaultDependencies pulls in After=sysinit.target, which itself is After=local-fs.target. -- Regards, Scott. signature.asc Description: PGP signature
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hello, Le 20/04/2017 à 12:27, Simon McVittie a écrit : > On Tue, 11 Apr 2017 at 17:25:00 +, Niels Thykier wrote: >> On Thu, 2 Mar 2017 17:59:02 +0100 Louis Bouchard >>wrote: >>> It now tests correctly and I am preparing an upload to our development >>> release. >> >> Any news on this? I cannot see an upload to unstable yet, did something >> hold you up? > > The "us" referred to in "our development release" appears to have been > Ubuntu, not Debian. > > However, the systemd unit proposed by Scott Leggett is not actually the > same as the one now shipped in Ubuntu zesty. Is this deliberate? Here > is a diff: > > --- ubuntu/debian/unattended-upgrades.service > +++ scott/debian/unattended-upgrades.service > @@ -1,7 +1,6 @@ > [Unit] > Description=Unattended Upgrades Shutdown > -DefaultDependencies=no > -Before=shutdown.target reboot.target halt.target network.target > local-fs.target > +After=network.target > Documentation=man:unattended-upgrade(8) > > [Service] > @@ -11,4 +10,4 @@ > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > TimeoutStopSec=900 > > [Install] > -WantedBy=shutdown.target > +WantedBy=multi-user.target > > I'm concerned that the version now shipped in Ubuntu might in fact be shut > down *after* the network is taken down, because Before/After dependencies > are about the order of startup: shutdown happens in the reverse order. > So if unattended-upgrades.service has After=network.target, as in Scott's > proposed unit, it will be started (which now does nothing) after > network.target is started, but stopped (which is where the real work > happens) before network.target is stopped. That seems like the right > thing, and the version in Ubuntu zesty seems like only a partial fix. > > S > First of all, I am sorry for not keeping up with my work in both bug reports. Recent events at my employer have made things a bit more difficult, including the recent release of Ubuntu Zesty. The fix in Zesty is indeed incomplete, but final freeze prevented me for including the definitive fix in Zesty. This should happen shortly. I am also preparing a .debdiff of a proposed debian fix that I will create and test today and tomorrow. In the meantime, here is the final unit that will be shiped : > [Unit] > > Description=Unattended Upgrades Shutdown > > After=network.target local-fs.target > > RequiresMountsFor=/var/log /var/run /var/lib /boot > > Documentation=man:unattended-upgrade(8) > > > > [Service] > > Type=oneshot > > RemainAfterExit=yes > > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > > TimeoutStopSec=900 > > > > [Install] > > WantedBy=multi-user.target > DefaultDependencies=no Needs to be removed as switching from shutdown.target to multi-user.target requires to have the DefaultDependencies RequireMountsFor= provides adequate dependencies when /var is separate. In changing WantedBy= I ran into Debian Bug #797108 which causes the service to not be properly enabled on upgrade. Working around that took longer than expected. So hopefully I will provide a final proposition for this by End of week. Kind regards, ...Louis -- Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu developer Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
On Tue, 11 Apr 2017 at 17:25:00 +, Niels Thykier wrote: > On Thu, 2 Mar 2017 17:59:02 +0100 Louis Bouchard >wrote: > > It now tests correctly and I am preparing an upload to our development > > release. > > Any news on this? I cannot see an upload to unstable yet, did something > hold you up? The "us" referred to in "our development release" appears to have been Ubuntu, not Debian. However, the systemd unit proposed by Scott Leggett is not actually the same as the one now shipped in Ubuntu zesty. Is this deliberate? Here is a diff: --- ubuntu/debian/unattended-upgrades.service +++ scott/debian/unattended-upgrades.service @@ -1,7 +1,6 @@ [Unit] Description=Unattended Upgrades Shutdown -DefaultDependencies=no -Before=shutdown.target reboot.target halt.target network.target local-fs.target +After=network.target Documentation=man:unattended-upgrade(8) [Service] @@ -11,4 +10,4 @@ ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown TimeoutStopSec=900 [Install] -WantedBy=shutdown.target +WantedBy=multi-user.target I'm concerned that the version now shipped in Ubuntu might in fact be shut down *after* the network is taken down, because Before/After dependencies are about the order of startup: shutdown happens in the reverse order. So if unattended-upgrades.service has After=network.target, as in Scott's proposed unit, it will be started (which now does nothing) after network.target is started, but stopped (which is where the real work happens) before network.target is stopped. That seems like the right thing, and the version in Ubuntu zesty seems like only a partial fix. S
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
On Thu, 2 Mar 2017 17:59:02 +0100 Louis Bouchardwrote: > Hello, > > [...] > > The unit was correctly installed but another issue made it show as an > incorrect > behavior. > > It now tests correctly and I am preparing an upload to our development > release. > > Thanks > > Kind regards, > > ...Louis > > -- > Louis Bouchard > Software engineer, Cloud & Sustaining eng. > Canonical Ltd > Ubuntu developer Debian Maintainer > GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61 > Hi Louis, Any news on this? I cannot see an upload to unstable yet, did something hold you up? Thanks, ~Niels
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
On Fri, 10 Feb 2017 at 16:31:25 +0100, Louis Bouchard wrote: > The proposed systemd unit change would break : > > Unattended-Upgrade::InstallOnShutdown "true"; > > as the network is no longer available to fetch the archive. You might be interested in unattended-upgrades patches that I wrote for SteamOS a few years ago: https://bugs.debian.org/741356 The SteamOS developers were concerned that an expectation that the network is up during shutdown was unrealistic, so I changed the behaviour to prepare and download an entire proposed upgrade and stash it in /var before notifying the user that they should reboot, invalidating the proposed upgrade if the apt state subsequently changes. A modified version of my patches seems to be still in production use. Unfortunately it seems the unattended-upgrades maintainer hasn't been able to review or integrate these. S
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hello, Le 21/02/2017 à 12:31, Scott Leggett a écrit : > Hi Louis, > > On Wed, 15 Feb 2017 14:34:58 +0100 Louis Bouchard >wrote: >> Hello, >> >> I may be wrong, but this clearly shows that the Unattended Upgrades Shutdown >> unit starts once the target Network is being brought down : > > I don't think the replacement unit I proposed was installed correctly on > your system. Could you double check? > >> Pinging google for 4 seconds is not sufficient, the Unattended upgrade >> shutdown >> can run for saveral minutes before completing. > > This is the express purpose of network.target. Here's the relevant > snippet from `man systemd.special`: > > network.target > This unit is supposed to indicate when network functionality is > available, but it is only very weakly defined what that is supposed > to mean, with one exception: at shutdown, a unit that is ordered > after network.target will be stopped before the network — to > whatever level it might be set up then — is shut down. It is hence > useful when writing service files that require network access on > shutdown, which should order themselves after this target, but not > pull it in. Also see Running Services After the Network is up[1] for > more information. Also see network-online.target described above. > The unit was correctly installed but another issue made it show as an incorrect behavior. It now tests correctly and I am preparing an upload to our development release. Thanks Kind regards, ...Louis -- Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu developer Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61 signature.asc Description: OpenPGP digital signature
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hi Louis, On Wed, 15 Feb 2017 14:34:58 +0100 Louis Bouchardwrote: > Hello, > > I may be wrong, but this clearly shows that the Unattended Upgrades Shutdown > unit starts once the target Network is being brought down : I don't think the replacement unit I proposed was installed correctly on your system. Could you double check? > Pinging google for 4 seconds is not sufficient, the Unattended upgrade > shutdown > can run for saveral minutes before completing. This is the express purpose of network.target. Here's the relevant snippet from `man systemd.special`: network.target This unit is supposed to indicate when network functionality is available, but it is only very weakly defined what that is supposed to mean, with one exception: at shutdown, a unit that is ordered after network.target will be stopped before the network — to whatever level it might be set up then — is shut down. It is hence useful when writing service files that require network access on shutdown, which should order themselves after this target, but not pull it in. Also see Running Services After the Network is up[1] for more information. Also see network-online.target described above. -- Regards, Scott. signature.asc Description: PGP signature
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hello, I may be wrong, but this clearly shows that the Unattended Upgrades Shutdown unit starts once the target Network is being brought down : > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Stopped target Network is > Online. > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Stopped target Network. > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Started Unattended Upgrades > Shutdown. > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Stopping ifup for ens3... > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Stopping Raise network > interfaces... > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Reloading. > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Reloading. > Feb 15 12:35:52 ZunattendedUpgrade dhclient[2799]: Killed old client process > Feb 15 12:35:52 ZunattendedUpgrade ifdown[2765]: Killed old client process > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Reloading. > Feb 15 12:35:52 ZunattendedUpgrade systemd[1]: Stopped Raise network > interfaces. > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: Internet Systems > Consortium DHCP Client 4.3.3 > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: Internet Systems Consortium > DHCP Client 4.3.3 > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: Copyright 2004-2015 Internet > Systems Consortium. > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: All rights reserved. > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: For info, please visit > https://www.isc.org/software/dhcp/ > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: Copyright 2004-2015 > Internet Systems Consortium. > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: All rights reserved. > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: For info, please visit > https://www.isc.org/software/dhcp/ > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: Listening on > LPF/ens3/52:54:00:69:a4:c4 > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: Listening on > LPF/ens3/52:54:00:69:a4:c4 > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: Sending on > LPF/ens3/52:54:00:69:a4:c4 > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: Sending on Socket/fallback > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: Sending on > LPF/ens3/52:54:00:69:a4:c4 > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: Sending on > Socket/fallback > Feb 15 12:35:53 ZunattendedUpgrade dhclient[2799]: DHCPRELEASE on ens3 to > 192.168.1.1 port 67 (xid=0x3e07e0e5) > Feb 15 12:35:53 ZunattendedUpgrade ifdown[2765]: DHCPRELEASE on ens3 to > 192.168.1.1 port 67 (xid=0x3e07e0e5) > Feb 15 12:35:53 ZunattendedUpgrade systemd[1]: Stopped ifup for ens3. Pinging google for 4 seconds is not sufficient, the Unattended upgrade shutdown can run for saveral minutes before completing. HTH, Kind regards, ...Louis -- Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu developer Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
On Fri, 10 Feb 2017 16:31:25 +0100 Louis Bouchardwrote: > Hi, >· > The proposed systemd unit change would break : >· > Unattended-Upgrade::InstallOnShutdown "true"; >· > as the network is no longer available to fetch the archive. >· No, it wouldn't. > As outlined in the systemd documentation : >· > "Given two units with any ordering dependency between them, if one unit is > shut > down and the other is started up, the shutdown is ordered before the start-up. > It doesn't matter if the ordering dependency is After= or Before=." >· Both units in this case are being shut down, so that paragraph doesn't apply. You can test that the network is available for yourself by adding anonther ExecStop line below the first: ExecStop=/bin/ping -c 4 google.com --· Regards, Scott. signature.asc Description: PGP signature
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hi, The proposed systemd unit change would break : Unattended-Upgrade::InstallOnShutdown "true"; as the network is no longer available to fetch the archive. As outlined in the systemd documentation : "Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up. It doesn't matter if the ordering dependency is After= or Before=." In that context, the network needs to remain available until completion of unattended-upgrade.service. I'm still looking for a combination that will work in all cases. Kind regards, ...Louis -- Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu developer Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Le jeudi 9 février 2017, 17 h 14 min 44 s CET Louis Bouchard a écrit : > The unattended-upgrade-shutdown script uses a lock in /var/run to check if an > upgrade job is running. After /var is unmounted, /var/run is no longer present Hi, I thought that by now all reference to /var/run could be replaced by simply "/run". This would solve this problem. Alexandre
Bug#809669: unattended-upgrades: files got created under /var/ mountpoint
Hello, For info, this also has the potential effect of blocking shutdown (see Ubuntu's LP: #1654600 [1]) for details. The unattended-upgrade-shutdown script uses a lock in /var/run to check if an upgrade job is running. After /var is unmounted, /var/run is no longer present and apt_pkg.get_lock() return -1 which normally means that the lock cannot be taken. In this case, -1 is caused by the fact that the /var/run path no longer exists. The lock appears to be present, so unattended-upgrade-shutdown waits for it to go away. The delay to timeout is 10 minutes so the shutdown may block for 10 minutes. Thought it was worth mentionning. Kind regards, ...Louis [1] https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600 -- Louis Bouchard Software engineer, Ubuntu Developer / Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61