[Touch-packages] [Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello!
I have changed Before to After and it is working
Here is what the unit looks like now:
[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
After=shutdown.target reboot.target halt.target network.target local-fs.target
Documentation=man:unattended-upgrade(8)
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStopSec=900
[Install]
WantedBy=shutdown.target
So the proper fix would be changing the Service to ExecStop= and Unit to After=
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to unattended-upgrades in
Ubuntu.
https://bugs.launchpad.net/bugs/1654600
Title:
unattended-upgrade-shutdown hangs when /var is a separate filesystem
Status in unattended-upgrades package in Ubuntu:
Fix Released
Status in unattended-upgrades source package in Xenial:
Fix Committed
Status in unattended-upgrades source package in Yakkety:
Fix Committed
Status in unattended-upgrades package in Debian:
New
Bug description:
[SRU justification]
This fix is needed to make sure that the system does not hang on shutdown
when /var is a speparate file system
[Impact]
System can hang up to 10 minutes if not fixed.
[Fix]
Change the systemd unit's ExecStart to an ExecStop so the unit is correctly
sequenced.
[Test Case]
In a VM with /var separated from the / file system, shutdown the VM. It will
hang for 10 minutes
[Regression]
None to be expected. A ExecStop unit will be sequenced before the Before=
units which is earlier than previously.
[Original description of the problem]
The systemd unit file unattended-upgrades.service is used to stop a running
unattended-upgrade
process during shutdown. This unit file is running together with all
filesystem
unmount services.
The unattended-upgrades service checks if the lockfile for unattended-upgrade
(in /var/run) exists, and if it does, there is an unattended-upgrade in
progress
and the service will wait until it finishes (and therefore automatically wait
at
shutdown).
However, if /var is a separate filesystem, it will get unmounted even though
/var/run
is a tmpfs that's still mounted on top of the /var/run directory in the /var
filesystem.
The unattended-upgrade script will fail to find lockfile, sleeps for 5
seconds, and
tries to check the lockfile again. After 10 minutes (the default timeout), it
will finally
exit and the system will continue shutdown.
The problem is the error handling in
/usr/share/unattended-upgrades/unattended-upgrade-shutdown
where it tries to lock itself:
while True:
res = apt_pkg.get_lock(options.lock_file)
logging.debug("get_lock returned %i" % res)
# exit here if there is no lock
if res > 0:
logging.debug("lock not taken")
break
lock_was_taken = True
The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an
error.
File descriptors are just C file descriptors, so they are always positive
integers.
The code should check the result to be negative, not positive. I have
attached a patch
to reverse the logic.
Additional information:
1)
Description: Ubuntu 16.04.1 LTS
Release: 16.04
2)
unattended-upgrades:
Installed: 0.90ubuntu0.3
Candidate: 0.90ubuntu0.3
Version table:
*** 0.90ubuntu0.3 500
500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64
Packages
500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386
Packages
100 /var/lib/dpkg/status
0.90 500
500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
3)
Fast reboot
4)
Very slow reboot (after a 10 minutes timeout)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
Hello,
Yes, I am fully aware that my /var/run->/run replacement in that script
is a hack.
To be clear I am not testing in Zesty but Ubuntu 16.04.2 LTS Xenial.
The setup is following:
/dev/sda1 on / type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc on /var type xfs (rw,relatime,attr2,inode64,noquota)
Every single machine we (my and my colleague) tested with the updated
unit still hangs.
As var as I can tell they never reboot.
I have posted a question regarding this issue to systemd mailing list,
and one of the responses was the following:
"""
> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> Before=shutdown.target reboot.target halt.target network.target
> local-fs.target
Well, you order if before local-fs.target, which means it is stopped
after local-fs.target. You get exactly what you ask for. If this is
wrong, fix unit definition, but we do not really know what is
appropriate for this service.
> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
> TimeoutStopSec=900
>
> [Install]
> WantedBy=shutdown.target
>
> It appears no one who is involved in this fix understands systemd (which is
> very worrying!).
>
> How does one would fix this unit so it is ran before the file systems get
> unmounted?
>
Order it
After=local-fs.target
"""
"Obviously" according to systemd mailing list if I want to things to run Before
I need to use After!
So I believe the confusion is with what exactly Before and After mean in
systemd units.
Today I will be testing various combinations with After/Before to see if
I can make sense of it.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to unattended-upgrades in
Ubuntu.
https://bugs.launchpad.net/bugs/1654600
Title:
unattended-upgrade-shutdown hangs when /var is a separate filesystem
Status in unattended-upgrades package in Ubuntu:
Fix Released
Status in unattended-upgrades source package in Xenial:
Fix Committed
Status in unattended-upgrades source package in Yakkety:
Fix Committed
Status in unattended-upgrades package in Debian:
New
Bug description:
[SRU justification]
This fix is needed to make sure that the system does not hang on shutdown
when /var is a speparate file system
[Impact]
System can hang up to 10 minutes if not fixed.
[Fix]
Change the systemd unit's ExecStart to an ExecStop so the unit is correctly
sequenced.
[Test Case]
In a VM with /var separated from the / file system, shutdown the VM. It will
hang for 10 minutes
[Regression]
None to be expected. A ExecStop unit will be sequenced before the Before=
units which is earlier than previously.
[Original description of the problem]
The systemd unit file unattended-upgrades.service is used to stop a running
unattended-upgrade
process during shutdown. This unit file is running together with all
filesystem
unmount services.
The unattended-upgrades service checks if the lockfile for unattended-upgrade
(in /var/run) exists, and if it does, there is an unattended-upgrade in
progress
and the service will wait until it finishes (and therefore automatically wait
at
shutdown).
However, if /var is a separate filesystem, it will get unmounted even though
/var/run
is a tmpfs that's still mounted on top of the /var/run directory in the /var
filesystem.
The unattended-upgrade script will fail to find lockfile, sleeps for 5
seconds, and
tries to check the lockfile again. After 10 minutes (the default timeout), it
will finally
exit and the system will continue shutdown.
The problem is the error handling in
/usr/share/unattended-upgrades/unattended-upgrade-shutdown
where it tries to lock itself:
while True:
res = apt_pkg.get_lock(options.lock_file)
logging.debug("get_lock returned %i" % res)
# exit here if there is no lock
if res > 0:
logging.debug("lock not taken")
break
lock_was_taken = True
The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an
error.
File descriptors are just C file descriptors, so they are always positive
integers.
The code should check the result to be negative, not positive. I have
attached a patch
to reverse the logic.
Additional information:
1)
Description: Ubuntu 16.04.1 LTS
Release: 16.04
2)
unattended-upgrades:
Installed: 0.90ubuntu0.3
Candidate: 0.90ubuntu0.3
Version table:
*** 0.90ubuntu0.3 500
500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64
Packages
500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386
Packages
100 /var/lib/dpkg/status
0.90 500
500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
500 http://nl.archive.ubuntu.com/ubuntu xenial
[Touch-packages] [Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem
I have tested this and it does NOT work.
I fixed the issue in couple of different ways:
One way is completely remove the unit (who in right mind does install
updates during reboot on production servers?). I will be actively doing
that on all machines I encounter.
Another way is this:
sed -i 's#/var/run/#/run/#g'
/usr/share/unattended-upgrades/unattended-upgrade-shutdown
So the problem with that sed is that the logging part will fail. So not really
a solution.
Systemd has proven to be inconsistent pile of crap so far...
I could not make systemd to run this unit before local-fs.target. It
appears that the documentation is misleading...
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to unattended-upgrades in
Ubuntu.
https://bugs.launchpad.net/bugs/1654600
Title:
unattended-upgrade-shutdown hangs when /var is a separate filesystem
Status in unattended-upgrades package in Ubuntu:
Fix Released
Status in unattended-upgrades source package in Xenial:
In Progress
Status in unattended-upgrades source package in Yakkety:
In Progress
Status in unattended-upgrades package in Debian:
New
Bug description:
[SRU justification]
This fix is needed to make sure that the system does not hang on shutdown
when /var is a speparate file system
[Impact]
System can hang up to 10 minutes if not fixed.
[Fix]
Change the systemd unit's ExecStart to an ExecStop so the unit is correctly
sequenced.
[Test Case]
In a VM with /var separated from the / file system, shutdown the VM. It will
hang for 10 minutes
[Regression]
None to be expected. A ExecStop unit will be sequenced before the Before=
units which is earlier than previously.
[Original description of the problem]
The systemd unit file unattended-upgrades.service is used to stop a running
unattended-upgrade
process during shutdown. This unit file is running together with all
filesystem
unmount services.
The unattended-upgrades service checks if the lockfile for unattended-upgrade
(in /var/run) exists, and if it does, there is an unattended-upgrade in
progress
and the service will wait until it finishes (and therefore automatically wait
at
shutdown).
However, if /var is a separate filesystem, it will get unmounted even though
/var/run
is a tmpfs that's still mounted on top of the /var/run directory in the /var
filesystem.
The unattended-upgrade script will fail to find lockfile, sleeps for 5
seconds, and
tries to check the lockfile again. After 10 minutes (the default timeout), it
will finally
exit and the system will continue shutdown.
The problem is the error handling in
/usr/share/unattended-upgrades/unattended-upgrade-shutdown
where it tries to lock itself:
while True:
res = apt_pkg.get_lock(options.lock_file)
logging.debug("get_lock returned %i" % res)
# exit here if there is no lock
if res > 0:
logging.debug("lock not taken")
break
lock_was_taken = True
The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an
error.
File descriptors are just C file descriptors, so they are always positive
integers.
The code should check the result to be negative, not positive. I have
attached a patch
to reverse the logic.
Additional information:
1)
Description: Ubuntu 16.04.1 LTS
Release: 16.04
2)
unattended-upgrades:
Installed: 0.90ubuntu0.3
Candidate: 0.90ubuntu0.3
Version table:
*** 0.90ubuntu0.3 500
500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64
Packages
500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386
Packages
100 /var/lib/dpkg/status
0.90 500
500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
3)
Fast reboot
4)
Very slow reboot (after a 10 minutes timeout)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1661611] Re: apt/unattended-upgrades stalls shutdown
I can confirm this bug also happens machines without LVM volumes. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unattended-upgrades in Ubuntu. https://bugs.launchpad.net/bugs/1661611 Title: apt/unattended-upgrades stalls shutdown Status in unattended-upgrades package in Ubuntu: Confirmed Bug description: When unattended-upgrades is installed, 9 out of 10 shutdowns/reboots hang while "starting unattended upgrades shutdown". This hang stalls the shutdown process for 5-10 mins. If I disable unnattended-upgrades via the /etc/apt/apt.conf.d/20auto- upgrades and/or 50unattended-upgrades, the problems occurs. If I terminate the service before shutdown/reboot (sudo service unattended-upgrades stop) the problem still occurs. If I remove the package (sudo apt remove unattended-upgrades) the problem no longer occurs. This occurs on a freshly installed version of Ubuntu Server 16.04.1 (both unattended-upgrades installed via install GUI or manual install of unattended-upgrades) Both Kern.log & syslog do not show the shutdown process (I believe because the filesystems have already unmounted) Original report: http://askubuntu.com/questions/878630/apt-unattended- upgrades-stalls-shutdown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1661611] Re: apt/unattended-upgrades stalls shutdown
Regarding presence of "Unattended-Upgrade::InstallOnShutdown" in /etc/apt/apt.conf.d/20auto-upgrades and /etc/apt/apt.conf.d /50unattended-upgrades: It does not exist in /etc/apt/apt.conf.d/20auto-upgrades and it is commented out in /etc/apt/apt.conf.d/50unattended-upgrades , Eg: //Unattended-Upgrade::InstallOnShutdown "true"; ** Changed in: unattended-upgrades (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unattended-upgrades in Ubuntu. https://bugs.launchpad.net/bugs/1661611 Title: apt/unattended-upgrades stalls shutdown Status in unattended-upgrades package in Ubuntu: Confirmed Bug description: When unattended-upgrades is installed, 9 out of 10 shutdowns/reboots hang while "starting unattended upgrades shutdown". This hang stalls the shutdown process for 5-10 mins. If I disable unnattended-upgrades via the /etc/apt/apt.conf.d/20auto- upgrades and/or 50unattended-upgrades, the problems occurs. If I terminate the service before shutdown/reboot (sudo service unattended-upgrades stop) the problem still occurs. If I remove the package (sudo apt remove unattended-upgrades) the problem no longer occurs. This occurs on a freshly installed version of Ubuntu Server 16.04.1 (both unattended-upgrades installed via install GUI or manual install of unattended-upgrades) Both Kern.log & syslog do not show the shutdown process (I believe because the filesystems have already unmounted) Original report: http://askubuntu.com/questions/878630/apt-unattended- upgrades-stalls-shutdown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
