From cloud-init point of view the solution now implemented make sense:
to run it before the apt-daily-upgrade. However, I wanted to add that
there are other use cases as well such as SSM documents being executed
on instances. These can be executed in batch at any time and may also
require installation of packages and thus interfere with these
unattended upgrades.
The execution of documents is not linked directly to cloud-init and may
be ran after the instances has been booted, so this falls in the other
category of having some kind queuing system or at least a centralized
way to obtain a lock to be able to use apt. At the moment there are
dozens of different possibilities how to get a mutex to be able to
execute apt, but somehow we couldn't find a bullet proof way that works
*every time*.
So maybe this does not really fit into this ticket, but to address that
this is only a partial fix to a bigger problem.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1693361
Title:
cloud-init sometimes fails on dpkg lock due to concurrent apt-
daily.service execution
Status in APT:
Fix Released
Status in cloud-init:
Fix Released
Status in apt package in Ubuntu:
Invalid
Status in cloud-init package in Ubuntu:
Fix Released
Status in cloud-init source package in Xenial:
Fix Released
Status in cloud-init source package in Yakkety:
Won't Fix
Status in cloud-init source package in Zesty:
Fix Released
Status in cloud-init source package in Artful:
Fix Released
Bug description:
=== Begin SRU Template ===
[Impact]
A cloud-config that contains packages to install (see below) or
'package_upgrade' will run 'apt-get update'. That can sometimes fail as a
result of contention with the apt-daily.service that updates that information.
Cloud-config showing the problem is just like:
$ cat my.yaml
#cloud-config
packages: ['hello']
[Test Case]
lxc-proposed-snapshot is
https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bin/lxc-proposed-snapshot
It publishes an image to lxd with proposed enabled and cloud-init upgraded.
a.) launch an instance with proposed version of cloud-init and some user-data.
This is platform independent. The test case demonstrates lxd.
$ printf "%s\n%s\n%s\n" "#cloud-config" "packages: ['hello']" \
"package_upgrade: true" > config.yaml
$ release=xenial
$ ref=proposed-$release
$ ./lxc-proposed-snapshot --proposed --publish $release $ref;
b.) start the instance
$ name=$release-1693361
$ lxc launch my-xenial "--config=user.user-data=$(cat config.yaml)
$ sleep 1
$ lxc exec $name -- tail -f /var/log/cloud-init.log
/var/log/cloud-init-output.log
# watch this boot.
c.) Look for evidence of systemd failure
journalctl -o short-precise | grep -i break
journalctl -o short-precise | grep -i order
[Regression Potential]
Regression chance here is low. Its possible that ordering loops
could occur. When that does happen, journalctl will mention it.
Unfortunately
in such cases systemd somewhat randomly picks a service to kil so behavior
is somewhat undefined.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=11121fe4
=== End SRU Template ===
apt-daily is now a systemd service rather than being invoked by
cron.daily. If one builds a custom AMI it is possible that the apt-
daily.timer will fire during boot. This can fire at the same time
cloud-init is running and if cloud-init loses the race the invocation
of apt (e.g. use of "packages:" in the config) will fail.
There is a lot of discussion online about this change to apt-daily
(e.g. unattended upgrades happening during business hours, delaying
boot, etc.) and discussion of potential systemd changes regarding
timers firing during boot (c.f.
https://github.com/systemd/systemd/issues/5659).
While it would be better to solve this in apt itself, I suggest that
cloud-init be defensive when calling apt and implement some retry
mechanism.
Various instances of people running into this issue:
https://github.com/chef/bento/issues/609
https://clusterhq.atlassian.net/browse/FLOC-4486
https://github.com/boxcutter/ubuntu/issues/73
https://unix.stackexchange.com/questions/315502/how-to-disable-apt-daily-service-on-ubuntu-cloud-vm-image
To manage notifications about this bug go to:
https://bugs.launchpad.net/apt/+bug/1693361/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
