[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
** Description changed: + [Impact] + + Docker uses containerd under the hood. When containerd is upgraded it + stops and restarts its service; docker stops when containerd stops but + doesn’t restart. Particularly when doing unattended upgrades, an SRU + fix rolled out for containerd can result in unexpected and widespread + service outages for docker. + + [Test Case] + + $ sudo apt install docker.io + $ sudo systemctl start docker + $ systemctl status docker | grep Active + Active: active (running) since[...] + $ systemctl status containerd | grep Active + Active: active (running) since[...] + + $ docker pull ubuntu/redis:latest + $ docker run -e REDIS_PASSWORD=1234 --network host \ + --name test-redis -d ubuntu/redis:latest + $ telnet localhost 6379 + $ docker container logs test-redis + + $ sudo apt install --reinstall containerd + $ systemctl status containerd | grep Active + Active: active (running) since + $ systemctl status docker | grep Active + Active: inactive (dead) since [...]; 8s ago + $ docker container logs test-redis + + [Where Problems Could Occur] + + The challenge with this issue is addressing all important corner cases, + and as such the biggest risk is that we miss a corner case and fail to + keep the two services running when they should. Areas to watch will be + failures during start/stop/restart/upgrade type operations. Issues + during runtime are unlikely to relate to this change. + + [Original Report] + Hello, Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen! From the log you find: 2020-12-01 06:40:13,881 INFO Starting unattended upgrades script 2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security 2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io 2020-12-01 06:40:13,882 INFO Initial whitelist (not strict): 2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils 2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log 2020-12-01 06:40:46,996 INFO All upgrades installed 2020-12-01 06:40:50,732 INFO Starting unattended upgrades script 2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security 2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io 2020-12-01 06:40:50,733 INFO Initial whitelist (not strict): Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment. This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own. PS: Not to say that on some servers the docker daemon did not even restart.. ** Summary changed: - unattended-upgrade still restarts blacklisted daemons + [SRU] unattended-upgrade still restarts blacklisted daemons -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: [SRU] unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
This bug was fixed in the package docker.io - 19.03.13-0ubuntu4 --- docker.io (19.03.13-0ubuntu4) hirsute; urgency=medium * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not stop when containerd is upgraded, by using Wants= rather than BindTo=. (LP: #1870514) * d/rules: Fix docker.io to not restart its service during package upgrades, to prevent service downtime from automatic updates via unattended-upgrade. (LP: #1906364) -- Bryce Harrington Fri, 04 Dec 2020 23:02:49 + ** Changed in: docker.io (Ubuntu Hirsute) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
Thanks @bryce for the detailed updates. For us personally its the second time we run into a docker daemon restart on production (including all container restarts). The only reason why this time it got much more attention is that a bug prevented a clean restart of the daemon, leaving containers offline. This fact tells us that most of the people never noticed that their containers were also during the last update restartet. (Which is fine because mostly a short restart, for example of a webserver nobody would notice) This is also fine of most of our servers because they running lightweight containers. The only reason why we noticed it (this and last time) is that we running heavy database containers (for example elasticsearch with couple of TBs) which have a restart time of minutes instead of seconds. Especially its critical (and loosing consistency) when containers of the same kind get restarted at the same time. All of this chain is triggered by the unattended updates. Thats why we excluded the automatic updates for the docker package. But this doesnt help if depending package updates still restarting the docker daemon/containers. Since the unattended package is still marked as 'wont-fix', and the dependencies problem is explained above.. The only way to safely prevent docker daemon restarts is to fully disable the whole unattended updates? Can someone confirm this? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
I'd like to give you all an update and outline our plans for this. The Canonical server team has made analysis of this issue a top priority. We've identified and tested out several possible theories. Our findings suggest that the breakage involves two distinct issues, one the BindTo= issue mentioned above, the other caused by a bug in the docker.io package causing the service to stop on package upgrade; see specifically the service stop command at the end of /var/lib/dpkg/info/docker.io.prerm. We'll use LP: #1870514 to track the former issue, and #1906364 the latter. LP: #1658691 gives some past background for reference. The tricky part is that unfortunately any change we make to docker.io requires the running of the prerm script (the version of the script already present on your system, not the one we'd be installing), and thus triggers the bug. In other words, updating your system to prevent the bug will cause one more docker stop. Thereafter, the upgrade will not restart the service when rolling out CVE fixes to either containerd or docker.io; it may prompt to do so if running interactively (e.g. https://imgur.com/2Za5dbQ.png), otherwise it should respect the debconf setting. We would appreciate feedback, testing and/or review of the proposed fix, available in this PPA: https://launchpad.net/~bryce/+archive/ubuntu/containerd-sru-lp1870514 -docker-dh/ ** Also affects: unattended-upgrades (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: docker.io (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: containerd (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: unattended-upgrades (Ubuntu Hirsute) Importance: Undecided Status: Won't Fix ** Also affects: docker.io (Ubuntu Hirsute) Importance: Undecided Status: Confirmed ** Also affects: containerd (Ubuntu Hirsute) Importance: Undecided Status: Confirmed ** Also affects: unattended-upgrades (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: docker.io (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: containerd (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: unattended-upgrades (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: docker.io (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: containerd (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: unattended-upgrades (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: docker.io (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: containerd (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: unattended-upgrades (Ubuntu Groovy) Status: New => Won't Fix ** No longer affects: containerd (Ubuntu) ** Changed in: unattended-upgrades (Ubuntu Focal) Status: New => Won't Fix ** Changed in: unattended-upgrades (Ubuntu Bionic) Status: New => Won't Fix ** Changed in: unattended-upgrades (Ubuntu Xenial) Status: New => Won't Fix ** Changed in: docker.io (Ubuntu Xenial) Importance: Undecided => High ** Changed in: docker.io (Ubuntu Xenial) Status: New => In Progress ** Changed in: docker.io (Ubuntu Xenial) Assignee: (unassigned) => Bryce Harrington (bryce) ** Changed in: docker.io (Ubuntu Xenial) Importance: High => Critical ** Changed in: docker.io (Ubuntu Bionic) Importance: Undecided => Critical ** Changed in: docker.io (Ubuntu Bionic) Status: New => In Progress ** Changed in: docker.io (Ubuntu Focal) Importance: Undecided => Critical ** Changed in: docker.io (Ubuntu Focal) Status: New => In Progress ** Changed in: docker.io (Ubuntu Groovy) Importance: Undecided => Critical ** Changed in: docker.io (Ubuntu Groovy) Status: New => In Progress ** Changed in: docker.io (Ubuntu Hirsute) Importance: Undecided => Critical ** Changed in: docker.io (Ubuntu Hirsute) Status: Confirmed => In Progress ** Changed in: docker.io (Ubuntu Hirsute) Assignee: (unassigned) => Bryce Harrington (bryce) ** No longer affects: containerd (Ubuntu Xenial) ** No longer affects: containerd (Ubuntu Bionic) ** No longer affects: containerd (Ubuntu Focal) ** No longer affects: containerd (Ubuntu Groovy) ** No longer affects: containerd (Ubuntu Hirsute) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
** Merge proposal linked: https://code.launchpad.net/~bryce/ubuntu/+source/docker.io/+git/docker.io/+merge/394913 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
This sounds like another dupe of https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1870514 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: docker.io (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: containerd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
Sorry but im not really on your side here and it also doesnt really solve the problem. I added docker.io to the blacklist because i want to take care manually about the upgrades, it should leave in any case this package alone. Adding here dependencies (containerd) for the packages will never solve the issue, also i have to maintain this list, because dependencies (at least in theorie) could change over time. Just to prevent docker.io from restart i have to add a huge set of packages (out of the list here https://packages.ubuntu.com/focal/docker.io) to prevent it from updating. On top i have to go through all of the first level packages, to check if it has dependencies on the next level, because those could trigger a restart of the first level package, which will chain on the package which i try to prevent. As far as i get it currently, if unattended-upgrades is not considering dependencies of blacklisted packages, theres no way except building an incredibly huge list of packages which you dont want to update, and regulary recalculate this list, because of maybe changed dependencies.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
Containerd unsuccessfully restarting docker is tracked at LP: #1870514. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
I'm sorry you experienced problems when unattended-upgrades applied security fixes. Failure to restart a service is either an issue with the package or with the local configuration, thus I mark docker.io and containerd being affected. The blacklist is listing _packages_ not services and not applied transitively to dependencies nor reverse dependencies, thus unattended- upgraded was and is expected to upgrade containerd. Please list containerd in the blacklist if you would like to prevent it to be upgraded and trigger a docker restart. ** Also affects: containerd (Ubuntu) Importance: Undecided Status: New ** Changed in: unattended-upgrades (Ubuntu) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
** Also affects: docker.io (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: unattended-upgrades (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1906364] Re: unattended-upgrade still restarts blacklisted daemons
We have at least 4 servers on which docker daemon was shutdown due to this unattended upgrade, and was not restarted. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906364 Title: unattended-upgrade still restarts blacklisted daemons To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1906364/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
