This upstream patch needs to be backported to the 1.2 series for Xenial. If left unfixed in Xenial it opens a timing window every hour with Ubuntu Cloud Archive where users can fall into the "Some packages could not be authenticated" state without Release.gpg that is described in the description. This state will not self correct until an hour has passed an another apt-get update is run. This really impacts automated deployment technologies such as Juju and Ansible because if they hit this hourly window with UCA their fallback retries on apt-get update will not work and the automated deployments fail. The noted upstream Debian bug is specifically about trying to do an apt-get update without adding the keys first. The timing window that users can hit will occur even if you add the keys first.
Now for background information to explain the assertions above. Ubuntu Cloud Archive updates its files, and more importantly the timestamps on its files including the Release and Release.gpg file every hour. The timestamps are updated to be 45 minutes past the hour. The UCA servers start to reflect these changes around 50 minutes after the hour with a rolling update of the Packages and then the Release.* files. They are not updated as an atomic unit as seen from an HTTP client. So the order of events is: 1. User or automation adds keys by installing the 'ubuntu-cloud-keyring' apt pacakge. 2. User adds the UCA repo using the Ansible apt_repository or other technique, possilbly just adding the repo to a sources list file under /etc/apt/sources/sources.list.d. 3. Either the tooling (apt_repository module) or the user triggers an apt-get update or other apt cache update trigger. If this cache update hits the timing window when UCA is being updated you can get into the state where you have the Release file but not the Release.gpg file without triggering a cache or apt-get update failure. A recreation Python main which uses straight python-apt can show this. I will attach my recreation program and output showing the error case. 4. At this point, as shown in the original description no further apt-get updates will fix the situation and any package installs from UCA will fail with "Some packages could not be authenticated". While the timing window may seem small, probably a minute each hour, with complex multi-node OpenStack deployments using Ansible we are seeing this occur fairly frequently. Given the 'juju' in the host name in the original description I suspect that multi-node orchestrated Juju charm deployments using UCA are also hitting this often. The bug is particularly harmful to automated deploy tooling because while the deploy tooling normally has apt-get update retries or periodic updates throughout the process, once this error state is entered the apt-get updates do not work to correct it until after an hour has passed and UCA has updated itself. The deployment tooling normally times and fails much sooner than an hour of retries. Here is the annotated log output of the recreation script: ##### # In this snippet we see apt update not pulling down the Release.gpg that was deleted right before the update to test # its ability to pull down a new copy. ##### Before update 2017-02-06 22:50:01.915 After update 2017-02-06 22:50:03.044 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421402.8 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg Before update 2017-02-06 22:50:03.045 After update 2017-02-06 22:50:06.387 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421404.05 mtime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421112.0 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg ##### # In this iteration of the loop we see that the apt update has removed the Release file as well as not pulling down the Release.gpg. # UCA's hourly update is in progress. ##### Before update 2017-02-06 22:50:06.387 After update 2017-02-06 22:50:07.536 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg ##### # In this iteration of the loop we see that the apt update has updates the packages file and recreated the Release file. # However, it did NOT pull down the Release.gpg file. If an automated tool were to hit this window they enter the bug state # where no amount of updates will fix this until after the next hourly UCA update. ##### Before update 2017-02-06 22:50:07.536 After update 2017-02-06 22:50:09.966 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421409.92 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_main_binary-ppc64el_Packages: 1486421409.92 mtime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_main_binary-ppc64el_Packages: 1486421111.0 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg ##### # In this iteration of the loop the Release file has been removed by the update call again. ##### Before update 2017-02-06 22:50:09.967 After update 2017-02-06 22:50:12.491 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg ##### # Now the update call put the release file back but not the Release.gpg, again, entering the bug state. ##### Before update 2017-02-06 22:50:12.492 After update 2017-02-06 22:50:14.869 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421414.82 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg ##### # Update removed the Release file again. ##### Before update 2017-02-06 22:50:14.870 After update 2017-02-06 22:50:17.336 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg ##### # Now finally, both Release and Release.gpg are there. ##### Before update 2017-02-06 22:50:17.336 After update 2017-02-06 22:50:19.885 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421419.83 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg: 1486421419.83 mtime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg: 1486421116.0 ##### # The recreate program goes along its way, having deleted the Release.gpg and it is not pulled down by update. ##### Before update 2017-02-06 22:50:19.886 After update 2017-02-06 22:50:22.208 ctime changed for file /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release: 1486421421.81 Path does not exist /var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release.gpg -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1657440 Title: apt won't redownload Release.gpg Status in apt package in Ubuntu: Confirmed Bug description: # apt --version apt 1.2.18 (amd64) xenial I got myself into a situation where a repository has a Release and a Release.gpg file, but apt is just ignoring the gpg one and won't download it via apt update for some reason: The repository in question is http://ubuntu- cloud.archive.canonical.com/ubuntu/dists/xenial-updates/newton/. See how locally I have just the Release file: root@juju-cb14ed-0-lxd-3:/var/lib/apt/lists# l *Release* -rw-r--r-- 1 root root 100K Jan 15 18:03 archive.ubuntu.com_ubuntu_dists_xenial-backports_InRelease -rw-r--r-- 1 root root 242K Apr 21 2016 archive.ubuntu.com_ubuntu_dists_xenial_InRelease -rw-r--r-- 1 root root 100K Jan 18 11:42 archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease -rw-r--r-- 1 root root 100K Jan 18 11:42 security.ubuntu.com_ubuntu_dists_xenial-security_InRelease -rw-r--r-- 1 root root 7.7K Jan 18 11:45 ubuntu-cloud.archive.canonical.com_ubuntu_dists_xenial-updates_newton_Release Now I try an update. See how the Release.gpg file gets a "Hit:" instead of a "Get:": root@juju-cb14ed-0-lxd-3:/var/lib/apt/lists# apt update Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Ign:3 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton InRelease Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB] Hit:5 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release Get:6 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release.gpg [543 B] Hit:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease Fetched 205 kB in 0s (395 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 8 packages can be upgraded. Run 'apt list --upgradable' to see them. And I can't install packages: root@juju-cb14ed-0-lxd-3:/var/lib/apt/lists# apt dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following NEW packages will be installed: python3-setuptools The following packages will be upgraded: dh-python dnsmasq-base python-pkg-resources python-setuptools python3-cryptography python3-pkg-resources python3-requests python3-urllib3 8 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 1,193 kB of archives. After this operation, 808 kB of additional disk space will be used. Do you want to continue? [Y/n] WARNING: The following packages cannot be authenticated! dh-python dnsmasq-base python-setuptools python-pkg-resources python3-pkg-resources python3-setuptools python3-cryptography python3-requests python3-urllib3 Install these packages without verification? [y/N] n E: Some packages could not be authenticated root@juju-cb14ed-0-lxd-3:/var/lib/apt/lists# Somehow apt is thinking it has the Release.gpg file, but it doesn't? This server is behind a squid proxy. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1657440/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
