[Touch-packages] [Bug 1624644] Re: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software
This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2 --- unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium * Don't check blacklist too early and report updates from not allowed origins as kept back. (LP: #1781176) * test/test_blacklisted_wrong_origin.py: Fix and enable test * Filter out progress indicator from dpkg log (LP: #1599646) * Clear cache when autoremoval fails (LP: #1779157) * Find autoremovable kernel packages using the patterns in APT's way (LP: #1815494) unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium * Start service after systemd-logind.service to be able to take inhibition lock (LP: #1806487) * Handle gracefully when logind is down (LP: #1806487) unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium * Backport to Xenial (LP: #1702793) * Revert to build-depending on debhelper (>= 9~) and dh-systemd * Revert configuration example changes to avoid triggering a debconf question * debian/postinst: Update recovery to be triggered on Xenial's package versions unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown() Performing upgrades in service's ExecStop did not work when the upgrades involved restarting services because systemd blocked other stop/start actions making maintainer scripts time out and be killed leaving a broken system behind. Running unattended-upgrades.service before shutdown.target as a oneshot service made it run after unmounting filesystems and scheduling services properly on shutdown is a complex problem and adding more services to the mix make it even more fragile. The solution of monitoring PrepareForShutdown() signal from DBus allows Unattended Upgrade to run _before_ the jobs related to shutdown are queued thus package upgrades can safely restart services without risking causing deadlocks or breaking part of the shutdown actions. Also ask running unattended-upgrades to stop when shutdown starts even in InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to UnattendedUpgradesShutdown class. (LP: #1778219) * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219) This allows more time for unattended-upgrades to shut down gracefully or even install a few packages in InstallOnShutdown mode, but is still a big step back from the 30 minutes allowed for InstallOnShutdown previously. Users enabling InstallOnShutdown node are advised to increase InhibitDelayMaxSec even further possibly to 30 minutes. - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown changes * Ignore "W503 line break before binary operator" because it will become the best practice and breaks the build * Stop using ActionGroups, they interfere with apt.Cache.clear() causing all autoremovable packages to be handled as newly autoremovable ones and be removed by default. Dropping ActionGroup usage does not slow down the most frequent case of not having anything to upgrade and when there are packages to upgrade the gain is small compared to the actual package installation. Also collect autoremovable packages before adjusting candidates because that also changed .is_auto_removable attribute of some of them. (LP: #1803749) (Closes: #910874) unattended-upgrades (1.1ubuntu1.18.04.6) bionic; urgency=medium * Unlock for dpkg operations with apt_pkg.pkgsystem_unlock_inner() when it is available. Also stop running when reacquiring the lock fails. Thanks to Julian Andres Klode for original partial patch (LP: #1789637) * Skip rebuilding python-apt in upgrade autopkgtests. Python-apt has a new build dependency making the rebuilding as is failing and the reference handling issue is worked around in unattended-upgrades already. (LP: #1781586) * Stop trying when no adjustment could be made and adjust package candidates only to lower versions (LP: #1785093) * Skip already adjusted packages from being checked for readjusting. This makes it clearer that the recursion ends and can also be a bit quicker. (LP: #1785093) unattended-upgrades (1.1ubuntu1.18.04.5) bionic; urgency=medium * Stop updating the system when reacquiring the dpkg system lock fails. (LP: #1260041) unattended-upgrades (1.1ubuntu1.18.04.4) bionic; urgency=medium * Redirect stderr output in upgrade-between-snapshots, too, otherwise it breaks the test sometimes (LP: #1781446) unattended-upgrades (1.1ubuntu1.18.04.3) bionic; urgency=medium * Redirect stderr output in upgrade-all-security, otherwise it breaks the test (LP: #1781446) unattended-upgrades (1.1ubuntu1.18.04.2) bionic; urgency=medium [ Balint Reczey ] * Clear cache when autoremoval is invalid for a package set marked for removal and cle
[Touch-packages] [Bug 1624644] Re: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software
@jarnos #61 this is a separate issue fixed later, calculating upgradable packages leaves the upgradable packages in the cache, and u-u counts them as the reverse dependencies of the first kernel to be removed. This keeps the first kernel on the system, but when there are no upgradable packages in a later run this kernel can be removed, too. See: https://github.com/mvo5/unattended-upgrades/commit/93d43fbcd53c5df5ce69a16b26a981bf06ce3085 https://github.com/mvo5/unattended-upgrades/commit/654898b05c933047ca8c97df655743aab0898db1 https://github.com/mvo5/unattended-upgrades/commit/1a39eb257ad786902de11add212879241919be44 In this particular case -extra was detected not autoremovable because of the dirty cache, but it was successfully removed when trying to remove linux-image-4.8.0-54-generic. The printout is confusing indeed, but no harm was done, since only valid autoremovals were performed. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unattended-upgrades in Ubuntu. https://bugs.launchpad.net/bugs/1624644 Title: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software Status in unattended-upgrades package in Ubuntu: Fix Released Status in update-manager package in Ubuntu: Fix Released Status in unattended-upgrades source package in Xenial: Fix Released Status in update-manager source package in Xenial: Fix Released Status in unattended-upgrades source package in Artful: Won't Fix Status in update-manager source package in Artful: Fix Released Bug description: [Impact] * Update-manager and unattended-upgrades install many kernel packages during the lifetime of a release but does not remove them automatically leading to those packages filling disk space potentially completely filling /boot and making the system unable to install updates or even boot. * Stable release users are impacted by this bug for years and their systems already collected many autoremovable unused kernel packages, thus they would benefit from backporting the fix greatly. * The bug is fixed by removing autoremovable (not currently booted) kernel packages when running unattended-upgrades or update-manager. Update manager offers the kernel removals when there are other updates to be installed. [Test Case] Note: test either update-manager or unattended-upgrades, not both at the same time. If you remove unused kernels by the former, you can not test the function in the latter. 1. Install kernel packages to be removed, mark them auto-installed and run apt's kernel hook script to make apt consider them autoremovable and simulate apt autoremove to get list of autoremovable packages: sudo apt install -y linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic sudo apt-mark auto linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic sudo /etc/kernel/postinst.d/apt-auto-removal apt autoremove --simulate 2. (for update-manager; add something for it to update as update- manager will not show removable packages, if there is not something to update, right?) Downgrade a package to be upgraded: sudo apt-get install -y --allow-downgrades ca- certificates=20160104ubuntu1 3. (update-manager). Run update-manager and observe that kernel packages are offered for removal in Details of updates. sudo update-manager 4. (update-manager) Click on Install Now and observe that the kernel packages are removed. 2. (unattended-upgrades, the fix comes in an update of u-u) Run unattended-upgrades manually and observe the removal of the autoremovable kernel packages: sudo unattended-upgrade -v [Regression Potential] The change may cause update-manager or unattanded-upgrades to remove used kernel packages or fail to install other package updates. [Other Info] The unattended-upgrades fix is uploaded with many other fixes and those may cause regressions in other areas in unattended-upgrades. [Original bug text] When using default settings for unattended-upgrade i.e. Unattended-Upgrade::Remove-Unused-Dependencies "false"; # default "false" Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; # default "true" in configuration file /etc/apt/apt.conf.d/50unattended-upgrades, unattended-upgrade is unable to remove packages that become unused in conjunction with updating by other software such as update-manager or apt full-upgrade. This is because unattended-upgrade compares the list of unneeded packages before and after it upgrades packages to detect which packages are new unused ones. Consequently, if user installs new kernels using e.g. update-manager, the excessive kernels will not be removed by unattended-upgrade, and eventually (small) /boot will become full. Expected behavior: handle removing of unused
[Touch-packages] [Bug 1624644] Re: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software
** Description changed: [Impact] * Update-manager and unattended-upgrades install many kernel packages during the lifetime of a release but does not remove them automatically leading to those packages filling disk space potentially completely filling /boot and making the system unable to install updates or even boot. * Stable release users are impacted by this bug for years and their systems already collected many autoremovable unused kernel packages, thus they would benefit from backporting the fix greatly. * The bug is fixed by removing autoremovable (not currently booted) kernel packages when running unattended-upgrades or update-manager. Update manager offers the kernel removals when there are other updates to be installed. [Test Case] + Note: test either update-manager or unattended-upgrades, not both at the + same time. If you remove unused kernels by the former, you can not test + the function in the latter. + 1. Install kernel packages to be removed, mark them auto-installed and - run apt's kernel hook script to make apt consider them autoremovable: + run apt's kernel hook script to make apt consider them autoremovable and + simulate apt autoremove to get list of autoremovable packages: sudo apt install -y linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic sudo apt-mark auto linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic sudo /etc/kernel/postinst.d/apt-auto-removal + apt autoremove --simulate - 2. Also downgrade a package to be upgraded: + 2. (for update-manager; add something for it to update as update- + manager will not show removable packages, if there is not something to + update, right?) Downgrade a package to be upgraded: sudo apt-get install -y --allow-downgrades ca- certificates=20160104ubuntu1 3. (update-manager). Run update-manager and observe that kernel packages are offered for removal in Details of updates. sudo update-manager 4. (update-manager) Click on Install Now and observe that the kernel packages are removed. - 3. (unattended-upgrades, the fix comes in an update of u-u) Run + 2. (unattended-upgrades, the fix comes in an update of u-u) Run unattended-upgrades manually and observe the removal of the autoremovable kernel packages: sudo unattended-upgrade -v [Regression Potential] The change may cause update-manager or unattanded-upgrades to remove used kernel packages or fail to install other package updates. [Other Info] The unattended-upgrades fix is uploaded with many other fixes and those may cause regressions in other areas in unattended-upgrades. [Original bug text] When using default settings for unattended-upgrade i.e. Unattended-Upgrade::Remove-Unused-Dependencies "false"; # default "false" Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; # default "true" in configuration file /etc/apt/apt.conf.d/50unattended-upgrades, unattended-upgrade is unable to remove packages that become unused in conjunction with updating by other software such as update-manager or apt full-upgrade. This is because unattended-upgrade compares the list of unneeded packages before and after it upgrades packages to detect which packages are new unused ones. Consequently, if user installs new kernels using e.g. update-manager, the excessive kernels will not be removed by unattended-upgrade, and eventually (small) /boot will become full. Expected behavior: handle removing of unused packages differently at least until other package management software installed by default can handle removing of new unused packages. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: unattended-upgrades 0.90 ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16 Uname: Linux 4.4.0-36-generic i686 ApportVersion: 2.20.1-0ubuntu2.1 Architecture: i386 CurrentDesktop: XFCE Date: Sat Sep 17 11:28:44 2016 InstallationDate: Installed on 2016-09-05 (11 days ago) InstallationMedia: Mythbuntu 16.04.1 LTS "Xenial Xerus" - Release i386 (20160719) PackageArchitecture: all SourcePackage: unattended-upgrades UpgradeStatus: No upgrade log present (probably fresh install) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unattended-upgrades in Ubuntu. https://bugs.launchpad.net/bugs/1624644 Title: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software Status in unattended-upgrades package in Ubuntu: Fix Released Status in update-manager package in Ubuntu: Fix Released Status in unattended-upgrades source package in Xenial: Fix Committed Status in update-manager source package in Xenial: Fix Released Status in unattended-upgrades source package in Artful: Won't Fix Status in update-manager source pac
[Touch-packages] [Bug 1624644] Re: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software
@rbalint about your test case: I wonder why linux-image-4.8.0-53-generic was not removed by u-u? Was it the booted kernel? You also did not run 'sudo /etc/kernel/postinst.d/apt-auto-removal' before running u-u. Your test case does not show how the kernels were installed. In the following, I show the extraction of terminal output of a more complete test case for u-u. It installs linux-image- extra-4.4.0-141-generic by apt. Output of 'apt autoremove --simulate' shows it would remove the kernel, and one unneeded package that is not a kernel related. Whereas u-u just removes the kernel (which may be the expected behavior). $ set -x; sudo apt install -y unattended-upgrades/xenial-proposed linux-image-extra-4.4.0-141-generic; sudo apt-mark auto linux-image-extra-4.4.0-141-generic; sudo /etc/kernel/postinst.d/apt-auto-removal; apt autoremove --simulate; sudo unattended-upgrade -v; set +x + sudo apt install -y unattended-upgrades/xenial-proposed linux-image-extra-4.4.0-141-generic Reading package lists... Done Building dependency tree Reading state information... Done unattended-upgrades is already the newest version (1.1ubuntu1.18.04.7~16.04.2). Selected version '1.1ubuntu1.18.04.7~16.04.2' (Ubuntu:16.04/xenial-proposed [all]) for 'unattended-upgrades' The following package was automatically installed and is no longer required: xscreensaver-data Use 'sudo apt autoremove' to remove it. Suggested packages: fdutils linux-doc-4.4.0 | linux-source-4.4.0 linux-tools linux-headers-4.4.0-141-generic The following NEW packages will be installed: linux-image-4.4.0-141-generic linux-image-extra-4.4.0-141-generic 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 58,7 MB of archives. After this operation, 224 MB of additional disk space will be used. Get:1 http://ubuntu.mirror.true.nl/ubuntu xenial-updates/main amd64 linux-image-4.4.0-141-generic amd64 4.4.0-141.167 [22,2 MB] Get:2 http://ubuntu.mirror.true.nl/ubuntu xenial-updates/main amd64 linux-image-extra-4.4.0-141-generic amd64 4.4.0-141.167 [36,5 MB] Fetched 58,7 MB in 26s (2 233 kB/s) Selecting previously unselected package linux-image-4.4.0-141-generic. (Reading database ... 332942 files and directories currently installed.) Preparing to unpack .../linux-image-4.4.0-141-generic_4.4.0-141.167_amd64.deb ... Examining /etc/kernel/preinst.d/ run-parts: executing /etc/kernel/preinst.d/intel-microcode 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic Done. Unpacking linux-image-4.4.0-141-generic (4.4.0-141.167) ... Selecting previously unselected package linux-image-extra-4.4.0-141-generic. Preparing to unpack .../linux-image-extra-4.4.0-141-generic_4.4.0-141.167_amd64.deb ... Unpacking linux-image-extra-4.4.0-141-generic (4.4.0-141.167) ... Setting up linux-image-4.4.0-141-generic (4.4.0-141.167) ... Running depmod. update-initramfs: deferring update (hook will be called later) Examining /etc/kernel/postinst.d. run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic run-parts: executing /etc/kernel/postinst.d/dkms 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic Error! Your kernel headers for kernel 4.4.0-141-generic cannot be found. Please install the linux-headers-4.4.0-141-generic package, or use the --kernelsourcedir option to tell DKMS where it's located Error! Your kernel headers for kernel 4.4.0-141-generic cannot be found. Please install the linux-headers-4.4.0-141-generic package, or use the --kernelsourcedir option to tell DKMS where it's located run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic update-initramfs: Generating /boot/initrd.img-4.4.0-141-generic run-parts: executing /etc/kernel/postinst.d/pm-utils 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic run-parts: executing /etc/kernel/postinst.d/update-notifier 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.15.0-46-generic Found initrd image: /boot/initrd.img-4.15.0-46-generic Found linux image: /boot/vmlinuz-4.4.0-143-generic Found initrd image: /boot/initrd.img-4.4.0-143-generic Found linux image: /boot/vmlinuz-4.4.0-141-generic Found initrd image: /boot/initrd.img-4.4.0-141-generic Found memtest86+ image: /memtest86+.elf Found memtest86+ image: /memtest86+.bin Found Ubuntu 16.04.1 LTS (16.04) on /dev/sdc2 done Setting up linux-image-extra-4.4.0-141-generic (4.4.0-141.167) ... run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-141-generic /boot/vmlinuz-4.4.0-141-generic run-parts: executing /etc/kernel/postinst.d/dkms 4.4.0-14
[Touch-packages] [Bug 1624644] Re: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software
rbalint, in the output there is some oddity: "Keeping auto-removable linux-image-extra-4.8.0-54-generic package(s) because it would also remove the following packages which should be kept in this step: libpam-systemd libsystemd0 libudev1 systemd systemd-sysv udev (Reading database ... 53554 files and directories currently installed.) Removing linux-image-extra-4.8.0-54-generic (4.8.0-54.57~16.04.1) ..." So it says it is keeping linux-image-extra-4.8.0-54-generic and right thereafter it is removing the same package. ** Summary changed: - By default settings unattended-upgrade does not automatically remove packages that become unused in conjunction with updating by other software + By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unattended-upgrades in Ubuntu. https://bugs.launchpad.net/bugs/1624644 Title: By default settings unattended-upgrade does not automatically remove kernel packages that become unused in conjunction with updating by other software Status in unattended-upgrades package in Ubuntu: Fix Released Status in update-manager package in Ubuntu: Fix Released Status in unattended-upgrades source package in Xenial: Fix Committed Status in update-manager source package in Xenial: Fix Released Status in unattended-upgrades source package in Artful: Won't Fix Status in update-manager source package in Artful: Fix Released Bug description: [Impact] * Update-manager and unattended-upgrades install many kernel packages during the lifetime of a release but does not remove them automatically leading to those packages filling disk space potentially completely filling /boot and making the system unable to install updates or even boot. * Stable release users are impacted by this bug for years and their systems already collected many autoremovable unused kernel packages, thus they would benefit from backporting the fix greatly. * The bug is fixed by removing autoremovable (not currently booted) kernel packages when running unattended-upgrades or update-manager. Update manager offers the kernel removals when there are other updates to be installed. [Test Case] 1. Install kernel packages to be removed, mark them auto-installed and run apt's kernel hook script to make apt consider them autoremovable: sudo apt install -y linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic sudo apt-mark auto linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic sudo /etc/kernel/postinst.d/apt-auto-removal 2. Also downgrade a package to be upgraded: sudo apt-get install -y --allow-downgrades ca- certificates=20160104ubuntu1 3. (update-manager). Run update-manager and observe that kernel packages are offered for removal in Details of updates. sudo update-manager 4. (update-manager) Click on Install Now and observe that the kernel packages are removed. 3. (unattended-upgrades, the fix comes in an update of u-u) Run unattended-upgrades manually and observe the removal of the autoremovable kernel packages: sudo unattended-upgrade -v [Regression Potential] The change may cause update-manager or unattanded-upgrades to remove used kernel packages or fail to install other package updates. [Other Info] The unattended-upgrades fix is uploaded with many other fixes and those may cause regressions in other areas in unattended-upgrades. [Original bug text] When using default settings for unattended-upgrade i.e. Unattended-Upgrade::Remove-Unused-Dependencies "false"; # default "false" Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; # default "true" in configuration file /etc/apt/apt.conf.d/50unattended-upgrades, unattended-upgrade is unable to remove packages that become unused in conjunction with updating by other software such as update-manager or apt full-upgrade. This is because unattended-upgrade compares the list of unneeded packages before and after it upgrades packages to detect which packages are new unused ones. Consequently, if user installs new kernels using e.g. update-manager, the excessive kernels will not be removed by unattended-upgrade, and eventually (small) /boot will become full. Expected behavior: handle removing of unused packages differently at least until other package management software installed by default can handle removing of new unused packages. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: unattended-upgrades 0.90 ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16 Uname: Linux 4.4.0-36-generic i686 ApportVersion: 2.20.1-0ubuntu2.1 Architecture: i386 CurrentDesktop: XFCE Date: Sat Sep 17 11:28:44 2016 InstallationDate: Installed on 2016-09-05 (11 days a
