Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On Thu, 15 Apr 2021 13:24:29 -0400 Antoine Beaupre wrote: > The release notes, in sections 4.2.2 and 4.8, actually suggest *three* > *different* ways of finding what are essential orphaned packages: > > aptitude search '~o' > aptitude search '?narrow(?installed, ?not(?origin(Debian)))' > apt-forktracer | sort > I think this bug has been fixed. Current text is pretty clear there are two separate issues: - "non-debian packages" (4.2.2, which has the second two commands): remove before upgrading (and (perhaps no-longer?) mentions orphaned packages at all) - "removed from debian" (4.8, which has the first command): can only be done after upgrading It is perhaps worth considering more clearly splitting "before the release" and "after the release" actions into separate chapters? (post bullseye?) - eg 4.7 and 4.8 could be put at the start of chapter 5 as they seem like 'issues to be aware of for the new release' as much as the are about 'upgrading'.
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On 19/12/2022 08:10, Paul Gevers wrote: Hi On 15-04-2021 19:39, Antoine Beaupré wrote: I actually forgot that bullseye itself introduces yet another one: apt list ~obsolete But (I'm running bookworm): paul@mulciber ~/release-team/release-notes $ apt list ~obsolete Listing... Error! E: input:2-9: error: Expected end of file ~obsolete ^^^ Did this work in the past? Doubt that ever worked as written — that should be « apt list '~o' ». (The ~ patterns are single letters followed by arguments.) To tail back through this bug, there's a mixture of different recipes considered here. They do different things and will produce very different results depending on whether they are run before or after editing the sources.list+apt update step. "obsolete" packages are packages that are currently installed but that package name is not available in any current apt source. The version of the package installed can be higher or lower than what is currently available. - If you s/bullseye/bookworm/+apt update, packages that are upgradable are not obsolete; the package names that have disappeared completely are, obsolete. - If you installed packages from bullseye-backports and then removed bullseye-backports from your sources.list, those packages aren't available any more, but are also not obsolete. Obsolete packages are matched by both the aptitude and apt searches ~o. Locally built/manually downloaded packages that are in no apt source will also match ~o; locally built/manually downloaded packages where the package name is also in an apt repo will _not_ be included. There's a second collection of packages where the currently installed version cannot be located in any available apt repository. While 'obsolete' packages was only about the package name, this is about the (name, version, architecture) tuple. This set of "not available" packages includes the yet-to-be upgraded packages, and the ones that are somehow at a higher version than what is currently available. So... the actual commands discussed: aptitude search '~o' apt list '~o' Both of these look for obsolete packages as defined above. On this machine it lists gcc-8-base linux-image-5.16.0-2-amd64-unsigned (left over from a previous release and churn in sid) but also "dpkg -i" installed packages like crossover and and draw.io (where those package names do not exist in Debian). My own backports are not included because the package names exist. Likewise packages like rclone I've grabbed the upstream .deb and installed it on this machine are not included. aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))' apt list | grep 'installed,local' This is a monster (it may well have been reinvented multiple times, but I will own up to responsibility for creating it at least one time c2009). It's filtering out apt's magic repo of 'now', which accompanies all installed packages (a bit like /var/lib/dpkg/status in the output of apt-cache policy foo), to find packages that are currently installed but for which (name, version, architecture) doesn't exist in a currently configured repo. On this machine, it matches some 3rd party packages that are installed via dpkg -i and for which there is no apt repo, like crossover (just like 'obsolete') as well as local backports that I've made that are not sitting in any repo, and those upstream .debs like rclone. Backports that I poked into my own local repo are not included because they are available. Packages that I installed directly out of sid onto this bullseye machine and have never upgraded are also on the list because the version part of the (name, version, arch) tuple doesn't match. (The treatment of local repositories is slightly different for the two versions of this command given above) aptitude search '?narrow(?installed, ?not(?origin(Debian)))' apt list '?narrow(?installed, ?not(?origin(Debian)))' This looks for packages not currently installable from a repo with 'Debian' as the name in the Release file, which includes the case where (name, version, arch) doesn't match. Because it's matching on the version string of the package, it spots that upstream .deb for rclone, the local backports, etc. It also adds in 3rd party packages that are in apt repositories, so on this machine signal-desktop and slack-desktop. Note that lots of 3rd party repositories are misconfigured which is unhelpful. Which of the above is useful depends on the stage of the upgrade. Looking for obsolete packages after the upgrade is an important part of the cleanup process. Looking for 3rd party packages before the upgrade is an important part of the preparation. Importantly, the output of any of these commands will depend on whether the sources.list has been edited; we see that quite frequently in #debian, where someone has edited the sources.list for the upgrade and is then freaked out by absolutely every single package sudde
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Hi On 15-04-2021 19:39, Antoine Beaupré wrote: I actually forgot that bullseye itself introduces yet another one: apt list ~obsolete But (I'm running bookworm): paul@mulciber ~/release-team/release-notes $ apt list ~obsolete Listing... Error! E: input:2-9: error: Expected end of file ~obsolete ^^^ Did this work in the past? Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On Ma, 20 apr 21, 11:36:30, Antoine Beaupré wrote: > > I am not sure we should tell people to "remove any non-Debian package" > before the upgrade. They might have legitimate reasons to have > third-party package repositories...? A dist-upgrade is difficult enough, reducing complexity introduced by non-Debian packages and/or repositories (of questionable quality) seems obvious to me. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On 2021-04-16 09:19:35, Justin B Rye wrote: > Antoine Beaupré wrote [...] aptitude search '?narrow(?installed, ?not(?origin(Debian)))' >>> >>> This one (apparently an improvement on the '~i(!~ODebian)' search that >>> was recommended for buster, though the logic is too subtle for me to >>> remember) is looking for a slightly different thing at a different >>> stage in the upgrade process: it's part of the section about getting >>> rid of "non-Debian packages" *before* the upgrade. The '~o' and >>> '!~ODebian' searches find different kinds of "unwanted" package. >> >> Maybe it would be worth clarifying that distinction then? > > I don't know how we make texts more separate than printing them > in two different places with explanatory paragraphs that talk > about different things! But let me have a look. > > # 4.2.2. Remove non-Debian packages > # > # Below there are two methods for finding installed packages that did > # not come from Debian, using either aptitude or apt-forktracer. > # Please note that neither of them are 100% accurate (e.g. the > # aptitude example will list packages that were once provided by > # Debian but no longer are, such as old kernel packages). > > Now that you come to mention it I've always thought that was a bad > example, since after all it isn't exactly a false-positive - old > linux-images really are no-longer-Debian packages, and if you've got > some lying around even before the upgrade, this would be an > appropriate time to get rid of them, as we go on to say a little > later. Wait, why *is* 4.2.2 before 4.2.3? shouldn't it be > * sort out the package database (pending actions or whatever) > * make sure you're at the latest point release > * remove any non-Debian packages > * similarly but separately, remove any obsolete packages > * tidy any relic configs > [...] > Maybe this is turning into the sort of disruptive low-priority change > that should wait for next time. Indeed. It does seem like 4.2.3 should be before 4.2.2. I am not sure we should tell people to "remove any non-Debian package" before the upgrade. They might have legitimate reasons to have third-party package repositories...? >> It might help if the former `~o` is expanded to `?obsolete` in the text, >> to make it clearer how it compares with the latter. > > Unfortunately people also use "obsolete" to mean "autoremovable"! > And this also makes the search less simple and easier to confuse with > the one in 4.2... I just meant to avoid using the shortcut, obscuring it won't help with this sort of confusiong. > [...] In my personal documentation, I've settled on `apt-forktracer`, >>> >>> I'd be interested to know what you find it useful for. >> >> I like that it shows the related versions in the archive, and that it >> has very terse output. > > Yes, but how do you come to be running a system with non-Debian > repositories in your sources and installing packages to inspect the > gory details without already realising you've done that? You have forgotten, it's been years since you messed around with your sources.list. > Now that we've got "https://deb.debian.org/debian/";, we're close to > being able to say that standard procedure is "for the duration of the > upgrade, comment out any lines that don't match that URL". I am not sure we want the release notes to standardize on that as well... But I have considered writing a parser that checks sources.list Origin headers and fixes things up correctly. Because my checklist is getting ridiculous: : Check for pinned, on hold, packages, and possibly disable && rm -f /etc/apt/preferences /etc/apt/preferences.d/* && rm -f /etc/apt/sources.list.d/backports.debian.org.list && rm -f /etc/apt/sources.list.d/backports.list && rm -f /etc/apt/sources.list.d/bullseye.list && rm -f /etc/apt/sources.list.d/buster-backports.list && rm -f /etc/apt/sources.list.d/experimental.list && rm -f /etc/apt/sources.list.d/incoming.list && rm -f /etc/apt/sources.list.d/proposed-updates.list && rm -f /etc/apt/sources.list.d/sid.list && rm -f /etc/apt/sources.list.d/testing.list && apt update && apt -y upgrade && >>> [---suture---] I actually forgot that bullseye itself introduces yet another one: apt list ~obsolete >>> >>> And indeed for section 4.8, which is dealing with tidying up *after* >>> the upgrade, it might make sense to recommend the use of apt instead >>> of aptitude here. >> >> Yeah, and then we get rid of aptitude in the docs in bullseye +1 :) > > Aptitude may no longer be recommended for dist-upgrades, but there are > still reasons to prefer it for day-to-day admin on stable systems. I'm not arguing for deprecating aptitude altogether, but it would seem to me that using less tools in the release notes would be better. > For bullseye-to-bookworm we'll probably want to use apt recipes > wherever possible, with at most a mention that aptitude can also do > this from the full
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Antoine Beaupré wrote >>> aptitude search '~o' >> >> This is nice and simple, but frankly as an aptitude user I wouldn't >> bother. Instead I'd do what the text just above mentions - launch >> aptitude, notice that there was a category for "Obsolete and Locally >> Created Packages" (which would tell me I'd somehow lost track of my >> kernel packages), and purge everything in that category from the >> full-screen interface, without going back to the commandline. > > I think the point here is that you can do: > > aptitude purge '~o' > > ... to avoid loading the GUI. I'd imagine it's to avoid having to deal with that interface - it's much more powerful (and indeed faster) if you're used to it, but we don't want novice users to need to guess it's "↓↓↓_gg"! >>> aptitude search '?narrow(?installed, ?not(?origin(Debian)))' >> >> This one (apparently an improvement on the '~i(!~ODebian)' search that >> was recommended for buster, though the logic is too subtle for me to >> remember) is looking for a slightly different thing at a different >> stage in the upgrade process: it's part of the section about getting >> rid of "non-Debian packages" *before* the upgrade. The '~o' and >> '!~ODebian' searches find different kinds of "unwanted" package. > > Maybe it would be worth clarifying that distinction then? I don't know how we make texts more separate than printing them in two different places with explanatory paragraphs that talk about different things! But let me have a look. # 4.2.2. Remove non-Debian packages # # Below there are two methods for finding installed packages that did # not come from Debian, using either aptitude or apt-forktracer. # Please note that neither of them are 100% accurate (e.g. the # aptitude example will list packages that were once provided by # Debian but no longer are, such as old kernel packages). Now that you come to mention it I've always thought that was a bad example, since after all it isn't exactly a false-positive - old linux-images really are no-longer-Debian packages, and if you've got some lying around even before the upgrade, this would be an appropriate time to get rid of them, as we go on to say a little later. Wait, why *is* 4.2.2 before 4.2.3? shouldn't it be * sort out the package database (pending actions or whatever) * make sure you're at the latest point release * remove any non-Debian packages * similarly but separately, remove any obsolete packages * tidy any relic configs [...] Maybe this is turning into the sort of disruptive low-priority change that should wait for next time. > It might help if the former `~o` is expanded to `?obsolete` in the text, > to make it clearer how it compares with the latter. Unfortunately people also use "obsolete" to mean "autoremovable"! And this also makes the search less simple and easier to confuse with the one in 4.2... [...] >>> In my personal documentation, I've settled on `apt-forktracer`, >> >> I'd be interested to know what you find it useful for. > > I like that it shows the related versions in the archive, and that it > has very terse output. Yes, but how do you come to be running a system with non-Debian repositories in your sources and installing packages to inspect the gory details without already realising you've done that? Now that we've got "https://deb.debian.org/debian/";, we're close to being able to say that standard procedure is "for the duration of the upgrade, comment out any lines that don't match that URL". >> [---suture---] >>> I actually forgot that bullseye itself introduces yet another one: >>> >>> apt list ~obsolete >> >> And indeed for section 4.8, which is dealing with tidying up *after* >> the upgrade, it might make sense to recommend the use of apt instead >> of aptitude here. > > Yeah, and then we get rid of aptitude in the docs in bullseye +1 :) Aptitude may no longer be recommended for dist-upgrades, but there are still reasons to prefer it for day-to-day admin on stable systems. For bullseye-to-bookworm we'll probably want to use apt recipes wherever possible, with at most a mention that aptitude can also do this from the fullscreen mode. But since we also point at 4.8 from 4.2, we can't do that yet; we don't want the extra complication of having to say "if you haven't upgraded yet then use aptitude". > So, TL;DR: we should have this before, to find and cleanup foreign > packages: > > aptitude search '?narrow(?installed, ?not(?origin(Debian)))' > > Presumably `apt list` might support that as well in bullseye? Afraid not - "E: Regex compilation error". -- JBR with qualifications in linguistics, experience as a Debian sysadmin, and probably no clue about this particular package
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On 2021-04-15 21:08:27, Justin B Rye wrote: > Antoine Beaupre wrote: >> The release notes, in sections 4.2.2 and 4.8, actually suggest *three* >> *different* ways of finding what are essential orphaned packages: > > I don't think you mean "orphan" in either of the senses known to > "https://wiki.debian.org/Glossary#orphan";. I should have said "obsolete", I think. >> aptitude search '~o' > > This is nice and simple, but frankly as an aptitude user I wouldn't > bother. Instead I'd do what the text just above mentions - launch > aptitude, notice that there was a category for "Obsolete and Locally > Created Packages" (which would tell me I'd somehow lost track of my > kernel packages), and purge everything in that category from the > full-screen interface, without going back to the commandline. I think the point here is that you can do: aptitude purge '~o' ... to avoid loading the GUI. >> aptitude search '?narrow(?installed, ?not(?origin(Debian)))' > > This one (apparently an improvement on the '~i(!~ODebian)' search that > was recommended for buster, though the logic is too subtle for me to > remember) is looking for a slightly different thing at a different > stage in the upgrade process: it's part of the section about getting > rid of "non-Debian packages" *before* the upgrade. The '~o' and > '!~ODebian' searches find different kinds of "unwanted" package. Maybe it would be worth clarifying that distinction then? It might help if the former `~o` is expanded to `?obsolete` in the text, to make it clearer how it compares with the latter. >> apt-forktracer | sort > > I've never quite been able to understand how it is that anybody > could get themselves into the situation of *needing* this specialised > package installed to work around the weirdness of their setup, but > still need to be told what it is that's unusual about their system. I actually find the output of apt-forktracer to be quite handy. >> Then I also know of those: >> >> apt-show-versions | grep -v /bullseye > > This is another package I've never needed to install on my stable > desktop precisely because it's my stable desktop. If I had a reason > to install it, presumably I'd already know about the reason, and step > one in the bullseye upgrade process should be to get rid of that. Yeah, although I guess that's the point of those commands: figure out if there's a mess in there. I find apt-show-versions to be comparable to apt-forktracer, but a bit more flexible. >> aptitude search '?narrow(?installed, ?not(?origin(Debian)))' > > Yes, that's the one you listed above. Ah yes, sorry for the dupe. >> aptitude search >> '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))' > > Does that do something similar to the above, but less intelligibly, or > something different? I frankly have no idea anymore. >> I frankly don't quite know where I stand with all this anymore, but I >> am getting the strong feeling we're sending an incoherent message >> here. :) > > It's two different messages in two different parts of the release > notes. The first message is roughly "before the upgrade, look at what > you've got installed. If it's a mix of complex pins and PPA packages > and nonsense like that, start by getting rid of all the crazy stuff". > Unfortunately, this relies on the reader to apply some common sense, > so we've fallen into the trap of replacing "subjective" advice with > "objective" diagnostic commandlines. I see. Maybe a quick oneliner explanation before the command could help alleviate that confusion then... > The second message is "after the upgrade, throw out all the stuff > that isn't supported any longer". This really is trivially easy to > automate, as long as you don't confuse it with the previous task. Same here, I guess. >> In my personal documentation, I've settled on `apt-forktracer`, > > I'd be interested to know what you find it useful for. I like that it shows the related versions in the archive, and that it has very terse output. >> but I >> suspect we might want to stick with `aptitude search '~obsolete'` >> because that matches other documentation in the release notes (and >> allows for easy purging). > > But it isn't looking for the same thing. Okay, so just stick to aptitude in both cases then, and don't introduce apt-forktracer. :) >> Is there any reason why we have all that diversity? >> >> What's the right way to do what we actually want here? > > [---suture---] >> I actually forgot that bullseye itself introduces yet another one: >> >> apt list ~obsolete > > And indeed for section 4.8, which is dealing with tidying up *after* > the upgrade, it might make sense to recommend the use of apt instead > of aptitude here. Yeah, and then we get rid of aptitude in the docs in bullseye +1 :) So, TL;DR: we should have this before, to find and cleanup foreign packages: aptitude search '?narrow(?installed, ?not(?origin(Debian)))' Presumably `apt list` mi
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Antoine Beaupre wrote: > The release notes, in sections 4.2.2 and 4.8, actually suggest *three* > *different* ways of finding what are essential orphaned packages: I don't think you mean "orphan" in either of the senses known to "https://wiki.debian.org/Glossary#orphan";. > aptitude search '~o' This is nice and simple, but frankly as an aptitude user I wouldn't bother. Instead I'd do what the text just above mentions - launch aptitude, notice that there was a category for "Obsolete and Locally Created Packages" (which would tell me I'd somehow lost track of my kernel packages), and purge everything in that category from the full-screen interface, without going back to the commandline. > aptitude search '?narrow(?installed, ?not(?origin(Debian)))' This one (apparently an improvement on the '~i(!~ODebian)' search that was recommended for buster, though the logic is too subtle for me to remember) is looking for a slightly different thing at a different stage in the upgrade process: it's part of the section about getting rid of "non-Debian packages" *before* the upgrade. The '~o' and '!~ODebian' searches find different kinds of "unwanted" package. > apt-forktracer | sort I've never quite been able to understand how it is that anybody could get themselves into the situation of *needing* this specialised package installed to work around the weirdness of their setup, but still need to be told what it is that's unusual about their system. > Then I also know of those: > > apt-show-versions | grep -v /bullseye This is another package I've never needed to install on my stable desktop precisely because it's my stable desktop. If I had a reason to install it, presumably I'd already know about the reason, and step one in the bullseye upgrade process should be to get rid of that. > aptitude search '?narrow(?installed, ?not(?origin(Debian)))' Yes, that's the one you listed above. > aptitude search > '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))' Does that do something similar to the above, but less intelligibly, or something different? > I frankly don't quite know where I stand with all this anymore, but I > am getting the strong feeling we're sending an incoherent message > here. :) It's two different messages in two different parts of the release notes. The first message is roughly "before the upgrade, look at what you've got installed. If it's a mix of complex pins and PPA packages and nonsense like that, start by getting rid of all the crazy stuff". Unfortunately, this relies on the reader to apply some common sense, so we've fallen into the trap of replacing "subjective" advice with "objective" diagnostic commandlines. The second message is "after the upgrade, throw out all the stuff that isn't supported any longer". This really is trivially easy to automate, as long as you don't confuse it with the previous task. > In my personal documentation, I've settled on `apt-forktracer`, I'd be interested to know what you find it useful for. > but I > suspect we might want to stick with `aptitude search '~obsolete'` > because that matches other documentation in the release notes (and > allows for easy purging). But it isn't looking for the same thing. > Is there any reason why we have all that diversity? > > What's the right way to do what we actually want here? [---suture---] > I actually forgot that bullseye itself introduces yet another one: > > apt list ~obsolete And indeed for section 4.8, which is dealing with tidying up *after* the upgrade, it might make sense to recommend the use of apt instead of aptitude here. > Apparently, those are also a thing: > > comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep > '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: > (.*)/\1/p' | sort -u) > apt list --installed | awk -F/ '/\[installed,local\]/{print $1}' If they're not getting shorter, you're going the wrong way. -- JBR with qualifications in linguistics, experience as a Debian sysadmin, and probably no clue about this particular package
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
I actually forgot that bullseye itself introduces yet another one: apt list ~obsolete Apparently, those are also a thing: comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u) apt list --installed | awk -F/ '/\[installed,local\]/{print $1}' -- I've got to design so you can put it together out of garbage cans. In part because that's what I started from, but mostly because I don’t trust the industrial structure—they might decide to suppress us weirdos and try to deny us the parts we need. - Lee Felsenstein
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Package: release-notes Severity: minor The release notes, in sections 4.2.2 and 4.8, actually suggest *three* *different* ways of finding what are essential orphaned packages: aptitude search '~o' aptitude search '?narrow(?installed, ?not(?origin(Debian)))' apt-forktracer | sort Then I also know of those: apt-show-versions | grep -v /bullseye aptitude search '?narrow(?installed, ?not(?origin(Debian)))' aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))' I frankly don't quite know where I stand with all this anymore, but I am getting the strong feeling we're sending an incoherent message here. :) In my personal documentation, I've settled on `apt-forktracer`, but I suspect we might want to stick with `aptitude search '~obsolete'` because that matches other documentation in the release notes (and allows for easy purging). Is there any reason why we have all that diversity? What's the right way to do what we actually want here? -- System Information: Debian Release: 10.9 APT prefers stable-debug APT policy: (500, 'stable-debug'), (500, 'stable'), (1, 'experimental'), (1, 'unstable'), (1, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.19.0-16-amd64 (SMP w/4 CPU cores) Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8), LANGUAGE=fr_CA.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled