Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On Thu, 15 Apr 2021 13:24:29 -0400 Antoine Beaupre wrote: > The release notes, in sections 4.2.2 and 4.8, actually suggest *three* > *different* ways of finding what are essential orphaned packages: > > aptitude search '~o' > aptitude search '?narrow(?installed, ?not(?origin(Debian)))' > apt-forktracer | sort > I think this bug has been fixed. Current text is pretty clear there are two separate issues: - "non-debian packages" (4.2.2, which has the second two commands): remove before upgrading (and (perhaps no-longer?) mentions orphaned packages at all) - "removed from debian" (4.8, which has the first command): can only be done after upgrading It is perhaps worth considering more clearly splitting "before the release" and "after the release" actions into separate chapters? (post bullseye?) - eg 4.7 and 4.8 could be put at the start of chapter 5 as they seem like 'issues to be aware of for the new release' as much as the are about 'upgrading'.
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On 19/12/2022 08:10, Paul Gevers wrote:
Hi
On 15-04-2021 19:39, Antoine Beaupré wrote:
I actually forgot that bullseye itself introduces yet another one:
apt list ~obsolete
But (I'm running bookworm):
paul@mulciber ~/release-team/release-notes $ apt list ~obsolete
Listing... Error!
E: input:2-9: error: Expected end of file
~obsolete
^^^
Did this work in the past?
Doubt that ever worked as written — that should be « apt list '~o' ».
(The ~ patterns are single letters followed by arguments.)
To tail back through this bug, there's a mixture of different recipes
considered here. They do different things and will produce very
different results depending on whether they are run before or after
editing the sources.list+apt update step.
"obsolete" packages are packages that are currently installed but that
package name is not available in any current apt source. The version of
the package installed can be higher or lower than what is currently
available.
- If you s/bullseye/bookworm/+apt update, packages that are upgradable
are not obsolete; the package names that have disappeared completely
are, obsolete.
- If you installed packages from bullseye-backports and then removed
bullseye-backports from your sources.list, those packages aren't
available any more, but are also not obsolete.
Obsolete packages are matched by both the aptitude and apt searches ~o.
Locally built/manually downloaded packages that are in no apt source
will also match ~o; locally built/manually downloaded packages where the
package name is also in an apt repo will _not_ be included.
There's a second collection of packages where the currently installed
version cannot be located in any available apt repository. While
'obsolete' packages was only about the package name, this is about the
(name, version, architecture) tuple. This set of "not available"
packages includes the yet-to-be upgraded packages, and the ones that are
somehow at a higher version than what is currently available.
So... the actual commands discussed:
aptitude search '~o'
apt list '~o'
Both of these look for obsolete packages as defined above. On this
machine it lists gcc-8-base linux-image-5.16.0-2-amd64-unsigned (left
over from a previous release and churn in sid) but also "dpkg -i"
installed packages like crossover and and draw.io (where those package
names do not exist in Debian). My own backports are not included because
the package names exist. Likewise packages like rclone I've grabbed the
upstream .deb and installed it on this machine are not included.
aptitude search
'?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
apt list | grep 'installed,local'
This is a monster (it may well have been reinvented multiple times,
but I will own up to responsibility for creating it at least one time
c2009). It's filtering out apt's magic repo of 'now', which accompanies
all installed packages (a bit like /var/lib/dpkg/status in the output of
apt-cache policy foo), to find packages that are currently installed but
for which (name, version, architecture) doesn't exist in a currently
configured repo. On this machine, it matches some 3rd party packages
that are installed via dpkg -i and for which there is no apt repo, like
crossover (just like 'obsolete') as well as local backports that I've
made that are not sitting in any repo, and those upstream .debs like
rclone. Backports that I poked into my own local repo are not included
because they are available. Packages that I installed directly out of
sid onto this bullseye machine and have never upgraded are also on the
list because the version part of the (name, version, arch) tuple doesn't
match. (The treatment of local repositories is slightly different for
the two versions of this command given above)
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
apt list '?narrow(?installed, ?not(?origin(Debian)))'
This looks for packages not currently installable from a repo with
'Debian' as the name in the Release file, which includes the case where
(name, version, arch) doesn't match. Because it's matching on the
version string of the package, it spots that upstream .deb for rclone,
the local backports, etc. It also adds in 3rd party packages that are in
apt repositories, so on this machine signal-desktop and slack-desktop.
Note that lots of 3rd party repositories are misconfigured which is
unhelpful.
Which of the above is useful depends on the stage of the upgrade.
Looking for obsolete packages after the upgrade is an important part of
the cleanup process. Looking for 3rd party packages before the upgrade
is an important part of the preparation.
Importantly, the output of any of these commands will depend on whether
the sources.list has been edited; we see that quite frequently in
#debian, where someone has edited the sources.list for the upgrade and
is then freaked out by absolutely every single package
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Hi On 15-04-2021 19:39, Antoine Beaupré wrote: I actually forgot that bullseye itself introduces yet another one: apt list ~obsolete But (I'm running bookworm): paul@mulciber ~/release-team/release-notes $ apt list ~obsolete Listing... Error! E: input:2-9: error: Expected end of file ~obsolete ^^^ Did this work in the past? Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On Ma, 20 apr 21, 11:36:30, Antoine Beaupré wrote: > > I am not sure we should tell people to "remove any non-Debian package" > before the upgrade. They might have legitimate reasons to have > third-party package repositories...? A dist-upgrade is difficult enough, reducing complexity introduced by non-Debian packages and/or repositories (of questionable quality) seems obvious to me. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On 2021-04-16 09:19:35, Justin B Rye wrote: > Antoine Beaupré wrote [...] aptitude search '?narrow(?installed, ?not(?origin(Debian)))' >>> >>> This one (apparently an improvement on the '~i(!~ODebian)' search that >>> was recommended for buster, though the logic is too subtle for me to >>> remember) is looking for a slightly different thing at a different >>> stage in the upgrade process: it's part of the section about getting >>> rid of "non-Debian packages" *before* the upgrade. The '~o' and >>> '!~ODebian' searches find different kinds of "unwanted" package. >> >> Maybe it would be worth clarifying that distinction then? > > I don't know how we make texts more separate than printing them > in two different places with explanatory paragraphs that talk > about different things! But let me have a look. > > # 4.2.2. Remove non-Debian packages > # > # Below there are two methods for finding installed packages that did > # not come from Debian, using either aptitude or apt-forktracer. > # Please note that neither of them are 100% accurate (e.g. the > # aptitude example will list packages that were once provided by > # Debian but no longer are, such as old kernel packages). > > Now that you come to mention it I've always thought that was a bad > example, since after all it isn't exactly a false-positive - old > linux-images really are no-longer-Debian packages, and if you've got > some lying around even before the upgrade, this would be an > appropriate time to get rid of them, as we go on to say a little > later. Wait, why *is* 4.2.2 before 4.2.3? shouldn't it be > * sort out the package database (pending actions or whatever) > * make sure you're at the latest point release > * remove any non-Debian packages > * similarly but separately, remove any obsolete packages > * tidy any relic configs > [...] > Maybe this is turning into the sort of disruptive low-priority change > that should wait for next time. Indeed. It does seem like 4.2.3 should be before 4.2.2. I am not sure we should tell people to "remove any non-Debian package" before the upgrade. They might have legitimate reasons to have third-party package repositories...? >> It might help if the former `~o` is expanded to `?obsolete` in the text, >> to make it clearer how it compares with the latter. > > Unfortunately people also use "obsolete" to mean "autoremovable"! > And this also makes the search less simple and easier to confuse with > the one in 4.2... I just meant to avoid using the shortcut, obscuring it won't help with this sort of confusiong. > [...] In my personal documentation, I've settled on `apt-forktracer`, >>> >>> I'd be interested to know what you find it useful for. >> >> I like that it shows the related versions in the archive, and that it >> has very terse output. > > Yes, but how do you come to be running a system with non-Debian > repositories in your sources and installing packages to inspect the > gory details without already realising you've done that? You have forgotten, it's been years since you messed around with your sources.list. > Now that we've got "https://deb.debian.org/debian/;, we're close to > being able to say that standard procedure is "for the duration of the > upgrade, comment out any lines that don't match that URL". I am not sure we want the release notes to standardize on that as well... But I have considered writing a parser that checks sources.list Origin headers and fixes things up correctly. Because my checklist is getting ridiculous: : Check for pinned, on hold, packages, and possibly disable && rm -f /etc/apt/preferences /etc/apt/preferences.d/* && rm -f /etc/apt/sources.list.d/backports.debian.org.list && rm -f /etc/apt/sources.list.d/backports.list && rm -f /etc/apt/sources.list.d/bullseye.list && rm -f /etc/apt/sources.list.d/buster-backports.list && rm -f /etc/apt/sources.list.d/experimental.list && rm -f /etc/apt/sources.list.d/incoming.list && rm -f /etc/apt/sources.list.d/proposed-updates.list && rm -f /etc/apt/sources.list.d/sid.list && rm -f /etc/apt/sources.list.d/testing.list && apt update && apt -y upgrade && >>> [---suture---] I actually forgot that bullseye itself introduces yet another one: apt list ~obsolete >>> >>> And indeed for section 4.8, which is dealing with tidying up *after* >>> the upgrade, it might make sense to recommend the use of apt instead >>> of aptitude here. >> >> Yeah, and then we get rid of aptitude in the docs in bullseye +1 :) > > Aptitude may no longer be recommended for dist-upgrades, but there are > still reasons to prefer it for day-to-day admin on stable systems. I'm not arguing for deprecating aptitude altogether, but it would seem to me that using less tools in the release notes would be better. > For bullseye-to-bookworm we'll probably want to use apt recipes > wherever possible, with at most a mention that aptitude can also do > this from the
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Antoine Beaupré wrote >>> aptitude search '~o' >> >> This is nice and simple, but frankly as an aptitude user I wouldn't >> bother. Instead I'd do what the text just above mentions - launch >> aptitude, notice that there was a category for "Obsolete and Locally >> Created Packages" (which would tell me I'd somehow lost track of my >> kernel packages), and purge everything in that category from the >> full-screen interface, without going back to the commandline. > > I think the point here is that you can do: > > aptitude purge '~o' > > ... to avoid loading the GUI. I'd imagine it's to avoid having to deal with that interface - it's much more powerful (and indeed faster) if you're used to it, but we don't want novice users to need to guess it's "↓↓↓_gg"! >>> aptitude search '?narrow(?installed, ?not(?origin(Debian)))' >> >> This one (apparently an improvement on the '~i(!~ODebian)' search that >> was recommended for buster, though the logic is too subtle for me to >> remember) is looking for a slightly different thing at a different >> stage in the upgrade process: it's part of the section about getting >> rid of "non-Debian packages" *before* the upgrade. The '~o' and >> '!~ODebian' searches find different kinds of "unwanted" package. > > Maybe it would be worth clarifying that distinction then? I don't know how we make texts more separate than printing them in two different places with explanatory paragraphs that talk about different things! But let me have a look. # 4.2.2. Remove non-Debian packages # # Below there are two methods for finding installed packages that did # not come from Debian, using either aptitude or apt-forktracer. # Please note that neither of them are 100% accurate (e.g. the # aptitude example will list packages that were once provided by # Debian but no longer are, such as old kernel packages). Now that you come to mention it I've always thought that was a bad example, since after all it isn't exactly a false-positive - old linux-images really are no-longer-Debian packages, and if you've got some lying around even before the upgrade, this would be an appropriate time to get rid of them, as we go on to say a little later. Wait, why *is* 4.2.2 before 4.2.3? shouldn't it be * sort out the package database (pending actions or whatever) * make sure you're at the latest point release * remove any non-Debian packages * similarly but separately, remove any obsolete packages * tidy any relic configs [...] Maybe this is turning into the sort of disruptive low-priority change that should wait for next time. > It might help if the former `~o` is expanded to `?obsolete` in the text, > to make it clearer how it compares with the latter. Unfortunately people also use "obsolete" to mean "autoremovable"! And this also makes the search less simple and easier to confuse with the one in 4.2... [...] >>> In my personal documentation, I've settled on `apt-forktracer`, >> >> I'd be interested to know what you find it useful for. > > I like that it shows the related versions in the archive, and that it > has very terse output. Yes, but how do you come to be running a system with non-Debian repositories in your sources and installing packages to inspect the gory details without already realising you've done that? Now that we've got "https://deb.debian.org/debian/;, we're close to being able to say that standard procedure is "for the duration of the upgrade, comment out any lines that don't match that URL". >> [---suture---] >>> I actually forgot that bullseye itself introduces yet another one: >>> >>> apt list ~obsolete >> >> And indeed for section 4.8, which is dealing with tidying up *after* >> the upgrade, it might make sense to recommend the use of apt instead >> of aptitude here. > > Yeah, and then we get rid of aptitude in the docs in bullseye +1 :) Aptitude may no longer be recommended for dist-upgrades, but there are still reasons to prefer it for day-to-day admin on stable systems. For bullseye-to-bookworm we'll probably want to use apt recipes wherever possible, with at most a mention that aptitude can also do this from the fullscreen mode. But since we also point at 4.8 from 4.2, we can't do that yet; we don't want the extra complication of having to say "if you haven't upgraded yet then use aptitude". > So, TL;DR: we should have this before, to find and cleanup foreign > packages: > > aptitude search '?narrow(?installed, ?not(?origin(Debian)))' > > Presumably `apt list` might support that as well in bullseye? Afraid not - "E: Regex compilation error". -- JBR with qualifications in linguistics, experience as a Debian sysadmin, and probably no clue about this particular package
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
On 2021-04-15 21:08:27, Justin B Rye wrote:
> Antoine Beaupre wrote:
>> The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
>> *different* ways of finding what are essential orphaned packages:
>
> I don't think you mean "orphan" in either of the senses known to
> "https://wiki.debian.org/Glossary#orphan;.
I should have said "obsolete", I think.
>> aptitude search '~o'
>
> This is nice and simple, but frankly as an aptitude user I wouldn't
> bother. Instead I'd do what the text just above mentions - launch
> aptitude, notice that there was a category for "Obsolete and Locally
> Created Packages" (which would tell me I'd somehow lost track of my
> kernel packages), and purge everything in that category from the
> full-screen interface, without going back to the commandline.
I think the point here is that you can do:
aptitude purge '~o'
... to avoid loading the GUI.
>> aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>
> This one (apparently an improvement on the '~i(!~ODebian)' search that
> was recommended for buster, though the logic is too subtle for me to
> remember) is looking for a slightly different thing at a different
> stage in the upgrade process: it's part of the section about getting
> rid of "non-Debian packages" *before* the upgrade. The '~o' and
> '!~ODebian' searches find different kinds of "unwanted" package.
Maybe it would be worth clarifying that distinction then?
It might help if the former `~o` is expanded to `?obsolete` in the text,
to make it clearer how it compares with the latter.
>> apt-forktracer | sort
>
> I've never quite been able to understand how it is that anybody
> could get themselves into the situation of *needing* this specialised
> package installed to work around the weirdness of their setup, but
> still need to be told what it is that's unusual about their system.
I actually find the output of apt-forktracer to be quite handy.
>> Then I also know of those:
>>
>> apt-show-versions | grep -v /bullseye
>
> This is another package I've never needed to install on my stable
> desktop precisely because it's my stable desktop. If I had a reason
> to install it, presumably I'd already know about the reason, and step
> one in the bullseye upgrade process should be to get rid of that.
Yeah, although I guess that's the point of those commands: figure out if
there's a mess in there.
I find apt-show-versions to be comparable to apt-forktracer, but a bit
more flexible.
>> aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>
> Yes, that's the one you listed above.
Ah yes, sorry for the dupe.
>> aptitude search
>> '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
>
> Does that do something similar to the above, but less intelligibly, or
> something different?
I frankly have no idea anymore.
>> I frankly don't quite know where I stand with all this anymore, but I
>> am getting the strong feeling we're sending an incoherent message
>> here. :)
>
> It's two different messages in two different parts of the release
> notes. The first message is roughly "before the upgrade, look at what
> you've got installed. If it's a mix of complex pins and PPA packages
> and nonsense like that, start by getting rid of all the crazy stuff".
> Unfortunately, this relies on the reader to apply some common sense,
> so we've fallen into the trap of replacing "subjective" advice with
> "objective" diagnostic commandlines.
I see. Maybe a quick oneliner explanation before the command could help
alleviate that confusion then...
> The second message is "after the upgrade, throw out all the stuff
> that isn't supported any longer". This really is trivially easy to
> automate, as long as you don't confuse it with the previous task.
Same here, I guess.
>> In my personal documentation, I've settled on `apt-forktracer`,
>
> I'd be interested to know what you find it useful for.
I like that it shows the related versions in the archive, and that it
has very terse output.
>> but I
>> suspect we might want to stick with `aptitude search '~obsolete'`
>> because that matches other documentation in the release notes (and
>> allows for easy purging).
>
> But it isn't looking for the same thing.
Okay, so just stick to aptitude in both cases then, and don't introduce
apt-forktracer. :)
>> Is there any reason why we have all that diversity?
>>
>> What's the right way to do what we actually want here?
>
> [---suture---]
>> I actually forgot that bullseye itself introduces yet another one:
>>
>> apt list ~obsolete
>
> And indeed for section 4.8, which is dealing with tidying up *after*
> the upgrade, it might make sense to recommend the use of apt instead
> of aptitude here.
Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)
So, TL;DR: we should have this before, to find and cleanup foreign
packages:
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
Presumably `apt list`
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Antoine Beaupre wrote:
> The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
> *different* ways of finding what are essential orphaned packages:
I don't think you mean "orphan" in either of the senses known to
"https://wiki.debian.org/Glossary#orphan;.
> aptitude search '~o'
This is nice and simple, but frankly as an aptitude user I wouldn't
bother. Instead I'd do what the text just above mentions - launch
aptitude, notice that there was a category for "Obsolete and Locally
Created Packages" (which would tell me I'd somehow lost track of my
kernel packages), and purge everything in that category from the
full-screen interface, without going back to the commandline.
> aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
This one (apparently an improvement on the '~i(!~ODebian)' search that
was recommended for buster, though the logic is too subtle for me to
remember) is looking for a slightly different thing at a different
stage in the upgrade process: it's part of the section about getting
rid of "non-Debian packages" *before* the upgrade. The '~o' and
'!~ODebian' searches find different kinds of "unwanted" package.
> apt-forktracer | sort
I've never quite been able to understand how it is that anybody
could get themselves into the situation of *needing* this specialised
package installed to work around the weirdness of their setup, but
still need to be told what it is that's unusual about their system.
> Then I also know of those:
>
> apt-show-versions | grep -v /bullseye
This is another package I've never needed to install on my stable
desktop precisely because it's my stable desktop. If I had a reason
to install it, presumably I'd already know about the reason, and step
one in the bullseye upgrade process should be to get rid of that.
> aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
Yes, that's the one you listed above.
> aptitude search
> '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
Does that do something similar to the above, but less intelligibly, or
something different?
> I frankly don't quite know where I stand with all this anymore, but I
> am getting the strong feeling we're sending an incoherent message
> here. :)
It's two different messages in two different parts of the release
notes. The first message is roughly "before the upgrade, look at what
you've got installed. If it's a mix of complex pins and PPA packages
and nonsense like that, start by getting rid of all the crazy stuff".
Unfortunately, this relies on the reader to apply some common sense,
so we've fallen into the trap of replacing "subjective" advice with
"objective" diagnostic commandlines.
The second message is "after the upgrade, throw out all the stuff
that isn't supported any longer". This really is trivially easy to
automate, as long as you don't confuse it with the previous task.
> In my personal documentation, I've settled on `apt-forktracer`,
I'd be interested to know what you find it useful for.
> but I
> suspect we might want to stick with `aptitude search '~obsolete'`
> because that matches other documentation in the release notes (and
> allows for easy purging).
But it isn't looking for the same thing.
> Is there any reason why we have all that diversity?
>
> What's the right way to do what we actually want here?
[---suture---]
> I actually forgot that bullseye itself introduces yet another one:
>
> apt list ~obsolete
And indeed for section 4.8, which is dealing with tidying up *after*
the upgrade, it might make sense to recommend the use of apt instead
of aptitude here.
> Apparently, those are also a thing:
>
> comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep
> '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package:
> (.*)/\1/p' | sort -u)
> apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'
If they're not getting shorter, you're going the wrong way.
--
JBR with qualifications in linguistics, experience as a Debian
sysadmin, and probably no clue about this particular package
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
I actually forgot that bullseye itself introduces yet another one:
apt list ~obsolete
Apparently, those are also a thing:
comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep
'^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package:
(.*)/\1/p' | sort -u)
apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'
--
I've got to design so you can put it together out of garbage cans. In
part because that's what I started from, but mostly because I don’t
trust the industrial structure—they might decide to suppress us
weirdos and try to deny us the parts we need.
- Lee Felsenstein
Bug#987017: recommends 3 different ways to find obsolete packages, pick one
Package: release-notes
Severity: minor
The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
*different* ways of finding what are essential orphaned packages:
aptitude search '~o'
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
apt-forktracer | sort
Then I also know of those:
apt-show-versions | grep -v /bullseye
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
aptitude search
'?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
I frankly don't quite know where I stand with all this anymore, but I
am getting the strong feeling we're sending an incoherent message
here. :)
In my personal documentation, I've settled on `apt-forktracer`, but I
suspect we might want to stick with `aptitude search '~obsolete'`
because that matches other documentation in the release notes (and
allows for easy purging).
Is there any reason why we have all that diversity?
What's the right way to do what we actually want here?
-- System Information:
Debian Release: 10.9
APT prefers stable-debug
APT policy: (500, 'stable-debug'), (500, 'stable'), (1, 'experimental'), (1,
'unstable'), (1, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.19.0-16-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8),
LANGUAGE=fr_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
