Bug#833482: [Aptitude-devel] Bug#833482: aptitude: doesn't detect obsolete candidate package (versions)

[Axel Beckert]

Hi,

while Manuel already lowered the severity and removed the security
tag, I also started to write a similar reply but haven't found time to
finish it.

As I have suggestions on how to do what you want just by configuring
aptitude differently, I'll still post my reply:

Christoph Anton Mitterer wrote:
> Aptitude doesn't seem to tell people when the candidate and/or installed 
> version
> of a package is obsolete.

I'm sorry, but that's not true. Obsolete packages show up in the
"Obsolete and Locally Created Packages" branch.

> - Debian seems to have removed the transcode package already back in March.
> - DMO still ships it however.

Then the package is not obsolete because it's downloadable. Please see
https://aptitude.alioth.debian.org/doc/en/ch02s04s05.html#searchObsolete
for what "obsolete" means in aptitude's context.

> - I do have the transcode package from Debian installed.
> - Via apt_preferences, all but a few packages from the DMO repos are 
> "disabled".
> 
> Thus I'd never get any candidate version from DMO, while aptitude still shows
> me the package not being obsolete.

I don't see the problem. That's what you configured, so that's what
you get.

> In a way, of course, it is not fully obsolete,

Exactly.

> but it will never get any updates thus no security updates either.

Yes, because you configured it to do so. If you wouldn't have added
the DMO repos, it would clearly show up in the "Obsolete and Locally
Created Packages" branch.

> This is also what I think makes this issue important/security:
> One ends up in a situation where the use will neither get updates (cause it's 
> no
> longer in Debian), nor will he even notice that this is the case (not being
> showed as obsolete).

I strongly disagree here. You ask for "A" despite your configuration
says "B".

BTW: You can get all those package if you use "aptitude search" or the
TUI's "limit" command with "~i ?any-version(!~O.) !~U !~o". This
matches both, packages newer than in the archive as well as packages
with newer versions available, but pinned down.

You can even use this pattern in the configurable grouping method for
package views (Aptitude::UI::Default-Grouping). So if you want this in
the TUI, it's already possible: Just prepend "pattern(~i
?any-version(!~O.) !~U !~o => Non-upgradable packages not from
archive, ?true ||)," to Aptitude::UI::Default-Grouping (i.e. before
"status") and then you get a new branch in the TUI which shows only
these packages. (Might need some more finetuning for further
corner-cases but works for me as commandline alias well enough for
years now.)

We might consider adding such a branch to aptitude's TUI, but at least
the implementation above is rather slow, so unless we find a more
efficient pattern or some other more direct way, I'd not add that. (So
Manuel's "wontfix" tag is correct for the time being.)

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#833482: [Aptitude-devel] Bug#833482: aptitude: doesn't detect obsolete candidate package (versions)

[Manuel A. Fernandez Montecelo]

Hi,

2016-08-06 2:13 GMT+01:00 Christoph Anton Mitterer :
> Mhh... a bit further below in the text, aside from all the discussion
> about whether this is supported or not,... I might have found some way
> to go how aptitude could easily (at least semantically) identify
> packages which are obsolete in the sense of: will this package (in the
> current apt_preferences/sources.list config) still receive updates
> (assuming that the repos themselves are still maintained) or not.

I can only guess, but I think that the original motivation is that
when you use stable or unstable, if some package is classified as
Obsolete, is a reason to take special care about it, and so you can
freely delete it if you don't care; or if you care, add it again to
the archive (if was orphaned and deleted because it was an old version
with low popularity but it's OK otherwise).

This probably predates auto-installed, so it must have been a big deal
back then to clear cruft.  This doesn't mean that you don't have to
take an eye on other packages that you mix and match from different
sources.

So Obsolete/Local doesn't have any relation to "how secure is to use a
package or not".  It's "whether it's still downloadable from
somewhere" or "I could not find this package in any repo, but it's
installed in this system".  This maps quite direcly to other
functionalities, like "the changelogs are only available locally" or
"there are new changelogs in the server that the user might want to
see before installing".

Also, there's the peculiarity of testing, when packages are
auto-removed and then pop back again into life, in the last few years.
Each suite has its own peculiarities.  Obsolete is pretty useless for
this case of auto-removed packages.


On the topic of security, you can have installed v1.3.1~rc1 of a
package from experimental with dangerous bugs or security issues, and
v1.2.1 from unstable will not be installed (in most common
configurations), despite the risk for your system.  Obsolete in
aptitude and security don't have much to do with each other, it's only
an artifact of mixing distros and only one of a myriad ways in which
things can go obfuscated in terms of security when playing with
different distros, pinning, etc.


>> (there are similar warnings about mixing different Debian
>> distributions/suites except in completely compatible overlays, e.g.
>> "security")
>
> Well than why not hardcoding all these repos, and no longer allowing
> people to configure them? Why not dropping apt_preferences if it's
> anyway "not supported" and can just cause insecure situations?

Beause in Debian we really like to give the option of shooting
themselves in the foot if they really insist on it.  Freedom and all
that.

We're not so keen on babysitting the users and monitor them with
twelve drones and raise an alarm in the police every time that they
step into a pond, which might be small pothole in the roadside or
might be a big lake.

Basically the contributors of Debian strive to support the most common
use cases, and it's hard enough, and a good enough gift to society, if
you ask me.

If people want to go playing around with our tools it's fine, but we
don't want to be healing them or paying the healthcare after hurting
themselves for using our tools in a way that we explicitly say that we
don't want to support, even if they go handwaving all around.


> In the end, what's really interesting about obsolete packages is:
> package foo won't get any further security updates. If you continue to
> use it, you're on your own.
>
> This information is currently not available to aptitude users, when
> they do *anything* that uses multiple repos, except those which are
> fully aligned (e.g. stable + security updates).

Since stable is the only suite that has guaranteed security updates,
all of your considerations before are not relevant.  unstable doesn't
have security support, testing doesn't either, external repos can
interfere with debian repos in ways that they take over, etc.

So your only relevant use-case for security is when using
stable+security, and there Obsoletes works well.


> And as I've said just above... if one doesn't care about security
> updates, than having the information whether something is obsolete or
> not is pretty pointless. Either one hasn't installed it and then it's
> simply gone... or one has it installed and can just happily continue to
> use it.

Actually, the way in which I have seen people to use Obsolete is
mainly to remove cruft, of packages that are no longer needed but are
marked as manually installed for some reason (bugs in aptitude/apt/etc
or not).

I never heard of people worried about obsolete packages in terms of
security, unless they are daemons or something special (often after
stable updated to next-stable).


Also, there's the peculiarity of testing, when packages are
auto-removed and then pop back again into life, in the last few years.
Each suite has its own 

Bug#833482: [Aptitude-devel] Bug#833482: aptitude: doesn't detect obsolete candidate package (versions)

[Christoph Anton Mitterer]

Mhh... a bit further below in the text, aside from all the discussion
about whether this is supported or not,... I might have found some way
to go how aptitude could easily (at least semantically) identify
packages which are obsolete in the sense of: will this package (in the
current apt_preferences/sources.list config) still receive updates
(assuming that the repos themselves are still maintained) or not.

Not sure how difficult it would be to implement the abstract criterion
into actual code, though ;)

Anyway, if you're as tired about the political discussion whether
things is de jure / de facto supported or not, just skip on to (*).

If it would be easy to implement, you could either replace the current
"obsolete packages with it" (as I try to motivate below, it doesn't
make that much sense in the current form), or perhaps add it as a new
category, "packages that don't receive updates in the current config".



On Sat, 2016-08-06 at 01:12 +0100, Manuel A. Fernandez Montecelo wrote:
> The underlying problem is that you think that using Debian + external
> repositories which don't play well with Debian is a supported
> scenario.

Well "supported" is always a relative thing. It may not be supported in
the sense that someone from Debian feels officially responsible, but
it's actively and actually quite powerfully supported on the technical
side starting with sources.list allowing for multiple repos up to e.g.
the mechanisms of apt_preferences.

So why not making these situations safe, simply by properly telling
users when their packages won't receive anymore updates?

It's a bit cheeky to have all these mechanisms in Debian, actually
advertising them in the documentation, for quite a while even depending
on them (e.g. when DMO was the only resort to get usable multimedia
capabilities into Debian),... and on the same time saying: "nope, don't
want to properly (technically) support these functionalities).

At least I found no reference in the sources.list, apt_preferences or
aptitude manpages that things will be insecure in certain situations,
when multiple repos are used.



> "In most situations if you stick with one distribution you should use
> that and not mix packages from other distributions. Many common
> breakages arise due to people running a distribution and trying to
> install Debian packages from other distributions. The fact that they
> use the same formatting and name (.deb) does not make them
> inmediately
> compatible."

This quite clearly refers rather to the problems resulting from
dependencies than to package management software not properly detecting
situations in which software is really obsolete.


> (there are similar warnings about mixing different Debian
> distributions/suites except in completely compatible overlays, e.g.
> "security")

Well than why not hardcoding all these repos, and no longer allowing
people to configure them? Why not dropping apt_preferences if it's
anyway "not supported" and can just cause insecure situations?


> If DMO had transcode as a well supported package (perhaps packaging a
> maintained fork while keeping the same name, for example), it
> wouldn't
> be a security problem at all, you could switch to it instead.
Not if one has pinned it down via apt_preferences before.
Again, I really think that the question of whether DMO is well
[security]supported or not is completely out of question here.
And there are so many security questionable things in Debian, that it's
probably not appropriate to point at other "projects", either.


>   From
> aptitude's POV, it's not "obsolete", because it's present in other
> repo.
And as I've said, that's the problem.


Think about it this way:

What is the motivation behind having the knowledge "whether a package
is obsolete or not"?

If one hasn't installed it in the first place, then knowing that
package foo is now obsolete is pretty much useless from the package
management PoV.
Sure one can see: Debian has had this in the past, but not longer.
So what?

If one has it installed, what's the use of seeing that it's now
obsolete? I'd say either:
- To see that one can remove it safely in case of dependencies cannot
longer be fulfilled because of it.
- To notice that this is basically out-of-Debian-warranty (i.e no more
  updates - whether security or not).

The former case is probably largely irrelevant, either other packages
Conflict/Break/Replace/etc. the obsolete package already (and then I
don't need the info that's is obsolete) or they don't (and then I could
just keep the obsolete package if I don't care for security/new
versions).

So IMO the important case if the 2nd. Now just not getting new upstream
versions (with new features) is perhaps nice to know, but not knowing
it doesn't hurt one much either. If new features are missed people will
search and find out whether package foo is upstream-dead or just gone
from Debian.

In the end, what's really interesting about obsolete packages is:
package foo won't get any 

Bug#833482: [Aptitude-devel] Bug#833482: aptitude: doesn't detect obsolete candidate package (versions)

[Manuel A. Fernandez Montecelo]

Control: tags -1 - security
Control: close -1


2016-08-05 23:06 GMT+01:00 Christoph Anton Mitterer :
> Control: reopen -1
> Control: tags -1 + security
>
> On Fri, 2016-08-05 at 22:49 +0100, Manuel A. Fernandez Montecelo wrote:
>> This effect is an interference caused by the repositories that you
>> use.
>> Debian doesn't sanction the use of any unofficial repositories, and
>> DMO
>> is well known in the community for causing all sorts of problems when
>> using along with the main Debian repositories, such as this one.
>
> So? The underlying problem exists whether or not DMO is well
> maintained.
> If any non-Debian repo is configured that uses the same package names
> (and I don't think that Debian claims namespace authority of all names)
> than it wouldn't be detected if the candidate versions are obsolete.

The underlying problem is that you think that using Debian + external
repositories which don't play well with Debian is a supported
scenario.

From: http://www.debian.org/doc/manuals/debian-faq/ch-choosing.en.html

"In most situations if you stick with one distribution you should use
that and not mix packages from other distributions. Many common
breakages arise due to people running a distribution and trying to
install Debian packages from other distributions. The fact that they
use the same formatting and name (.deb) does not make them inmediately
compatible."

(there are similar warnings about mixing different Debian
distributions/suites except in completely compatible overlays, e.g.
"security")


Using repos which cause namespace clashes and the "overlay" doesn't
have the "base" into account is prone to this kind of situation.  It
also happens with backports sometimes (see below).

If DMO had transcode as a well supported package (perhaps packaging a
maintained fork while keeping the same name, for example), it wouldn't
be a security problem at all, you could switch to it instead.  From
aptitude's POV, it's not "obsolete", because it's present in other
repo.  It doesn't even know where it came from, you could have
installed that package/version locally with dpkg.

The fact that it was removed from unstable is not an indication that
it's a security problem either.  It's only "obsolete" or a "security
problem" from your POV, because you know extra information that
aptitude cannot know.

The only solution for this particular problem is DMO stopping to ship
this package, if it indeed has security problems; and for this and the
more general problem of packages dropping from repositories while
present in others is you to monitor such packages, with the tools that
aptitude gives to you (and Obsolete is the wrong tool in this case).


It's not a security problem related to Debian or aptitude in any case,
and neither aptitude maintainers or the security team will be able to
do anything about this, so it's not actionable as a security issue.


>> If the example was with another repository which is well maintained
>> and
>> co-operative, e.g. "mozilla.debian.net" containing a package
>> "iceweasel"
>> for compatibility (which was removed from Debian), the package
>> shouldn't
>> be considered obsolete.
>
> The problem happens with these as well:
> Consider I have had transcode installed from sid (where it's been gone
> in the meantime) and also enabled stable (where it's still in), I'd
> expect the same as I've seen it with DMO.

Mixing stable and unstable is not supported either, precisely because
it leads to this kind of problem.


>> Obsolete from aptitude is "installed locally but not found in any
>> repo",
>> which works well for the main intended uses of aptitude.
> Well and that's the problem here, the definition of obsolete is kinda
> wrong.
> Or perhaps not wrong, but it doesn't tell you when a package no longer
> will get updates (which is however the interesting thing about
> obsolete/non-obsolete).

We already discussed this in previous bugs.

aptitude, as many other Debian tools, is designed with the main idea
in mind that users will have supported versions, e.g. only stable,
unstable, and so on.  In these cases, "obsoletes" works well and
always as expected.

Anything else, you're on your own.


>> So it's not an important bug, and aptitude is not the cause of the
>> security issue -- using DMO is.
> I don't think DMO has anything to do with...

Well, yes, to be fair DMO hasn't anything to do with it... again, the
actual problem is that you think that you can use both unstable and
DMO and report bugs as if it was a supported scenario ;)


> The "bug" or let's call it "deficiency" or "missing feature" in
> aptitude is:
> It doesn't show when packages will no longer get updates via the
> configured repos and as per configured apt_preferences.
>
> This is however quite important, as it also happens when just mixing
> Debian repos.
> Take backports... if one adds jessie-backports, and e.g. installs
> icinga2 from there, perhaps pinned via apt_preferences(8).
> It's still 

Bug#833482: [Aptitude-devel] Bug#833482: aptitude: doesn't detect obsolete candidate package (versions)

[Christoph Anton Mitterer]

Control: reopen -1
Control: tags -1 + security

On Fri, 2016-08-05 at 22:49 +0100, Manuel A. Fernandez Montecelo wrote:
> This effect is an interference caused by the repositories that you
> use.
> Debian doesn't sanction the use of any unofficial repositories, and
> DMO
> is well known in the community for causing all sorts of problems when
> using along with the main Debian repositories, such as this one.

So? The underlying problem exists whether or not DMO is well
maintained.
If any non-Debian repo is configured that uses the same package names
(and I don't think that Debian claims namespace authority of all names)
than it wouldn't be detected if the candidate versions are obsolete.


> If the example was with another repository which is well maintained
> and
> co-operative, e.g. "mozilla.debian.net" containing a package
> "iceweasel"
> for compatibility (which was removed from Debian), the package
> shouldn't
> be considered obsolete.

The problem happens with these as well:
Consider I have had transcode installed from sid (where it's been gone
in the meantime) and also enabled stable (where it's still in), I'd
expect the same as I've seen it with DMO.



> Obsolete from aptitude is "installed locally but not found in any
> repo",
> which works well for the main intended uses of aptitude.
Well and that's the problem here, the definition of obsolete is kinda
wrong.
Or perhaps not wrong, but it doesn't tell you when a package no longer
will get updates (which is however the interesting thing about
obsolete/non-obsolete).


> So it's not an important bug, and aptitude is not the cause of the
> security issue -- using DMO is.
I don't think DMO has anything to do with...

> In a deeper sense, the package is dead upstream, thus not maintained,
> thus obsolete and a potential security liability, and that's the
> reason
> why it was removed from Debian.
But the problem will happen with any other package, whether it's
maintained or not,...


The "bug" or let's call it "deficiency" or "missing feature" in
aptitude is:
It doesn't show when packages will no longer get updates via the
configured repos and as per configured apt_preferences.

This is however quite important, as it also happens when just mixing
Debian repos.
Take backports... if one adds jessie-backports, and e.g. installs
icinga2 from there, perhaps pinned via apt_preferences(8).
It's still contained in jessie (at a much lower version), if it would
now be removed from backports, because lack of manpower or whatever,
users wouldn't notice that their installed version is obsolete in the
sense that it won't get any further security updates.

Since this has nothing to with DMO, reopening.

Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Bug#833482: [Aptitude-devel] Bug#833482: aptitude: doesn't detect obsolete candidate package (versions)

[Manuel A. Fernandez Montecelo]

Control: severity -1 wishlist
Control: tags -1 - security + wontfix
Control: close -1


Hi,

2016-08-02 15:00 Christoph Anton Mitterer:

Package: aptitude
Version: 0.8.2-1
Severity: important
Tags: security


Hi.

I've just stumbled over the following:
Aptitude doesn't seem to tell people when the candidate and/or installed version
of a package is obsolete.

Example:
- Debian seems to have removed the transcode package already back in March.
- DMO still ships it however.
- I do have the transcode package from Debian installed.
- Via apt_preferences, all but a few packages from the DMO repos are "disabled".

Thus I'd never get any candidate version from DMO, while aptitude still shows
me the package not being obsolete.
In a way, of course, it is not fully obsolete, but it will never get any updates
thus no security updates either.

This is also what I think makes this issue important/security:
One ends up in a situation where the use will neither get updates (cause it's no
longer in Debian), nor will he even notice that this is the case (not being
showed as obsolete).


This effect is an interference caused by the repositories that you use.
Debian doesn't sanction the use of any unofficial repositories, and DMO
is well known in the community for causing all sorts of problems when
using along with the main Debian repositories, such as this one.

Among others, it uses packages with a higher epoch so they always take
precedence over Debian's, even if it's 4:1.2.0 versus 3:1.2.1 or
1:1.4.0.

So it's not aptitude's fault if it's fed with bogus data/information for
the repo, really, and the repository tries actively to screw with the
official Debian packages and play with the versioning system in ways
that cause this kind of problems.

If the example was with another repository which is well maintained and
co-operative, e.g. "mozilla.debian.net" containing a package "iceweasel"
for compatibility (which was removed from Debian), the package shouldn't
be considered obsolete.

Obsolete from aptitude is "installed locally but not found in any repo",
which works well for the main intended uses of aptitude.


So it's not an important bug, and aptitude is not the cause of the
security issue -- using DMO is.

If it's for a matter of security, that repository shouldn't be used at
all, so merely installing stuff from it is a big security liability
compared to Debian and many other well maintained repos.

In a deeper sense, the package is dead upstream, thus not maintained,
thus obsolete and a potential security liability, and that's the reason
why it was removed from Debian.


Cheers.
--
Manuel A. Fernandez Montecelo