I think I've managed to take everyone's comments into account, one way or
another. Comments appreciated by COB Tuesday.
Thanks,
Danek
======================================================================
Introduction and Motivation
---------------------------
An obsolete package is a version of a package which no longer delivers any
meaningful content, compared to previous versions. Marking a package
obsolete indicates that a repo is no longer in the business of delivering
any bits under that name.
Obsoletion can be used for two purposes: to coerce the removal of content
from the system because it is no longer supported, or applicable to the
system; and to reassociate a package to a new name. You can think of these
two cases as "content obsoletion" and "name obsoletion".
Because obsoletion is connected to a version, and each published version
can either be obsolete or not, it means that technically a package cannot
be obsolete, but a package version can be. When noting that a package
(package stem) is obsolete, a version or version range must be associated
with that information.
An obsolete package may be replaced by one or more packages; this is
treated as a rename (in the case of replacement by a single package), or a
split (in the case of multiple replacements).
At the moment, we obsolete a package by clearing out its content and
publishing that empty package so that on upgrade, all of its actions are
uninstalled from the system, even if the package itself isn't.
Package obsoletion has two primary motivations:
- User confusion when a package ceases to have content (but, due to
operational restrictions) continues to be published, empty. These
packages show up with zero bytes, no description or classification, but
are tempting to install by their names and the fact that they're
republished every build. So when nothing comes along with that
installation, it looks like a mistake on the distribution creator's
part, bugs are filed, etc.
- Ambiguity when a package is renamed to have the same substem as another
package (including possibly the original name). For instance, we want
to rename "netbeans" to "developer/netbeans", but because "pkg install
netbeans" can refer to either one, the install will fail because of the
ambiguity. If the old package name is republished as an obsolete
package, the system can resolve (or reduce) this ambiguity by only
considering the non-obsolete packages.
There are two related concepts, "toxic" and "historical", which will not be
covered here. Toxic packages are those which have been determined to have
such problems with them (catastrophic bugs, legal issues, etc) that they
need to be excised from the repository -- removed from the catalog, have
their manifests removed, and possibly even have all their (unique) files
removed. Historical packages are those which are so old that there's
little point in having them in the catalog, particularly since if catalog
is getting bloated with lots of package versions (from, say, a number of
years of a bi-weekly release schedule).
Marking a package toxic or historical are operations that are done after
publication, but must refer to a specific package version. Thus they
cannot be done by adding a package attribute to the manifest, but must be
separate operations. Therefore these operations can (and will) be deferred
to a future project.
Mechanism
---------
In the spirit of allowing the catalog to be merely a cache of data stored
in the manifest, a package will be marked obsolete by setting the package
attribute "pkg.obsolete" to "true" in the package manifest. Similarly, a
renamed package will have the attribute "pkg.renamed" to "true" in the
manifest, as well as one or more require-type dependencies.
However, since many of the operations that will need to know about the
obsolete state of a package shouldn't have to parse many manifests to get
at it, this state should be maintained in the catalog as well. This
implies that the publication server will need to parse the manifest more
intelligently than it does currently in order to extract this information.
No change to the catalog format or to the /catalog operation is required.
Instead of a "V" tag for an FMRI, an obsolete package version can be
marked as "O", with any renaming to be found as dependencies in its
manifest. The upcoming key/value-pair based catalog format should be able
to handle obsolete package simply as well, likely tagging the fmri with
"pkg.obsolete=true" or "pkg.renamed=true" just as in the manifest.
These attributes may be filtered by variant or facet tags, though this is
likely to be rare. However this *will* require a catalog format that takes
variants and filters into account.
Contents of an Obsolete Package
-------------------------------
Package rename (and split) is done by having dependencies in the package
pointing to the replacements. Thus the package is not completely empty, as
it may have depend actions.
Obsolete packages should also have metadata attached to them, allowing them
to describe themselves, or even have search terms associated with them.
Information about what the package was is still useful information, as long
as it's clear that this particular version is obsolete. This obviously
requires set actions to be delivered with the package.
No other data or metadata is required or desired for obsolete packages, so
no other action types should be allowed in an obsolete package. Note,
though, that a package could possibly be obsoleted for one variant, but not
another (on sparc, say, but not x86), so technically any given variant
should be empty except for depend and set actions, but not all variants may
be.
Packages marked with "pkg.renamed=true" must contain one or more
require-type depend actions. Packages marked with "pkg.obsolete=true" must
not contain any depend actions.
Checks should be performed at both publication and installation time that
these constraints hold true. It wouldn't be unreasonable for a client to
fail when installing such a package, as it's an indication of poor package
construction, and is likely a bug in same.
Behavior
--------
In the simple case, where a package is obsoleted without an associated
rename, and nothing on the system depends on it, the package will be
uninstalled on upgrade, and on initial install will emit a message, but not
fail the installation.
The state of the package will be "obsolete" in both "pkg list" and "pkg
info" output, though because at present "pkg info" fails when given a
package name that's not installed, "pkg info -r" will be required to get
this information.
When an obsolete package has dependencies, the dependencies will be
followed and installed as usual. The obsolete package will follow the
behavior it would if it didn't have dependencies (uninstall on upgrade, no
install on install).
A package can be constrained at an obsolete version.
A package can be unobsoleted (resurrected) by publishing a new version of
the package which doesn't have the pkg.obsolete attribute.
If an obsoleted package name is resurrected, then it is assumed that it
continues to satisfy dependencies as did versions prior to obsoletion.
This allows a mistaken rename to be corrected, but not for new content to
be delivered under an old name.
Behavior: Dependencies on Obsolete Packages
-------------------------------------------
A package can have a require-type dependency on an obsolete package, as
different distributions may skew in their understanding of the current
package namespace. For example, a blastwave package may depend on SUNWcsl,
even though SUNWcsl has been split up and renamed into many other packages.
Doing otherwise would be highly unfriendly.
In this case, the obsolete package should not be uninstalled, for
performance reasons, as the dependencies other packages have on it will
need to appear to be satisfied (via, say, "pkg verify"). This could be
made unnecessary with the new catalog format, but a different codepath for
dependency verification may not be desired. Some convergence will be
necessary here.
The case of a package depending on an obsolete package which itself has no
dependencies (ie, is EOLed) is more problematic. This means that the
functionality delivered by the obsoleted package has gone away entirely,
but another package is still presumably depending on that functionality.
In this case, the system should prevent either the upgrade to the obsolete
version of the package, or prevent the installation of the dependent
package (depending on the scenario at hand).
There are other, less palatable options to resolve the breakage:
- Uninstall the dependent packages. This is undesirable, since otherwise
desired functionality is removed from the system without the express
consent of the user.
- Mark the package obsolete, but don't uninstall it, or remove any of its
contents from the system until its dependents are removed. The
maintenance of this state, particularly over a longer period of time,
may be quite difficult. Still, aside from the management issues, it
may be the most user-friendly option.
If the system is capable of finding a non-obsolete version of the package
which satisfies all other system constraints, then this version should be
chosen instead. This will require the SAT solver to be available.
Other behaviors are enumerated in Test Cases, below.
Publication Changes
-------------------
As mentioned above, the obsolete attribute will need to make its way into
the catalog, so the publication server code will need to parse the manifest
looking for this attribute. Given the upcoming catalog format changes
putting dependency information into the catalog, this won't present any
extra burden on publication.
Given the problems with depending on an obsolete package, the server should
not allow such dependencies. This should be thought of as a distro-wide
(or at least repo-wide) flag day. In the case that a package is being
renamed, the dependent packages should be changing their depend actions to
match the new name, and in the case that a package is simply being EOLed,
any other package depending on it should simply be treated as a bug.
Any dependency checking needs to be careful not to fail a transaction if a
dependency can be satisfied with a non-obsolete package version, even if
the obsolete package version could otherwise satisfy it. Similarly, if
publication is being done within the context of an incorporation, so must
this dependency check.
- server should expand dependencies to unambiguous stems (or fail add?)
- the server might want to refuse publication of an obsolete package with
adjacent versions (no non-obsolete version "in between")
OpenSolaris Operational Changes
-------------------------------
re-publication of obsolete packages, inclusion in entire, etc.
Currently, we clear out a package's import file on obsoletion, leaving a
comment behind explaining what we've done and why. This will be replaced
by adding a new keyword for solaris.py to interpret: obsolete. It will
take no arguments, and it simply adds "set name=pkg.obsolete value=true" to
the manifest, while allowing only other set actions and depend statements
to be added. Note that because it won't be importing any SVr4 packages, we
will have to add by hand the metadata that would otherwise have been
populated from the pkginfo file (summary, description).
We also republish the empty packages each version after they've been
emptied. There's no particular reason for that now, and we should
definitely stop when we're publishing properly obsolete packages.
In addition, obsolete packages should not, generally, be constrained by
entire or any other large-scale incorporations. Doing so would be rather
unfriendly:
- if an unbundled package A depends on a WOS package B which gets
obsoleted in build X, then upgrading through build X would fail due to
this constraint (because A would no longer have its dependency
satisfied). Without the constraint, it would be allowed for the system
to stay on the latest non-obsolete version of the package (or at least
the one that was already installed on the system), which is probably
what the user wants. This wouldn't be the normal course, at least from
the first -- since package B is now unconstrained, it will be upgraded
to the latest, obsolete version, and so be removed anyway, but the
system could be enhanced later to allow it to remain on the system.
- If a leaf package got EOLed, a user might be annoyed that it was
removed from the system -- perhaps they'd rather keep the latest (or
installed) version around.
In general, doing the right thing for these situations involves
understanding user intent, which we do not currently capture.
In the case of an obsolete package undergoing a rename, the old package
name needs to be constrained to prevent the pre-obsoletion version and its
post-obsoletion replacements from being installed at the same time. This
could be done by any package normally incorporating these packages (such as
"entire"), but it may make more sense to have the replacements incorporate
(or place an optional dependency on) the old package name instead, as it
co-locates the potential problem and its solution. Which to use is
effectively a decision for the distribution to make.
Getting the correct version of the obsolete packages to get the right
replacements would depend on the SAT solver. If that isn't available, then
we could choose to continue incorporating (though at the original obsolete
version) the obsolete package only if it has replacements.
Test Cases
----------
Legend:
0 No package (transition from/to implies install/uninstall)
=> transition
A package "A"
Ao obsolete package "A"
A' new version of package "A"
A (-> B) A depends on B
A (<- B) B depends on A
1 install an obsolete package
0 => Ao
effectively a no-op (Ao is not on the system)
2 install a package with a dependency on an obsolete leaf package
0 => A (-> Bo)
fail
3 install a package with a dependency on a renamed package
0 => A (-> Bo (-> C))
A and C are installed, Bo is introduced as empty package for
bookkeeping
4 install an obsolete package with a dependency on another package (rename)
0 => Ao (-> B)
package B is installed, Ao is not on the system
5 upgrade a package to an obsolete version
A => Ao
package A is removed (Ao is not on the system)
6 upgrade a package to a version with a dependency on an obsolete leaf package
A => A' (-> Bo)
fail
A (-> B) => A' (-> Bo)
fail (or not; equivalent to 8b, see comment in 8a)
7 upgrade a package to a version with a dependency on a renamed package:
A => A' (-> Bo (-> C))
A' and C are installed, Bo is introduced as empty package for
bookkeeping
A (-> B) => A' (-> Bo (-> C))
A' and C are installed, B is emptied out and marked obsolete for
bookkeeping
A (-> B) => A (-> Bo (-> C))
[image-update only] C is installed, B is emptied out and marked
obsolete for bookkeeping
8 upgrade a package to an obsolete leaf version when another depends on it
A => Ao (<- B)
fail (maybe not: this could wedge an image-update when an unbundled
package depends on a package that's getting obsoleted)
A (<- B) => Ao (<- B')
ditto; equivalent to 6b
9 upgrade a package to an obsolete version with dependencies
A => Ao (-> B)
package B is installed, package A is removed (see 7[ab] for cases
where it would stick around)
10 upgrade a package to a version on which an obsolete package has a
dependency (this doesn't seem terribly interesting).
A => A' (<- Bo)
11 install an ambiguous name where only one match is non-obsolete
12 upgrade a package to an obsolete version with a dependency on a package
which has a (potentially) ambiguous name
13 Publishing an obsolete package with a dependency on an obsolete package
doesn't make sense -- fail publication?
A (-> B) => Ao (-> Bo)
Packages A and B are removed.
A (-> B (-> C)) => A' (-> Bo (-> Co))
Package A' is installed, package C is removed, package B is emptied
out and marked obsolete for bookkeeping
14 upgrade a package to an obsolete version, and then upgrade/install to a
resurrected version.
A => Ao => A'
A should be removed, and then added (by explicit install, not upgrade)
Specific Examples
-----------------
- system has "netbeans" on it; publish "developer/netbeans" and do an
"image-update" or "install netbeans". This will check both for the
netbeans switch, but also that the dependency libnb-xml switches over,
too. (11, 12)
- install a package that's going to be obsoleted (SUNWbrasero). publish
the obsolete version. image-update or re-install the package, and make
sure it goes away. (1, 5)
- install a package that's going to be obsoleted and replaced
(SUNWgnome-dtstart -> SUNWdesktop-startup). Publish the new set of
packages, image-update or re-install, and see that it goes away. (4, 9)
- install a package that's going to be obsoleted and a package that
depends on it. Publish the obsolete version. image-update or
re-install, and see that it fails. (Can't find an example!) (6)
- ditto, but the obsolete package has dependencies -- is getting renamed
(SUNWdbus-bindings -> SUNWdbus-glib+SUNWdbus-python24, SUNWcheese).
The update/install should succeed. (7)
- Make sure that SUNWfirefox-apoc-adapter goes from installable to
obsolete to installable. (EOFed in 95, resurrected in 116.)
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
