Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[David Kalnischkies]

On Sat, Jun 25, 2016 at 12:04:58PM -0700, Josh Triplett wrote:
> > See https://wiki.debian.org/HelmutGrohne/MultiarchSpecChanges (and
> > links) for more examples, potential solutions and problems/shortcomings
> > from each of them.
> 
> The "Install-Same-As" header on that page looks ideal for this case.

The devil is in the details, e.g.: "It is only considered installed if
it is installed for all architectures that the listed package is
installed for" which translates roughly to "implement it in ALL tools,
wait for these tools to be released as Debian stable, use it in
stable+1" as "considered installed" is a dpkg thing. It also forbids the
use of "magic" as hat doesn't play nice with hard rules about package
installation states.


> Do you know if anyone has looked into what it would take to implement
> Install-Same-As in apt?

I haven't and so I somehow doubt anyone else has, but who knows…


The problem with a possible implementation is that it is in effect
a "conditional auto-install enhances" and conditions are hard to
implement. Theoretically it is possible to implement that type of
conditions with a bunch of virtual packages as a workaround, but in
practice apts dependency resolver will not like it, for roughly the same
reasons it isn't as good as aptitude in solving sudoku puzzles (not that
aptitude will invite you with open arms either, it will just not slam
the door in your face).  So, an implementation effecting all libapt
users instantly via a workaround I would consider highly unlikely. If we
talk implementing conditions for real, we are in the flying pigs for
human transport department.

That would leave us with adding it as "magic" somewhere between
apt(-get) and libapt which depending on where it is placed exactly will
effect more or less other libapt users which is very rougly proportional
to the complexity of implementing the magic in that place. And in all
the other places elsewhere to catch the rest of the fish.


Disclaimer: This is just an educated guess. I could be entirely wrong.
It would be more predictable if more than a few rough ideas existed.
Cornercases like upgrading existing systems, opt-in/out configuration
and syntax, if multiple packages are mentioned – assuming that is legal
– is it an OR or an AND and if the later, does OR exist as | then?, is
it a versioned relation, keeping API and/or ABI, what if v2 of a package
adds/modifies/removes the field, interaction with autoremove………


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Josh Triplett]

David Kalnischkies wrote:
> On Sat, Jun 25, 2016 at 02:01:27AM -0700, Josh Triplett wrote:
> > Sven Joachim wrote:
> > > On 2016-06-24 23:01 -0700, Josh Triplett wrote:
> > > > Some packages, if installed on any architecture, must be installed for
> > > > every enabled architecture.  Most notably, an NSS or PAM module package,
> > > > if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
> > > > for every enabled architecture to avoid breaking programs for that
> > > > architecture.
> 
> See https://wiki.debian.org/HelmutGrohne/MultiarchSpecChanges (and
> links) for more examples, potential solutions and problems/shortcomings
> from each of them.

The "Install-Same-As" header on that page looks ideal for this case.

The bootstrap issue would certainly need a solution, but it doesn't seem
too hard to have an option/configuration to ignore such dependencies.
Someone installing a specialized build chroot might want that options as
well.  For instance, a build chroot could install an
/etc/apt/apt.conf.d/* file containing:

Ignore-Install-Same-As: libc6, libpam0g

I think that largely addresses the other cons as well.

Do you know if anyone has looked into what it would take to implement
Install-Same-As in apt?

- Josh Triplett

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Ian Jackson]

Josh Triplett writes ("Re: Handling Multi-Arch packages that must be installed 
for every enabled architecture?"):
> On Sat, Jun 25, 2016 at 07:08:39PM +0100, Ian Jackson wrote:
> > * LD_PRELOAD hacks need their .so installing for all architecture "for
> >   which they are going to be used" (whatever that means)
> 
> Such as fakeroot and fakechroot?

Right, faketime too.  I maintain one: authbind.

> > * The language plugin problem.
> 
> Can you elaborate on this?  Do you mean languages that have native-code
> plugins?

Yes.

Example:

sauce [all] contains pure Tcl code.

It has a #! line naming a specific Tcl interpreter (let's say 8.5, for
now).  So it Depends tcl8.5 (or whatever).  tcl8.5 is (or should be)
M-A foreign.  It contains the Tcl script interpreter executable.

sauce uses adns to do its primary DNS queries.  This is done via a Tcl
plugin in the chiark-tcl package.  sauce loads this with the Tcl
"load" builtin, which loads a named shared object from the library
search path and calls a specific init function.

So sauce Depends libtcl-chiark-1.

libtcl-chiark-1 is Arch: any.  It contains only shared objects.  It
needs to be coinstallable.  It should be M-A: same.

Somehow we need to arrange that if sauce:all's dependency on tcl8.5
is satisfied by tcl8.5:X, and libtcl-chiark-1:X is installed.

Ian.

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Josh Triplett]

Simon McVittie wrote:
> On Fri, 24 Jun 2016 at 23:01:21 -0700, Josh Triplett wrote:
> > Some packages, if installed on any architecture, must be installed for
> > every enabled architecture.  Most notably, an NSS or PAM module package,
> > if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
> > for every enabled architecture to avoid breaking programs for that
> > architecture.
>
> This might well be true for PAM. For NSS, it used to be true, but based
> on work I did on nss-mdns a while ago, I'm reasonably sure that was a
> glibc bug that was later fixed. (See
> .)

Glad to hear that the NSS case has gotten a bit better.  But that still
means applications for an architecture without the configured NSS module
installed can't resolve names in the expected way.  Effectively, any NSS
module not shipped with glibc won't work reliably across all
applications without a solution to this problem.

> > As one possible solution for this problem (but not an ideal one, just a
> > thought experiment), dpkg could support a new value for "Multi-Arch",
> > "Multi-Arch: every".
> 
> A flag to indicate "keep this in sync across architectures" seems a
> good idea, but I think this is perhaps a job for apt rather than dpkg
> (and apt will need to understand it anyway). Using a separate field
> would avoid the need to add the field to dpkg, wait for a new stable,
> and only then start adding it to packages.

Good point; if dpkg and apt will both ignore any field they don't
understand, then this doesn't need to wait for a stable cycle.

> Perhaps something more like this? (I don't really like the field name,
> but it's the best I could think of right now.)
> 
> Package: libpam-whatever
> Multi-arch: same
> Follow-architectures-of: libpam0g
>
> which would mean: if you install libpam-whatever:amd64, you have
> amd64, i386 and armhf architectures enabled in dpkg, and you have
> libpam0g:amd64 and libpam0g:i386 but not libpam0g:armhf, then
> libpam-whatever:i386 (only) is also automatically selected for
> installation; and if you subsequently enable arm64 and install
> libpam0g:arm64, then libpam-whatever:arm64 is also
> automatically selected?

That sounds perfect.

NSS modules would then "Follow-Arch-Of: libc6".  And PAM modules would
"Follow-Arch-Of: libpam0g".

Looking at the link provided elsewhere in the thread, this looks similar
to the "Install-Same-As" header suggested on
https://wiki.debian.org/HelmutGrohne/MultiarchSpecChanges .  Of the
proposals on that page, this seems like the most viable.

- Josh Triplett

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Josh Triplett]

On Sat, Jun 25, 2016 at 07:08:39PM +0100, Ian Jackson wrote:
> Josh Triplett writes ("Handling Multi-Arch packages that must be installed 
> for every enabled architecture?"):
> > That would solve the problem for the couple of cases it has come up in,
> > but it seems far from ideal; I'd welcome an cleaner alternative
> > solution.  Notably, this doesn't work well for plugin packages for
> > libraries less critical than glibc; it's not even ideal for PAM, as not
> > every enabled architecture will have packages depending on libpam0g.
> > The real dependency is "if any package on the architecture depends on
> > package X, and package Y is installed, package Y:arch must be
> > installed", but that's excessively complicated.
> > 
> > Any ideas on how to solve this problem?
> 
> I don't have really good suggestions, but I wanted to mention two
> additional problems which are very nearby:
> 
> * LD_PRELOAD hacks need their .so installing for all architecture "for
>   which they are going to be used" (whatever that means)

Such as fakeroot and fakechroot?

> * The language plugin problem.

Can you elaborate on this?  Do you mean languages that have native-code
plugins?

- Josh Triplett

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Ian Jackson]

Josh Triplett writes ("Handling Multi-Arch packages that must be installed for 
every enabled architecture?"):
> That would solve the problem for the couple of cases it has come up in,
> but it seems far from ideal; I'd welcome an cleaner alternative
> solution.  Notably, this doesn't work well for plugin packages for
> libraries less critical than glibc; it's not even ideal for PAM, as not
> every enabled architecture will have packages depending on libpam0g.
> The real dependency is "if any package on the architecture depends on
> package X, and package Y is installed, package Y:arch must be
> installed", but that's excessively complicated.
> 
> Any ideas on how to solve this problem?

I don't have really good suggestions, but I wanted to mention two
additional problems which are very nearby:

* LD_PRELOAD hacks need their .so installing for all architecture "for
  which they are going to be used" (whatever that means)

* The language plugin problem.

Ian.

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[David Kalnischkies]

On Sat, Jun 25, 2016 at 02:01:27AM -0700, Josh Triplett wrote:
> Sven Joachim wrote:
> > On 2016-06-24 23:01 -0700, Josh Triplett wrote:
> > > Some packages, if installed on any architecture, must be installed for
> > > every enabled architecture.  Most notably, an NSS or PAM module package,
> > > if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
> > > for every enabled architecture to avoid breaking programs for that
> > > architecture.

See https://wiki.debian.org/HelmutGrohne/MultiarchSpecChanges (and
links) for more examples, potential solutions and problems/shortcomings
from each of them.

So, as usual, what looks like a simple and easy thing is actually
a complexity monster eating little kids^W DDs for breakfast.

(That isn't to say stop all discussion… this discussion died down and in
case we still need a solution it should 'just' be resumed from the last
good state rather than from zero)

> > > As one possible solution for this problem (but not an ideal one, just a
> > > thought experiment), dpkg could support a new value for "Multi-Arch",
> > > "Multi-Arch: every".  This value would imply "Multi-Arch: same", but if
> > > installed, would additionally cause dpkg to act the same way it does for
> > > Essential packages: install the package when enabling the architecture.
> > 
> > This is not at all what dpkg does, the Essential flag only means that
> > dpkg will not remove the package in question, unless given the
> > --force-remove-essential switch.
> 
> It's been a while since I'd set up "dpkg --add-architecture i386" on a new
> system, so I'd misremembered.  I had thought that doing so (or subsequently
> installing an i386 package) would force the installation of Essential packages
> for i386 for any package that was "Multi-Arch: same".  Apparently not.

Nitpick, but there are no essential M-A:same packages (first, because
libraries are forbidden from being essential and M-A:same applies mostly
only to them and second, a package tends to be essential because it
ships an architecture-independent commandline interface – hence most of
them are M-A:foreign – so important that noone can be bothered to depend
on it).


> > > (And when installing the package, dpkg would need to require
> > > it for every supported architecture; dpkg could refuse to configure the
> > > package if any enabled architecture doesn't have it unpacked.)
> > 
> > One problem here is that dpkg does not even know which packages are
> > available. […]

Another nitpick, but dpkg does know (to a certain extend). That isn't
all to important in the suggested case here as if you would model it as
a hard-requirement as suggested here with configure-refusal or later
with an automatic deconfiguration the package would need to exist for
all architectures anyhow as if you allow it to be not available for one
it isn't a hard-requirement anymore, but a magical recommends depending
on which sources a user has currently configured (and happend to be
successful in downloading).

Such a requirement also prevents packages from adding/removing
architectures (probably very very uncommon) and makes the interaction
all around pretty strange: I add a new architecture via dpkg and
instantly ever package manager screams at me that my system is broken.
But that is already written in the wiki…


> > I think such problems are better solved in apt: apt-get dist-upgrade
> > already reinstalls every Essential package, the same way it could ensure
> 
> That sounds quite reasonable to me.  The question then becomes how apt

It does to you and me perhaps, but apts handling of essentials (which
aptitude has copied and extended to prio:required in recent versions) is
the source of constant complains.

With every magic implemented in apt it should also be considered what
that means for all non-src:apt package managers be them libapt-based or
not as for better or worse apt(-get) is by far not the only thing
dealing with packages.


For Multi-Arch itself I managed to hide away most of it behind implicit
dependency relations, versioned provides and 'strange' virtual packages
for the libapt-based ones which made that transition quite "easy" all
things considered, but we can't pretend it will always be that "easy"…


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Josh Triplett]

On Sat, Jun 25, 2016 at 09:37:37AM +0200, Ben Hutchings wrote:
> On Fri, 2016-06-24 at 23:01 -0700, Josh Triplett wrote:
> > Some packages, if installed on any architecture, must be installed for
> > every enabled architecture.  Most notably, an NSS or PAM module package,
> > if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
> > for every enabled architecture to avoid breaking programs for that
> > architecture.
> 
> What if I enable an architecture only for cross-building and don't
> intend to run programs from that architecture?

How much harm does it do to have (for instance) libnss-foo installed on
that architecture?  By comparison, how much harm does it do to *not*
have libnss-foo installed for an architecture but have it configured in
/etc/nsswitch.conf because you have libnss-foo installed for your main
system architecture?

> > As one possible solution for this problem (but not an ideal one, just a
> > thought experiment), dpkg could support a new value for "Multi-Arch",
> > "Multi-Arch: every".  This value would imply "Multi-Arch: same", but if
> > installed, would additionally cause dpkg to act the same way it does for
> > Essential packages: install the package when enabling the architecture.
> > (And when installing the package, dpkg would need to require installing
> > it for every supported architecture; dpkg could refuse to configure the
> > package if any enabled architecture doesn't have it unpacked.)
> > 
> > That would solve the problem for the couple of cases it has come up in,
> > but it seems far from ideal; I'd welcome an cleaner alternative
> > solution.  Notably, this doesn't work well for plugin packages for
> > libraries less critical than glibc; it's not even ideal for PAM, as not
> > every enabled architecture will have packages depending on libpam0g.
> > The real dependency is "if any package on the architecture depends on
> > package X, and package Y is installed, package Y:arch must be
> > installed", but that's excessively complicated.
> 
> I don't think we can ever say there is a hard dependency here.
> Instead of adding another Multi-Arch value, how about adding
> 'Recommends: Y:every' to Y?  (This also moves the problem up from dpkg
> to APT.)

Recommends doesn't seem like the appropriate dependency relationship for
a package required to run programs.  Choosing not to install Recommends
can produce suboptimal behavior, but "programs fail to run" seems
excessive for not installing a Recommends.

Y could "Depends: Y:every", though; that would solve the problem, and as
you said, raise it from dpkg to apt.  Not perfect, as you don't (for
instance) need libpam-foo:somearch installed unless you have
libpam0g:somearch installed, but better than breaking.

Seems like the ideal solution would be (assuming a new virtual package
"pam-module" that PAM module packages provided) some dependency field in
libpam0g like "Depends: pam-module:same-if-any".  That would act like
"Depends: pam-module:same-as-libpam0g" if the system had pam-module
installed for any architecture.  I don't know how much complexity that
would introduce, though.

- Josh Triplett

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Ben Hutchings]

On Fri, 2016-06-24 at 23:01 -0700, Josh Triplett wrote:
> Some packages, if installed on any architecture, must be installed for
> every enabled architecture.  Most notably, an NSS or PAM module package,
> if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
> for every enabled architecture to avoid breaking programs for that
> architecture.

What if I enable an architecture only for cross-building and don't
intend to run programs from that architecture?

> As one possible solution for this problem (but not an ideal one, just a
> thought experiment), dpkg could support a new value for "Multi-Arch",
> "Multi-Arch: every".  This value would imply "Multi-Arch: same", but if
> installed, would additionally cause dpkg to act the same way it does for
> Essential packages: install the package when enabling the architecture.
> (And when installing the package, dpkg would need to require installing
> it for every supported architecture; dpkg could refuse to configure the
> package if any enabled architecture doesn't have it unpacked.)
> 
> That would solve the problem for the couple of cases it has come up in,
> but it seems far from ideal; I'd welcome an cleaner alternative
> solution.  Notably, this doesn't work well for plugin packages for
> libraries less critical than glibc; it's not even ideal for PAM, as not
> every enabled architecture will have packages depending on libpam0g.
> The real dependency is "if any package on the architecture depends on
> package X, and package Y is installed, package Y:arch must be
> installed", but that's excessively complicated.

I don't think we can ever say there is a hard dependency here.
Instead of adding another Multi-Arch value, how about adding
'Recommends: Y:every' to Y?  (This also moves the problem up from dpkg
to APT.)

Ben.

> Any ideas on how to solve this problem?

-- 

Ben Hutchings
Reality is just a crutch for people who can't handle science fiction.


signature.asc
Description: This is a digitally signed message part

Re: Handling Multi-Arch packages that must be installed for every enabled architecture?

[Sven Joachim]

On 2016-06-24 23:01 -0700, Josh Triplett wrote:

> Some packages, if installed on any architecture, must be installed for
> every enabled architecture.  Most notably, an NSS or PAM module package,
> if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
> for every enabled architecture to avoid breaking programs for that
> architecture.
>
> As one possible solution for this problem (but not an ideal one, just a
> thought experiment), dpkg could support a new value for "Multi-Arch",
> "Multi-Arch: every".  This value would imply "Multi-Arch: same", but if
> installed, would additionally cause dpkg to act the same way it does for
> Essential packages: install the package when enabling the architecture.

This is not at all what dpkg does, the Essential flag only means that
dpkg will not remove the package in question, unless given the
--force-remove-essential switch.

> (And when installing the package, dpkg would need to require installing
> it for every supported architecture; dpkg could refuse to configure the
> package if any enabled architecture doesn't have it unpacked.)

One problem here is that dpkg does not even know which packages are
available.  Another one: should "dpkg --add-architecture" automatically
deconfigure any such "Multi-Arch: every" package?

I think such problems are better solved in apt: apt-get dist-upgrade
already reinstalls every Essential package, the same way it could ensure
that NSS or PAM modules are pulled in for every enabled architecture.

Cheers,
   Sven

Handling Multi-Arch packages that must be installed for every enabled architecture?

[Josh Triplett]

Some packages, if installed on any architecture, must be installed for
every enabled architecture.  Most notably, an NSS or PAM module package,
if enabled in /etc/nsswitch.conf or /etc/pam.d respectively, must exist
for every enabled architecture to avoid breaking programs for that
architecture.

As one possible solution for this problem (but not an ideal one, just a
thought experiment), dpkg could support a new value for "Multi-Arch",
"Multi-Arch: every".  This value would imply "Multi-Arch: same", but if
installed, would additionally cause dpkg to act the same way it does for
Essential packages: install the package when enabling the architecture.
(And when installing the package, dpkg would need to require installing
it for every supported architecture; dpkg could refuse to configure the
package if any enabled architecture doesn't have it unpacked.)

That would solve the problem for the couple of cases it has come up in,
but it seems far from ideal; I'd welcome an cleaner alternative
solution.  Notably, this doesn't work well for plugin packages for
libraries less critical than glibc; it's not even ideal for PAM, as not
every enabled architecture will have packages depending on libpam0g.
The real dependency is "if any package on the architecture depends on
package X, and package Y is installed, package Y:arch must be
installed", but that's excessively complicated.

Any ideas on how to solve this problem?

- Josh Triplett