Re: Bug#839046: debootstrap: enable --merged-usr by default

[Raphael Hertzog]

On Tue, 13 Feb 2018, Julien Cristau wrote:
> On 02/13/2018 04:29 PM, Raphael Hertzog wrote:
> > I believe that we have had quite some testing already last time and I
> > would be surprised if we got many more RC bugs on dpkg that had to be fixed
> > on a short timeframe. I guess nobody can give you any assurance but
> > I'm sure that you can downgrade those bugs pointing to this discussion
> > and showing that this was part of the deal.
> 
> If we break user systems we don't get to downgrade bugs on the basis
> that "we knew we were doing a half assed job with this transition".
> That's not part of the deal we're making with our users.

Granted, but RC bugs against dpkg does not translate into "break user
systems". I would be very surprised if we found out a tool using dpkg
--search that would break the system on which it is running because
it did not get an answer to a specific dpkg --search query.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Julien Cristau]

On 02/13/2018 04:29 PM, Raphael Hertzog wrote:
> I believe that we have had quite some testing already last time and I
> would be surprised if we got many more RC bugs on dpkg that had to be fixed
> on a short timeframe. I guess nobody can give you any assurance but
> I'm sure that you can downgrade those bugs pointing to this discussion
> and showing that this was part of the deal.
> 
If we break user systems we don't get to downgrade bugs on the basis
that "we knew we were doing a half assed job with this transition".
That's not part of the deal we're making with our users.

Cheers,
Julien

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Raphael Hertzog]

On Mon, 12 Feb 2018, Ansgar Burchardt wrote:
> Guillem Jover writes:
> > In any case, I looked the other day into implementing the
> > --map-pathname option for dpkg-query, and I've got most of the code
> > ready. The problem is that this requires adding support for config
> > files and config fragments to dpkg-query, which has the additional
> > problem of making it possible to mess with the --showformat option
> > and breaking the expectations from maintscripts, for example. The
> > other problem is with the search behavior changing depending on the
> > packages providing those mappings being installed (because dpkg would
> > certainly not provide them).
> 
> So who should provide them?  debootstrap?  base-files?

It certainly makes sense for debootstrap to create those files given it
creates the symlinks in the first place.

An alternative solution could be to reuse the usrmerge package and let
debootstrap install this package if it exists. That way the usrmerge
package would exist until Debian switched entirely to put everything into
/usr/bin.

> The correct long-term fix is probably for packages to eventually install
> to the location in /usr anyway.  That doesn't work without some
> transition period of 1-2 releases though.

Ack.

On Mon, 12 Feb 2018, Guillem Jover wrote about usrmerge:
> This seems like a nice PoV for people to play with it, in the same way
> local admins can use, to some extent, symlinks to redirect subtrees to
> other mount points, but I don't see how this can be seen as a
> production-ready solution shipped by default, TBH.

Note that the change in debootstrap does not install usrmerge currently.
It only creates the required symlinks.


But this is enough to confuse "dpkg -S". This used to break dpkg-shlibdeps
and was the main reason for the initial revert. Fortunately dpkg-shlibdeps has
been fixed to try multiple paths until it can find the package owning the
library.


It might be that we will find other tools confused by "dpkg -S" non-answer
on some /lib/* or /usr/lib/* paths and some end users will definitely be 
surprised by
this.

Obviously we can document the problem in release notes but it would be
better to have a clean solution like suggested by Guillem:

> In any case, I looked the other day into implementing the
> --map-pathname option for dpkg-query, and I've got most of the code
> ready.

If this is forthcoming in the buster timeframe, it seems reasonable to
go ahead and re-enable the merged-usr by default. However to be able to
ship the new configuration files once their format is known, it would be
better for debootstrap to install a package whose role will be to provide
those configuration files ASAP after dpkg starts to support them.

> The problem is that this requires adding support for config
> files and config fragments to dpkg-query, which has the additional
> problem of making it possible to mess with the --showformat option
> and breaking the expectations from maintscripts, for example.

Forbid those options there? Do not parse those files if you detect
an environment variable DPKG_RUNNING_VERSION?

> The other problem is with the search behavior changing depending on the
> packages providing those mappings being installed (because dpkg would
> certainly not provide them).

That's the whole point of it so I would not consider this a problem.

> So I'll eventually try to find a solution for the dpkg-query issue,
> because it's a long-standing paper-cut in dpkg for local admins. But
> I'd not be very amused if this hack is enabled by default again and
> suddenly RC bugs start popping up in dpkg again, and people start
> pressuring with RCness to get those fixed again because well, it's
> obviously breaking people's systems…

Are you considering both "usrmerge" and "debootstrap creating symlinks" as
hacks even once they would provide mapping data for dpkg --search?

I believe that we have had quite some testing already last time and I
would be surprised if we got many more RC bugs on dpkg that had to be fixed
on a short timeframe. I guess nobody can give you any assurance but
I'm sure that you can downgrade those bugs pointing to this discussion
and showing that this was part of the deal.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Ansgar Burchardt]

Guillem Jover writes:
> In any case, I looked the other day into implementing the
> --map-pathname option for dpkg-query, and I've got most of the code
> ready. The problem is that this requires adding support for config
> files and config fragments to dpkg-query, which has the additional
> problem of making it possible to mess with the --showformat option
> and breaking the expectations from maintscripts, for example. The
> other problem is with the search behavior changing depending on the
> packages providing those mappings being installed (because dpkg would
> certainly not provide them).

So who should provide them?  debootstrap?  base-files?

The correct long-term fix is probably for packages to eventually install
to the location in /usr anyway.  That doesn't work without some
transition period of 1-2 releases though.

Ansgar

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Guillem Jover]

On Mon, 2018-02-05 at 22:32:05 +, Holger Levsen wrote:
> On Mon, Feb 05, 2018 at 08:19:33PM +0100, Julien Cristau wrote:
> > On Sat, Feb  3, 2018 at 09:16:40 +0100, Marco d'Itri wrote:
> > > On Dec 23, md wrote:
> > > > On Dec 20, Julien Cristau  wrote:
> > > > > > This change was reverted in 1.0.87 as dpkg-shlibdeps didn't cope
> > > > > > properly with a merged-usr system. Thus reopening this bug report 
> > > > > > for
> > > > > > that version.
> > > > > > 
> > > > > > The dpkg-shlibdeps bugs has been fixed [1] in the mean time. So it 
> > > > > > would
> > > > > > be great if this bug report could be re-considered.

> > > > > That'll be after stretch now.

> > > > Stretch was been released long ago: please re-enable --merged-usr in 
> > > > debootstrap.

> > > There has not been any negative feedback, can we move on please?

> > Is there buy-in from the dpkg maintainer?

As I've mentioned in the past, I think the usrmerge filesystem layout
has merit and can solve some issues, and to state it very clearly, I
have no technical issue with it *what*so*ever*. But the same can be
said about the non-usrmerge layout with multiple mount-points, which
while not general anymore in Debian, can still be used perfectly fine
on controlled subsets of packages and custom built kernels w/o reliance
on an initramfs.

What I still find to be terrible is the way it's been tried to be
deployed in Debian, via the usrmerge package, which does not support
reverting the change (#848626), for which there's (AFAIK) no way to
select not using this irreversible hack from d-i, which breaks
dpkg-query --search (at least #848622 and #858331). This seems like
a nice PoV for people to play with it, in the same way local admins
can use, to some extent, symlinks to redirect subtrees to other mount
points, but I don't see how this can be seen as a production-ready
solution shipped by default, TBH.

In any case, I looked the other day into implementing the
--map-pathname option for dpkg-query, and I've got most of the code
ready. The problem is that this requires adding support for config
files and config fragments to dpkg-query, which has the additional
problem of making it possible to mess with the --showformat option
and breaking the expectations from maintscripts, for example. The
other problem is with the search behavior changing depending on the
packages providing those mappings being installed (because dpkg would
certainly not provide them).


So I'll eventually try to find a solution for the dpkg-query issue,
because it's a long-standing paper-cut in dpkg for local admins. But
I'd not be very amused if this hack is enabled by default again and
suddenly RC bugs start popping up in dpkg again, and people start
pressuring with RCness to get those fixed again because well, it's
obviously breaking people's systems…


Thanks,
Guillem

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Philipp Kern]

On 02/11/2018 02:40 PM, Philip Hands wrote:
> That being the case, I think we should let the people volunteering to do
> the work to get on with it without delay. That way there will be plenty
> of time to address any real downsides that might be revealed.

Something I did notice yesterday by accident is that ldd points to
libraries being loaded from /lib. Which is fine except that then you
can't put them into dpkg -S because dpkg does not know about the
aliasing that happens here. I suppose this is because /lib/* is listed
before /usr/lib/* in /etc/ld.so.conf.d/*-linux-gnu.conf. I suppose if
one wants to use the output to find the packages associated, one needs
to roundtrip through realpath once. But then again you might miss files
actually installed by dpkg into /lib.

So I think the question of "is there buy-in from the dpkg maintainers to
support the outcome" is not a bad one. ;-)

Kind regards
Philipp Kern



signature.asc
Description: OpenPGP digital signature

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Philip Hands]

Hi Ian,

You're not citing any concrete examples of things that will supposedly
break.

AFAICS the closest you got was:

On Sat, 10 Feb 2018, Ian Jackson  wrote:
> Another bad consequence is that some existing configurations that do
> not, for whatever reason, mount /usr early, will be harder to set up.

Which might be a fair point, except that late mounting of /usr became
explicitly unsupported with the release of Stretch:

  
https://www.debian.org/releases/stretch/armel/release-notes/ch-information.en.html#late-mounting-usr

The adoption of usrmerge was blocked before on the basis that it was too
late in the release cycle.  We are now fairly early in the release cycle.

That being the case, I think we should let the people volunteering to do
the work to get on with it without delay. That way there will be plenty
of time to address any real downsides that might be revealed.

Cheers, Phil.

P.S. I've not even installed usrmerge on any systems as yet, so please
don't assume that I'm a rabid supporter of this effort -- it just seems
entirely sane to me, whereas the pushback you pointed at ... not so much.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Bug#839046: debootstrap: enable --merged-usr by default

[Holger Levsen]

On Mon, Feb 05, 2018 at 08:19:33PM +0100, Julien Cristau wrote:
> On Sat, Feb  3, 2018 at 09:16:40 +0100, Marco d'Itri wrote:
> > On Dec 23, md wrote:
> > > On Dec 20, Julien Cristau  wrote:
> > > > > This change was reverted in 1.0.87 as dpkg-shlibdeps didn't cope
> > > > > properly with a merged-usr system. Thus reopening this bug report for
> > > > > that version.
> > > > > 
> > > > > The dpkg-shlibdeps bugs has been fixed [1] in the mean time. So it 
> > > > > would
> > > > > be great if this bug report could be re-considered.
> > > > That'll be after stretch now.
> > > Stretch was been released long ago: please re-enable --merged-usr in 
> > > debootstrap.
> > There has not been any negative feedback, can we move on please?
> Is there buy-in from the dpkg maintainer?

cc:ing them.


-- 
cheers,
Holger


signature.asc
Description: PGP signature