Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

[Holger Levsen]

On Fri, Aug 11, 2023 at 10:56:03PM +0200, Helmut Grohne wrote:
> > what about cdebootstrap?
> cdebootstrap (and mmdebstrap) never implemented a merging step[1] and to
> this date rely on the usrmerge package doing it at postinst time. Once
> base-files ships the aliasing symlinks, both will produce /usr-merged
> trees without any modifications. The reason that we need a change to
> debootstrap is that its current merging implementation breaks when
> base-files ships aliasing symlinks.
> 
> So the main reason for doing this change to debootstrap is that it
> enables us to continue supporting cdebootstrap and mmdebstrap without
> any changes there.

ah, thank you!


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Just because other people are also responsible, does not mean you are not
responsible.


signature.asc
Description: PGP signature

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

[Helmut Grohne]

Hi Holger,

On Fri, Aug 11, 2023 at 09:28:51AM +, Holger Levsen wrote:
> On Fri, Aug 11, 2023 at 09:38:02AM +0100, Luca Boccassi wrote:
> > > This is implemented in
> > > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96
>  
> what about cdebootstrap?

cdebootstrap (and mmdebstrap) never implemented a merging step[1] and to
this date rely on the usrmerge package doing it at postinst time. Once
base-files ships the aliasing symlinks, both will produce /usr-merged
trees without any modifications. The reason that we need a change to
debootstrap is that its current merging implementation breaks when
base-files ships aliasing symlinks.

So the main reason for doing this change to debootstrap is that it
enables us to continue supporting cdebootstrap and mmdebstrap without
any changes there.

Helmut

[1] For mmdebstrap there is a merged-usr hook that can do it. Johannes
will migrate it to the same post-merging approach I am proposing for
debootstrap here.

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

[Holger Levsen]

On Fri, Aug 11, 2023 at 09:38:02AM +0100, Luca Boccassi wrote:
> > This is implemented in
> > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96
 
what about cdebootstrap?


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

"In just 6 decades, roughly the life span of a blue whale, humans took blue 
whale
population down from 360,000 to just 1,000. In one century, whalers killed two
million baleen whales, which together weighed twice as much as all wild mammals
on Earth today."
https://www.theatlantic.com/science/archive/2021/11/whaling-whales-food-krill-iron/620604/


signature.asc
Description: PGP signature

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

[Luca Boccassi]

On Fri, 11 Aug 2023 at 05:52, Helmut Grohne  wrote:
>
> Hi,
>
> This is picking up on the debootstrap matter and is kinda crucial.
>
> On Thu, Jul 13, 2023 at 01:31:04AM +0100, Luca Boccassi wrote:
> > > After having sorted this out, what part of your safety concerns with 3C
> > > do remain?
> >
> > Nothing, as that stemmed from a misunderstanding of what the
> > implementation would have required, and that's cleared now.
>
> So we finally removed the misunderstanding with Luca and I imply that
> this also removes Sam's concern (as he was inheriting the
> misunderstanding from Luca).
>
> Let me briefly recap the most important pieces. The proposal at hand is
> changing debootstrap in unstable, testing, stable and oldstable. Rather
> than merging /usr before the initial unpack, it will merge after the
> initial unpack but before running maintainer scripts. Therefore
> base-files can ship aliasing symlinks without triggering tar errors from
> debootstrap and once it does, the merging step in debootstrap
> automatically becomes a noop. With this change in place, we can move
> forward without changing cdebootstrap nor mmdebstrap.
>
> This is implemented in
> https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96
> and reviewed by Luca Boccassi and Simon McVittie. Thank you two. I have
> tested this change for bootstrapping buster, bullseye, bookworm and
> trixie on amd64 without hitting regressions.
>
> Do we have any more disagreement with this approach or implementation?
> If you review the MR, don't hesitate to leave a positive or negative
> comment on it. We want to make sure that this doesn't break stuff as its
> exposure is high.
>
> I intend to merge and NMU this change before too long and Simon McVittie
> intends to prepare stable and oldstable uploads with this change and the
> change to make --variant=buildd /usr-merged for trixie and beyond.
> Having these changes in oldstable (and thus affecting buildds) is a
> precondition for lifting the moratorium, so we'd like to move forward
> soon.

Ship it!

If you need help with NMUs to unstable or *stable let me know.

Kind regards,
Luca Boccassi

Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

[Helmut Grohne]

Hi,

This is picking up on the debootstrap matter and is kinda crucial.

On Thu, Jul 13, 2023 at 01:31:04AM +0100, Luca Boccassi wrote:
> > After having sorted this out, what part of your safety concerns with 3C
> > do remain?
> 
> Nothing, as that stemmed from a misunderstanding of what the
> implementation would have required, and that's cleared now.

So we finally removed the misunderstanding with Luca and I imply that
this also removes Sam's concern (as he was inheriting the
misunderstanding from Luca).

Let me briefly recap the most important pieces. The proposal at hand is
changing debootstrap in unstable, testing, stable and oldstable. Rather
than merging /usr before the initial unpack, it will merge after the
initial unpack but before running maintainer scripts. Therefore
base-files can ship aliasing symlinks without triggering tar errors from
debootstrap and once it does, the merging step in debootstrap
automatically becomes a noop. With this change in place, we can move
forward without changing cdebootstrap nor mmdebstrap.

This is implemented in
https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96
and reviewed by Luca Boccassi and Simon McVittie. Thank you two. I have
tested this change for bootstrapping buster, bullseye, bookworm and
trixie on amd64 without hitting regressions.

Do we have any more disagreement with this approach or implementation?
If you review the MR, don't hesitate to leave a positive or negative
comment on it. We want to make sure that this doesn't break stuff as its
exposure is high.

I intend to merge and NMU this change before too long and Simon McVittie
intends to prepare stable and oldstable uploads with this change and the
change to make --variant=buildd /usr-merged for trixie and beyond.
Having these changes in oldstable (and thus affecting buildds) is a
precondition for lifting the moratorium, so we'd like to move forward
soon.

Helmut

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Tue, 11 Jul 2023 at 09:44, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Tue, Jul 11, 2023 at 12:27:04AM +0100, Luca Boccassi wrote:
> > You have said in the original mail and on the writeup that this option
> > requires all the affected packages to be upgraded at the same time,
> > and in the correct order, or things will break. What happens if any of
>
> This definitely is a misunderstanding. At this time, I am not sure where
> that misunderstanding originated and I don't even think it is worth
> figuring out, but I wont mind a reference either.
>
> I meant to say that the option requires the involved (~10) packages to
> be *uploaded* (rather than upgraded) at the same time. And that
> requirement originates from the bootstrapping aspect rather than an
> upgrading aspect. Bootstrapping will be broken from the point in time of
> the first upload of that set until the last package from that set has
> been built.
>
> I would not dare propose a solution that'd require upgrades to happen in
> a specific order or way that is not expressible to apt. My impression is
> that we have significant consensus on smooth upgrades being a core
> feature of Debian. This should have failed your plausibility filter.

Interesting - I guess from here from the original mail:

> On the flip side, there is a demo for #3c showing that we can move most
> of the things except for a hand full of packages and then flip the
> switch (for bootstrapping) in unstable by uploading those packages
> simultaneously. The biggest downside of this probably is the inherent
> fragility of this approach. Even if this is extensively tested before
> uploading chances are good that we still break something unforeseen in
> the process.

in my head 'uploading' became associated with 'updating', so I thought
the implementation of this process required some precise runtime
constraints. Good to know that it's not the case then.

> > those packages are held back, for whatever reason? This is the
> > fragility aspect that I am worried about, and that is not an issue at
> > all if we just fix mmdebstrap to do the right thing as debootstrap
> > already does.
>
> There are two mechanisms that shall ensure that any (valid according to
> relationships) order and and held packages (up to not performing the
> operation) will work. One is the Pre-Depends on usrmerge-support. If you
> pin that package as absent for otherwise force its absence, apt will
> simply refuse to upgrade everything else and your system will be stuck
> at upgrading entirely. If you hold back any other package, it may keep
> shipping files in aliased locations. The protective diversion mechanism
> (DEP17-M4) will ensure that this does not cause the aliasing links to
> disappear if you upgrade it later.
>
> After having sorted this out, what part of your safety concerns with 3C
> do remain?

Nothing, as that stemmed from a misunderstanding of what the
implementation would have required, and that's cleared now.

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Luca,

On Tue, Jul 11, 2023 at 12:27:04AM +0100, Luca Boccassi wrote:
> You have said in the original mail and on the writeup that this option
> requires all the affected packages to be upgraded at the same time,
> and in the correct order, or things will break. What happens if any of

This definitely is a misunderstanding. At this time, I am not sure where
that misunderstanding originated and I don't even think it is worth
figuring out, but I wont mind a reference either.

I meant to say that the option requires the involved (~10) packages to
be *uploaded* (rather than upgraded) at the same time. And that
requirement originates from the bootstrapping aspect rather than an
upgrading aspect. Bootstrapping will be broken from the point in time of
the first upload of that set until the last package from that set has
been built.

I would not dare propose a solution that'd require upgrades to happen in
a specific order or way that is not expressible to apt. My impression is
that we have significant consensus on smooth upgrades being a core
feature of Debian. This should have failed your plausibility filter.

> those packages are held back, for whatever reason? This is the
> fragility aspect that I am worried about, and that is not an issue at
> all if we just fix mmdebstrap to do the right thing as debootstrap
> already does.

There are two mechanisms that shall ensure that any (valid according to
relationships) order and and held packages (up to not performing the
operation) will work. One is the Pre-Depends on usrmerge-support. If you
pin that package as absent for otherwise force its absence, apt will
simply refuse to upgrade everything else and your system will be stuck
at upgrading entirely. If you hold back any other package, it may keep
shipping files in aliased locations. The protective diversion mechanism
(DEP17-M4) will ensure that this does not cause the aliasing links to
disappear if you upgrade it later.

After having sorted this out, what part of your safety concerns with 3C
do remain?

I see that Sam and Guillem dislike my proposal of abusing diversions in
this way, but I honestly see little alternatives for doing this in a
safe way. In essence, we are introducing a symlinks vs directory
conflict for the aliasing symlinks and if anything goes wrong you may
end up without /bin/sh or the ELF loader. While diversions were not
meant for this situation, reasoning about them is relatively straight
forward for the purpose of what we need here. Critically, we don't need
any properties about the renamed location and we only need the property
that dpkg leaves the diverted location unmodified. I fully acknowledge
that I propose using diversions outside their specification. However, we
already use dpkg outside its specification in general (due to having
merged /usr). As such, we already have entered the land of relying on
implementation-defined behavior. Raphael's mitigation of making dpkg
more careful about deleting aliased files (DEP17-M3) also is about
temporarily extending dpkg in such an implementation-defined way. The
reason why I see diversions as favourable here is that they are a
similarly ugly mechanism that is readily available in the upgrade to
trixie and that all of them are of varying temporary nature:
 * DEP17-M4: We need these from the point in time where we start moving
   files out of aliased locations until the point in time where
   base-files has recorded all aliasing links in the dpkg database. From
   an unstable pov, this is probably is less than a year. From a
   bookworm->trixie pov, the diversions will be added and removed during
   the upgrade.
 * DEP17-M6: The diversion of dpkg-divert is not a protective diversion
   and is probably needed in trixie and forky.
 * DEP17-M8: These protective diversions are short lived during an
   individual package upgrade from preinst to postinst.
 * DEP17-M9: These protective diversions are longer lived. They probably
   need to be present in trixie. I hope that the majority of cases can
   rather delete an unnecessary empty directory than set up a diversion.
 * DEP17-M10: These protective diversions are short lived during an
   individual package upgrade from preinst to postinst.
 * DEP17-M14: The diversion of update-alternatives is not a protective
   diversiion and is probably needed for trixie and forky.

You see that the majority of these diversions is short lived. Since I
propose introducing them on-demand rather than automatically, their
number should be low. If it were not the case that this abuse of
diversions were temporary, I would be opposed to it. What makes it
attractive to me is that the alternatives also seem to be abusing dpkg
and the diversion abuse works right now.

Helmut

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Simon Richter]

Hi,

On 7/11/23 00:55, Sam Hartman wrote:


* The more I look at this, I think the real complexity is not in
   bootstrapping, but is in the rest of  the proposal for canonicalizing
   paths.  I am very uncomfortable overall; it seems complicated enough
   that we probably will break something somewhere.  I do not see anry
   better options though.  I think this affects things in a couple ways:



   * I hope we can put the bootstrapping decision behind us soon and
 focus on the harder problem, because I think bootstrapping is a bit
 of a bikeshed at this point.


There is only one important decision to be made about bootstrapping: do 
we want to extend the protocol, or not?


If yes, then the rest of the bootstrapping process can be decided after 
we have a solution for upgrades, and especially should not put 
additional constraints there, so I'd explicitly avoid making a decision 
here that will then reappear with a prefix of "but we already decided 
that" during the complex part.


If no, then bootstrapping becomes part of the constraint set, same as 
"upgrades need to be facilitated through packages whose installation 
order is defined through package relationships as interpreted by the 
current stable release of apt" we also get "packages need to be 
installable by unpacking their data member through tar, then 
subsequently installing the package over that with the just-unpacked 
version of dpkg."


And because the bootstrap scenario uses a as-of-yet unreleased version 
of the dpkg package, we have way more freedom there than with the 
upgrade process, so optimizing this first is the best way to sink a lot 
of cost into a solution.


   Simon

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Sun, 9 Jul 2023 at 22:07, Helmut Grohne  wrote:
>
> Hi Sam,
>
> Thanks for trying to wrap your head around the complexity.
>
> On Sat, Jul 08, 2023 at 07:57:40AM -0600, Sam Hartman wrote:
> > So for me, a 3C proposal would have two components:
> >
> > 1) An explanation of what the archive looks like at time of bootstrap
> > (and changes to any bootstrap programs) so I can reason about whether
> > bootstrap works.
>
> I hope this one is simple.
>  * All packages ship all of their files in canonical locations.
>  * base-files ships all of the aliasing symbolic links and their target
>directories.
>  * Given that base-files installs the symbolic links, all programs are
>immediately working after unpack prior to running any maintainer
>scripts.
>  * Consequently, cdebootstrap and mmdebstrap just work without any
>modification.
>  * An unmodified debootstrap fails (unpacking base-files due to -k).
>+ We modify debootstrap such that it first unpacks and then merges.
>OR
>+ We stop passing the -k flag to tar. (Though we need a better
>  understanding for why that was added post jessie.)
>
> > 2) An argument of safety of upgrades focused on the changes and why
> > those changes are safe both for unstable upgrades and for bookworm
> > upgrades.
>
> As far as I understand the question here, it is about those aspects that
> are specific to the 3C solution as opposed to a 3B solution. In both
> cases, we move all of the files to their canonical locations. I'm not
> sure whether the protective diversions for aliasing links (DEP17-M4) are
> something we ultimately need in all scenarios, but in case of 3C, we
> quite certainly need them to make upgrades safe, so that's an aspect to
> consider here. The other aspect of course is shipping the symlinks in
> base-files (DEP17-M11). So what could go wrong here?
>
> In an upgrade scenario from unstable or from bookworm, we'd have to
> unpack and configure usrmerge-support before unpacking base-files, since
> that becomes a Pre-Depends of base-files. usrmerge-support.preinst would
> verify that the filesystem is merged already (much like usr-is-merged
> does) except that it does not tolerate
> /etc/unsupported-skip-usrmerge-conversion anymore, so any system using
> that mechanism will fail this preinst. Then usrmerge-support.preinst
> would install the protective diversions (DEP17-M4) on behalf of
> base-files. Since these are --no-rename, the filesystem is not modified
> and since we just verified that all the affected locations really are
> symbolic links rather than directories, dpkg-divert wouldn't error out
> about diverting a directory. In any case, usrmerge-support is eventually
> configured (without a postinst), which allows unpacking base-files.
> Whenever we unpack base-files (now or for subsequent upgrades), dpkg
> will create each aliasing symlink with a temporary name and rename(2) it
> to the final destination. Since rename(2) atomic, the aliasing symlinks
> will be never go missing. When upgrading or removing any other package,
> dpkg may consider removing an aliasing symlink as that package may be
> the last package to ship an aliased file. When this happens, the removal
> of the directory will be redirected to an innocent location via the
> protective diversion. Since diversions only match exactly (they are not
> meant to be used for directories), files installed "below" the diverted
> aliasing links (i.e. aliased files) will be entirely unaffected by the
> protective diversions and dpkg will operate on them as usual.
>
> The most common failure mode during upgrades seen by users likely will
> be when /etc/unsupported-skip-usrmerge-conversion exists and the system
> isn't actually merged.
>
> I have a hard time figuring out what else could go wrong here and that's
> probably because I'm biased towards 3C. On the other hand, the reason
> for me to like it is because I see very little that could go wrong (in
> addition to what can already go wrong due to moving all the files). I
> hope that others can use this detailed description of what happens to
> construct possible failure cases such that we can better understand the
> risk here.

You have said in the original mail and on the writeup that this option
requires all the affected packages to be upgraded at the same time,
and in the correct order, or things will break. What happens if any of
those packages are held back, for whatever reason? This is the
fragility aspect that I am worried about, and that is not an issue at
all if we just fix mmdebstrap to do the right thing as debootstrap
already does.

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

> "Timo" == Timo Röhling  writes:


Nod, I was wrong.  Wanted to ex plicitly acknowledge that, although I
think it is also obvious from other mails.

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

Hi.
I have read both of your messages over the weekend multiple times.
I don't think replying point-by-point is going to be all that helpful,
although if there are any cases where you'd like me to do that, I will.

* I am really impressed with the work you are putting in on all this.
  You have done an amazing job at a really hard problem.

* Mostly, I popped into this discussion to try and help confirm
 consensus.  I think my participation has come close to outliving its
 usefulness.  I am not involved in this problem enough to be running
 experiments, and I am probably not going to go dig into the guts of
 dpkg to start looking for problems that we might be overlooking.
 

* The more I look at this, I think the real complexity is not in
  bootstrapping, but is in the rest of  the proposal for canonicalizing
  paths.  I am very uncomfortable overall; it seems complicated enough
  that we probably will break something somewhere.  I do not see anry
  better options though.  I think this affects things in a couple ways:

  * I hope we can put the bootstrapping decision behind us soon and
focus on the harder problem, because I think bootstrapping is a bit
of a bikeshed at this point.

  * I hope that we're open to better ideas of doing things proposed even
fairly late in the process.  If someone finds ways of removing
complexity  that we don't see now, I think it is worth seriously
considering even if implementation has already started.

* I think the bootstrapping decision may not be something we need a
  project consensus on.  If the people involved can get to something
  that works for them, that's probably good enough.  So, mmdebstrap
  maintainer, people who have worked on debootstrap recently, with no
  significant objections from base-files, glibc, or other major
  essential packages.

* I think your most recent mail really helps explain 3C, and I agree
  that is a proposal that someone sufficiently knowledgable could
  evaluate for safety.  I do not feel comfortable enough with my
  knowledge of dpkg-divert to say "looks good to me," although I am now
  more comfortable with 3C than I was earlier.

* Mitigation M-4 introduces what appear to me to be some undesirable
  properties  we should at least document.  We appear to be depending
  heavily on limitations of dpkg-divert.  In particular, we're depending
  on the idea that diversions don't work for directories.  So we're
  introducing yet more cases where dpkg's view of the world is different
  than reality.  We're doing more things that would make it difficult
  for the dpkg maintainer if they actually wanted to enhance dpkg to
  better understand what was going on.  If dpkg wanted to support
  diverting directories, it would now also need to support these kind of
  fake protective diversions.

* I specifically withdraw my concerns about changing debootstrap to move
  directories and create symlinks after the initial unpack.  I was
  imagining that the complexity would be similar to usrmerge.  But as
  you point out in your patch, removing the atomicity requirement makes
  things far simpler.

* I think I still prefer 3B to 3C if only because it might avoid M-4
  (and I think M-8 might be less problematic than M-4 at least if we can
  avoid protecting directories with M-8).
  However, if we were voting I'd rank both 3C and 3B above none of the
  above.  I'd rank "let the people doing the work decide" well above
  either 3B or 3C.

If there is anything else I can do to help put bootstrapping behind us at
least for now and help get to a concrete proposal for canonicalizing
files, let me know.
I think that proposal will require as much review as we can get.

--Sam


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Sam,

Thanks for trying to wrap your head around the complexity.

On Sat, Jul 08, 2023 at 07:57:40AM -0600, Sam Hartman wrote:
> So for me, a 3C proposal would have two components:
> 
> 1) An explanation of what the archive looks like at time of bootstrap
> (and changes to any bootstrap programs) so I can reason about whether
> bootstrap works.

I hope this one is simple.
 * All packages ship all of their files in canonical locations.
 * base-files ships all of the aliasing symbolic links and their target
   directories.
 * Given that base-files installs the symbolic links, all programs are
   immediately working after unpack prior to running any maintainer
   scripts.
 * Consequently, cdebootstrap and mmdebstrap just work without any
   modification.
 * An unmodified debootstrap fails (unpacking base-files due to -k).
   + We modify debootstrap such that it first unpacks and then merges.
   OR
   + We stop passing the -k flag to tar. (Though we need a better
 understanding for why that was added post jessie.)

> 2) An argument of safety of upgrades focused on the changes and why
> those changes are safe both for unstable upgrades and for bookworm
> upgrades.

As far as I understand the question here, it is about those aspects that
are specific to the 3C solution as opposed to a 3B solution. In both
cases, we move all of the files to their canonical locations. I'm not
sure whether the protective diversions for aliasing links (DEP17-M4) are
something we ultimately need in all scenarios, but in case of 3C, we
quite certainly need them to make upgrades safe, so that's an aspect to
consider here. The other aspect of course is shipping the symlinks in
base-files (DEP17-M11). So what could go wrong here?

In an upgrade scenario from unstable or from bookworm, we'd have to
unpack and configure usrmerge-support before unpacking base-files, since
that becomes a Pre-Depends of base-files. usrmerge-support.preinst would
verify that the filesystem is merged already (much like usr-is-merged
does) except that it does not tolerate
/etc/unsupported-skip-usrmerge-conversion anymore, so any system using
that mechanism will fail this preinst. Then usrmerge-support.preinst
would install the protective diversions (DEP17-M4) on behalf of
base-files. Since these are --no-rename, the filesystem is not modified
and since we just verified that all the affected locations really are
symbolic links rather than directories, dpkg-divert wouldn't error out
about diverting a directory. In any case, usrmerge-support is eventually
configured (without a postinst), which allows unpacking base-files.
Whenever we unpack base-files (now or for subsequent upgrades), dpkg
will create each aliasing symlink with a temporary name and rename(2) it
to the final destination. Since rename(2) atomic, the aliasing symlinks
will be never go missing. When upgrading or removing any other package,
dpkg may consider removing an aliasing symlink as that package may be
the last package to ship an aliased file. When this happens, the removal
of the directory will be redirected to an innocent location via the
protective diversion. Since diversions only match exactly (they are not
meant to be used for directories), files installed "below" the diverted
aliasing links (i.e. aliased files) will be entirely unaffected by the
protective diversions and dpkg will operate on them as usual.

The most common failure mode during upgrades seen by users likely will
be when /etc/unsupported-skip-usrmerge-conversion exists and the system
isn't actually merged.

I have a hard time figuring out what else could go wrong here and that's
probably because I'm biased towards 3C. On the other hand, the reason
for me to like it is because I see very little that could go wrong (in
addition to what can already go wrong due to moving all the files). I
hope that others can use this detailed description of what happens to
construct possible failure cases such that we can better understand the
risk here.

If this procedure sounds risky to you anyway, please be aware of two
other aspects. For one thing, this protective diversion mechanism will
be required for other files (DEP17-M8) if we follow the current
consensus of moving files (DEP17-M2), so we'll bear the general risks of
protective diversions in any case - not just when doing 3C. Therefore it
is questionable whether we can attribute these risks to the choice of 3C
at all. Then, if you consider DEP17-M4 too risky in general, my current
understanding is that the only safe alternative mitigation for the loss
of aliasing symlinks (DEP17-P9) is shipping all of the aliasing symlinks
as directories in some package (DEP17-M5). If going that route, we must
distribute those links to multiple packages to escape from Pre-Depends
loops and we technically violate the mostly agreed principle that no
packages ship any aliased paths (DEP17-M2) for these few cases in
trixie. Is that really less complex and less risky?

> * Your debootstrap changes 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

Hi Helmut,

Let me restate once again that I think you are doing a stellar job at
tackling this problem, and you have my thanks for it. From what I can
see, I think 99% of us are on the same page for 90% of the problem.

The remaining part on how to make bootstrapping safe is the last bit.
And in the end it is not a big deal, if consensus is going the other
way, that's fine, let's do that and get this over with. However until
we get there, I must say I still agree with Sam:

On Sat, 8 Jul 2023 at 14:58, Sam Hartman  wrote:
> * You do not talk much about upgrades.  Upgrades happen against a moving
>   archive.
>   There, talking about the changes, and why the changes are always safe
>   is important.
>
> So for me, a 3C proposal would have two components:
>
> 1) An explanation of what the archive looks like at time of bootstrap
> (and changes to any bootstrap programs) so I can reason about whether
> bootstrap works.
>
> 2) An argument of safety of upgrades focused on the changes and why
> those changes are safe both for unstable upgrades and for bookworm
> upgrades.
>
> * Your debootstrap changes seem overly complicated and would in and of
>   themselves push me against 3C.  First, you don't seem to be thinking
>   about buster, which also needs to bootstrap usrmerged, doesn't it.
>   Second, is there a way we could simply change how debootstrap calls
>   tar?
>   I think asking debootstrap to not create the symlinks before is a big
>   ask.

I have not spent as much time as you have, so I do not have
reproducers or such things for what worries me, only a general sense
that I agree with the above - I think for 3C the upgrade flow is
under-documented. I am not worried about how bootstrap works in that
case. As already mentioned, I think if bootstraping a new sid chroot
is borked for a day or two it's not the end of the world - it happens,
it's sid.
The upgrade path however affects every running machine there is.
unstable -> unstable first, then testing -> testing, and then
oldstable -> stable in the future. Having complex machinery there
sometimes is just necessary, and it comes with important caveats. See
the usrmerge package for example - we needed to do a live conversion
for old systems, so it was necessary to have it, and it works well,
however it has corner cases that it just cannot handle (eg: overlay
where the OS is the base layer) and where we just raise our hands and
nope out. I don't know if what you propose would have similar issues
or not, but given there _is_ an alternative here to fix bootstrapping
that doesn't require delegating complex machinery to the packages
themselves, I very much prefer that, as to me it obviously seems
inherently less risky.

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> Hi Sam,
Helmut> On Fri, Jul 07, 2023 at 08:50:49AM -0600, Sam Hartman wrote:
>> 
>> TL;DR: It looks like if we do not want to encode merged /usr into
>> the bootstrap protocol, we must keep some aliases and cannot move
>> all files in data.tar.

Helmut> Reading both of your recent mails left me very confused. It
Helmut> is now obvious to me that we have a misunderstanding (at
Helmut> least one) and I am not exactly sure how we can get to a
Helmut> point where we talk about the same things.

Helmut> In your second mail, you classify 3C (the one about not
Helmut> changing the bootstrap protocol by adding aliasing symlinks
Helmut> to base-files) as a category 1 solution (where we leave some
Helmut> files in their aliased location to facilitate bootstrap). In
Helmut> reality, 3C is fully incompatible with category 1 as the
Helmut> premise of 3C is that every essential package has all of its
Helmut> files moved. This directly contradicts your TL;DR here.

Yes, we definitely have a misunderstanding.
I am at a conference today without much focus for Debian.
I did try to read your mail, but mostly my brain rebelled and argued
that's too complicated.

I'll try again tomorrow evening US time or Monday when I can give it
real focus.

Here's how we got to the understanding.

* I asked for a specific proposal to evaluate for 3C

* you responded:


>> So I think that to argue for 3C you need a specific proposal
>> about what happens.

Helmut> https://lists.debian.org/20230517093036.ga4104...@subdivi.de

>Unfortunately, we're not yet done here. When (for instance) dash is the
>last package to move /bin/sh and such to /usr and you upgrade dash, dpkg
>notices that no package owns /bin anymore. Thus it helpfully deletes
>/bin (the symlink).  You're not happy when this happens. We remember the
>silver bullet: diversions! So dash.preinst could dpkg-divert --no-rename
>--divert /bin.usrmerged -add /bin and dash.postinst could revert that.
>Unfortunately, such a diversion affects every package but dash and we
>want it exactly the other way round. So what we could do is pass
>--package dash-usrmerged (which must not exist). Then it'll actually
>keep /bin safe. Unfortunately, we don't know whether dash or bash will
>be the last package owning /bin, so both of them need this diversion and
>this is a conflict.

>And as if that wasn't enough, we also run into issues around hard links.
>As dpkg unpacks a canonicalized gzip, it notices that /bin/gunzip (which
>is scheduled for deletion) has the same inode as the new
>/usr/bin/uncompress (because gunzip and uncompress are hard linked). As
>far as I can see, all we get here is a warning and both files survive
>the unpack. It is not clear to me whether this happens by chance or
>reliably.

>dpkg: warning: old file '/bin/uncompress' is the same as several new
>files! (both '/usr/bin/gunzip' and '/usr/bin/uncompress')^
>dpkg: warning: old file '/bin/gunzip' is the same as several new files!
>(both '/usr/bin/gunzip' and '/usr/bin/uncompress')

>Given what I've seen, I'm fairly convinced that I haven't reached the
>bottom of it and I'm ready to conclude that this approach is fragile - a
>property that is most unwelcome when we deal with the essential set.
>So what's left is category 1. I looked into what the minimum set of

So, I assumed we were in a category 1 solution.
Because I read your message that you pointed to as a proposal for how 3C
worked, and before the concluding paragraph you said we were in a
category 1 solution.

I at least understand a category 1 solution.
It's fairly easy to reason about; I even think it's safe.
I don't like it for the architectural reasons I explained.
But well, if you tell me to go read a message that will contain a
proposal for what you want to do, and the last thing I read there--the
only thing that your analysis claims works--is category 1, that's going
to be left in my mind.

I really am trying to understand here, but this is exceeding the level
of complexity I can keep in my head.
That in and of itself is concerning: I'm well above average at being
able to work through this sort of thing.
I'll take a stab at your most recent email later.

Things I noticed already though:


* You are describing things in terms of changing.
Bootstrapping happens against a static archive.  It might be easier to
analyze if you describe what the end state looks like in terms of
bootstrapping rather than changes to get there.

* You do not talk much about upgrades.  Upgrades happen against a moving
  archive.
  There, talking about the changes, and why the changes are always safe
  is important.

So for me, a 3C proposal would have two components:

1) An explanation of what the archive looks like at time of bootstrap
(and changes to any bootstrap programs) so I can reason about whether
bootstrap works.

2) An argument of safety 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Timo Röhling]

Hi Gioele,

* Gioele Barabucci  [2023-07-08 10:53]:
Even the most convoluted and lock-stepped procedure can surely be 
carried out over a single day in an all-hands-on-deck effort. 
Especially if the files of non-critical packages are moved before the 
flag day.

I agree.

Maybe my wording was too harsh; as a matter of fact, I think the
flag day will work just fine, and even if we stretch the upload over
more than a dinstall cycle, the absolute worst that will happen is that
bootstrapping will not work for a few hours or days.

I initially thought that the unpack order would be simpler to
implement without breaking compatibility with older releases, but
after Helmut's sibling reply in this thread, I don't think there is
much of a difference, if at all.


Cheers
Timo


--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Gioele Barabucci]

On 08/07/23 09:44, Timo Röhling wrote:

Alternatively, we could move all files except for the few critical
ones (/bin/sh, dynamic loader)


Allow me to add some hard data to this discussion.

In essential (proper) there are 153 files that need to be moved, 
distributed across 15 packages (+ base-files).


Packages and number of files to be moved:

  1 libc-bin
  1 login
  1 sed
  1 tar
  2 bash
  3 dash
  3 debianutils
  3 dpkg
  3 grep
  5 hostname
  7 sysvinit-utils
 14 gzip
 28 coreutils
 36 util-linux
 45 ncurses-base

Even the most convoluted and lock-stepped procedure can surely be 
carried out over a single day in an all-hands-on-deck effort. Especially 
if the files of non-critical packages are moved before the flag day.


Regards,

--
Gioele Barabucci

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Sam,

On Fri, Jul 07, 2023 at 08:50:49AM -0600, Sam Hartman wrote:
> 
> TL;DR:
> It looks like if we do not want to encode merged /usr into the bootstrap
> protocol, we must  keep some aliases and cannot move all files in
> data.tar.

Reading both of your recent mails left me very confused. It is now
obvious to me that we have a misunderstanding (at least one) and I am
not exactly sure how we can get to a point where we talk about the same
things.

In your second mail, you classify 3C (the one about not changing the
bootstrap protocol by adding aliasing symlinks to base-files) as a
category 1 solution (where we leave some files in their aliased location
to facilitate bootstrap). In reality, 3C is fully incompatible with
category 1 as the premise of 3C is that every essential package has all
of its files moved. This directly contradicts your TL;DR here.

Let me try to ignore much of the past conversation and instead explain
the bootstrap-relevant part of the transition plan that I see as
favourable (i.e. 3C). Consider this an opinionated presentation for one
of multiple ways to move forward.

We first move towards a category 1 solution where we move files to their
canonical locations as much as possible without breaking the current way
of bootstrapping that relies on either pre-creating the symlinks
(debootstrap) or usrmerge.postinst (mmdebstrap, cdebootstrap). This
process has lots of non-obvious details that I'll skip for the sake of
the bootstrap topic here. Let us for a moment assume that we'd manage to
get to this category 1 solution where most files (in essential packages)
have moved their files to canonical locations and we're left with some
packages (e.g. libc6, dash, util-linux, ...) that could not move their
files.

There is one aspect, I want to get into more detail as it is partially
relevant to bootstrapping and this is protecting the aliasing symlinks
(DEP17-P9). Here, I am selecting DEP17-M4 as the relevant mitigation.
That amounts to adding a usrmerge-support package that creates
protective diversions for the aliasing symbolic links and assigns them
to base-files. Both base-files and any package that moves files out of
aliased locations has to gain Pre-Depends: usrmerge-support in order for
this method to be effective. So we'll likely see debhelper's
${misc:Pre-Depends} that we added for multiarch-support return.

Until this point, we can still decide whether we do 3B or 3C and I am
explaining what happens in the 3C choice now.

Concurrently with the earlier changes, we modify debootstrap.
debootstrap requires uploads to bookworm and bullseye anyway, because we
will have to change --variant=buildd to become merged for trixie and
forward while currently debootstrap always creates --variant=buildd as
unmerged. On top of this necessary change, we add a change relevant to
3C. Currently, debootstrap creates the aliasing symbolic links prior to
the initial package unpack. I want to swap these operations. In doing it
afterwards, debootstrap cannot just create the symlinks but may have to
perform an actual merge in much the same way that usrmerge does now
except for dropping the atomicity requirement (as we don't need that in
the bootstrap setting). Other than that, this change is fully backwards
compatible (since bootstrapping as unmerged and the installing usrmerge
also works) and will continue to debootstrap a merged bullseye, bookworm
and trixie in the same way as before. I'll get into why we do this in a
moment.

Then, we assume that updating debootstrap has happened, usrmerge-support
exists in trixie and the number of essential packages shipping files in
aliased locations is about 10. We now move away from category 1. At this
point my original categorization becomes a little difficult, so I defer
explaining what we move to. We prepare changes to canonicalize all those
remaining packages (only essential ones) to canonicalize their paths and
also prepare a change to base-files to change the type of the aliasing
symlinks from directories to symlinks. While it technically is a
directory-to-symlink conversion from a dpkg point of view, that
conversion has already happened on the filesystem level, so this
practically is a change to the dpkg database only. Now we upload all of
these packages concurrently. This is when mmdebstrap and cdebootstrap
temporarily break. Once all of the binary packages have been built and
installed into the archive, things should work again.

Due to having modified deboostrap, we arguably have moved into category
4. Since that change is backwards-compatible and has been uploaded to
bullseye and bookworm, we can kinda pretend that the bootstrap protocol
never was different and therefore say that we move into category 2, but
without the reasons originally given for why this cannot work. Had we
skipped that change to debootstrap, unpacking base-files would make
debootstrap fail, because it passes -k to tar and when tar would try to
create the /bin -> usr/bin symlink 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Timo Röhling]

Hi Sam,

* Sam Hartman  [2023-07-07 08:50]:

TL;DR:
It looks like if we do not want to encode merged /usr into the bootstrap
protocol, we must  keep some aliases and cannot move all files in
data.tar.
I think removing all aliasing is important and so I am firmly in the
camp of doing usrmerge in the bootstrap protocol.

We seem to agree on the goal (if by "removing all aliasing" you mean
stop shipping any files in /bin and /lib), but I firmly disagree
with your premise.

Let me elaborate.


So, my understanding of the specific proposal is that:

We in are in category 1:

1. Don't move. We just keep those files that require a particular
   location (such as /bin/sh or the dynamic loader) in their
   non-canonical location. As such, maintainer scripts will be able to
   run and perform the conversion to symbolic links afterwards.

In Helmut's mail to which you are referring, he later explains that
the bootstrap categories come from two orthogonal questions:

Q1. Who is responsible for the top-level symlinks?
Q2. Are there any packages which install files through a top-level
symlink, i.e, an aliased path?

We want the answer of Q2 to be No. The answer to Q1 has, as far as I
can tell, three alternatives:

Q1a. Symlinks are created by the bootstrap tool early on
Q1b. Symlinks are shipped in a package (e.g. base-files) and will
 be created in the initial unpack phase
Q1c. Symlinks are created by a usrmerge conversion, either in a
 maintainer script in the configuration phase or in the final
 cleanup phase by the bootstrap tool

Q1a puts us in Category 4 in Helmut's taxonomy (or as I am going to
call it from now on, HC-4); I'll skip that discussion for now.

Q1c defers the symlink creation to such a late stage that we run
into the ABI problems outlined in other mails; basically, this is
the answer that may force us to ship the dynamic loader in /lib{64,}
and the POSIX shell in /bin forever, and firmly entrench the
usrmerge conversion into our bootstrap process. In my opinion, we
can rule this out completely, because it does not finish the
usrmerge transition as much as hide it away in a dark place where
nobody dares to look. We would end up in HC-1 or HC-3.

Q1b is where it gets interesting, because the way our bootstrapping
protocol currently works, *all* packages will be unpacked first, and
only *then* the configuration phase begins and maintainer scripts
are executed. This means that we *can* move all files to their
canonical paths, because the top-level symlinks will be created
juuust early enough. Depending on the answer to Q2, we end up in
HC-1 or HC-2.

Q1b is my favorite, because it allows us to move the files, finish
the transition, and regain the property that a Debian system is
fully described by package metadata (HC-2).

Q1b has one important failure mode while there is any package left
in the essential set which has not moved its files to the canonical
location: as the initial unpack order is undefined, it is possible
that such a package gets unpacked early enough to create real /bin
or /lib directories, which would then conflict with the subsequently
unpacked top-level symlinks. I think this is the "fragility" other
people keep referring to. Any other usrmerge transition
problem will affect Q1a as much as Q1b, because after the unpack
phase, both are in the exact same state.

The simplest solution for the unpack problem would be to amend the
bootstrap protocol and have base-files be unpacked first. As soon as
all packages have properly transitioned and moved their files to
/usr, we are free to remove that requirement again, because the
unpack order will no longer matter.

Alternatively, we could move all files except for the few critical
ones (/bin/sh, dynamic loader), and then have a coordinated flag day
upload where we add the base-files symlinks *and* move all remaining
files, but this feels quite a bit more fragile to me.


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

> "Sam" == Sam Hartman  writes:

Sam> TL;DR: It looks like if we do not want to encode merged /usr
Sam> into the bootstrap protocol, we must keep some aliases and
Sam> cannot move all files in data.tar.  I think removing all
Sam> aliasing is important and so I am firmly in the camp of doing
Sam> usrmerge in the bootstrap protocol.


I was chatting with Klee Dienes this morning about this issue, and we
had an interesting discussion I'd like to share here.
I want to emphasize that Klee has not been following the discussion, so
if he comes up with interesting ways to think about things, great, but
he's not currently a participant in trying to come to consensus here.

It sounds like we've convinced ourselves we have some sort of technical
debt we're going to need to carry from the usrmerge transition, and we
probably will never be able able to get away from it:

1) In Bootstrap option 3B, we encode doing the usrmerge symlinks into
the bootstrap protocol.  Even if we somehow later create the symlinks in
a data.tar, we need to retain this debt in the bootstrap protocol as
long as we want to be able to bootstrap versions of Debian that do not
encode things in data.tar.

2) If we  take Bootstrap 3C category 1, and some files never get moved
to their canonical path, then we have technical debt we must maintain in
essential packages, basically for ever.

In effect, there is technical debt because the canonical locations of
some files   differs from their canonical interface locations.
For example, /bin/sh will live at /usr/bin/sh.
And as long as that's true (we expect for ever), we have to say that
somewhere.

Okay, so how do we decide where to put technical debt?

Klee argued that one point of lower layers is to make things easier and
cleaner for layers above them.

I.E. there may be some magic in the bootstrap layer and that's good if
it makes things cleaner or more pure for the layer above (the essential
packages).
Similarly, we put complexity in essential packages and make it easier to
write non-essential packages.

We've seen this over the years.
For example, back in the day, bootstrap had to create devices.
It still has to arrange for a /proc and a /sys.
We managed to follow this principle by pushing devices down a layer
further effectively into the kernel and udev, and because of that
bootstrap is simpler than it used to be.

But Klee argued that putting this in bootstrap is nice because you can
handle it in one place rather than spreading it across every essential
package containir a binary that cannot move.

One assumption behind 3C at least initially is that we were avoiding a
combinatorial explosion in changes to bootstrap tool.
That's certainly true in
https://lists.debian.org/20230517093036.ga4104...@subdivi.de

That message assumes 3B is not really an option, and we're looking at
changes to number of bootstrap tools times number of releases of Debian
and all its derivatives.

But as we've explored, we've learned that the combinatorial explosion is
more like 2 (3 if you count cdebootstrap).  Josh has agreed that we can
implement 3B in a small number of lines of code in the bootstrap
implementations, at least if we're willing to require people
bootstrapping pre-stretch to say they do not want merged /usr.

And so, we can concentrate the complexity of our technical debt into a
smaller number of places if we adopt 3B.

I also think it is possible (although unlikely) that as maintainer
scripts in essential packages change over the years, the number of
binaries affected by 3C category 1 may increase.  I.E. we may find
ourselves moving a binary out of /usr back to /bin or being unable to
use that in a future maintainer script.
That seems like a mess to me.
I agree the set of circumstances where we need to do that is small, but
I think itmay be  nonzero.

--Sam


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

TL;DR:
It looks like if we do not want to encode merged /usr into the bootstrap
protocol, we must  keep some aliases and cannot move all files in
data.tar.
I think removing all aliasing is important and so I am firmly in the
camp of doing usrmerge in the bootstrap protocol.

> "Helmut" == Helmut Grohne  writes:

Helmut> Therefore, my premise of us agreeing on shipping the
Helmut> symlinks in base-files likely is wrong despite the number of
Helmut> vocal proponents.

I think we'd like to do that if we can make it work.
I think you convinced us the price of that is too high deep in the
message you quote below.

>> So I think that to argue for 3C you need a specific proposal
>> about what happens.

Helmut> https://lists.debian.org/20230517093036.ga4104...@subdivi.de

Ah thanks.
I will admit that mail didn't get the initial attention it perhaps
deserved.
And even reading it now I don't see a clearly articulated proposal for
3C.
I was put off by a number of things in the mail including the idea that
what we now call 3B would require encoding derivative-specific knowledge
into the bootstrapping protocol.
And so by the time you started talking about specifics of 3C I had
already disagreed with enough of the message that I had tuned out.
(I made those disagreements clear; my suspicion is that others who also
disagreed with other earlier parts of the message tuned out the 3C
specifics.)

So, my understanding of the specific proposal is that:

We in are in category 1:
> 1. Don't move. We just keep those files that require a particular
>location (such as /bin/sh or the dynamic loader) in their
>non-canonical location. As such, maintainer scripts will be able to
>run and perform the conversion to symbolic links afterwards.

That is, there are some essential files that always remain in
non-canonical locations.

You also go through some analysis and make it clear that category 2 is
fragile:

> 2. Move and ship links. Since we unpack all essential data.tar before
>running the first maintainer script, having one package contain the
>compatibility symlinks is enough to fix the problem.

I agree with your analysis that category 2 is fragile.
I definitely had not payed adequate attention to this before, even
though I did read chunks of the message.


My understanding the argument for 3C is roughly the following:

>This gets me to the core of my argument: the way that a Debian chroot should
>look like should be described by the packages that get installed into the
>chroot and not by an outside tool. Yes, an outside tool is needed to do the
>installation but that tool should rely on the package metadata to make choices
>instead of encoding timestamp or release name dependent codepaths.

So it sounds like we are faced with two choices:

1) Keep some aliasing long-term in the data.tar files and get the
bootstrap described by the packages rather than by the bootstrap protocol


2) Change the bootstrap protocol and remove aliasing entirely.

I am firmly in camp 2.  I think that unless we are going to fix dpkg to
support aliasing, we need to remove aliasing entirely.


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Holger Levsen]

On Fri, Jul 07, 2023 at 09:55:05AM +0200, Helmut Grohne wrote:
> Thus far, my impression was that temporarily (<1week, preferably <1day)
> breaking the ability to debootstrap was an acceptable risk and is
> something we experience every now and then anyway (with adduser most
> recently).

actually, python3-mininmal (#1040316) is the most recent case of debootstrap
being broken, not adduser (#1039709). :) AFAIK.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

If we'd ban all cars from cities tomorrow, next week we will wonder why we
waited for so long.


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Simon Richter]

Hi,

On 7/7/23 16:55, Helmut Grohne wrote:


If we have a consensus we're unwilling to wait for a patch, it doesn't
matter whether that's because:



While you try to remove the reasoning for this point of view
from the consensus, the "willing to wait" language implies a reason of
"too slow" already.


The main problem I see with that language is that it is entirely 
passive, as if a patch would appear by waiting for it to do so.


Right now, I seem to be the only person working on one, which is clearly 
ridiculous, as I can invest maybe a total of four hours per week into 
it, so obviously I need to be realistic here and say that it is a bad 
idea for the project to wait for me to implement this, and I don't want 
to be in the position of having the project wait for me either because I 
already have that in 60 hours of my life between Monday and Friday.



We also have quite some disagreement on what "the patch" is in terms of
what semantics would help here.


Yes, that's underspecified. I've tried to get started on this in 
 [1], that is by no 
means complete but it has not generated any comments either.


If there is interest in dpkg changes as a solution, then I could add 
this to the document.



I carefully avoided adding reasoning to the proposed consensus as I was
seeing our disagreement on reasoning. I now see how that second sentence
could be read as precluding a dpkg patch in general. Would it help to
add "+For the purpose of finalizing the transition,+ dpkg is not
augmented ..." to leave open the possibility to add such a change but
still implying that we are not waiting for it to happen?


Yes, that was why I objected to that wording as well. I do agree that we 
need to move forward without waiting for a patch, if only out of the 
self-serving interest that I don't want to sit on the critical path.



I think we need consensus on decisions and confirmation that everyone
feels heard.



Heh. I see how you get there. I agree with that latter part and tried
to use the agreement on problems and mitigations as a vehicle for
ensuring that everyone feels heard. Evidently, that does not work out
either.


It's the best way I can see to reach a consensus, but consensus, like 
voting, can only allocate resources and modify procedures.


If we have a viable technical procedure that finishes the transition, 
then we can get consensus to implement it.


If we have a technical procedure that requires changing a procedure to 
be viable (such as lock-step upgrade of packages in the archive so the 
installer is never presented with an inconsistent set), then we can get 
consensus to modify the procedure and implement the solution.


However, if there are open questions about the technical viability of a 
solution, then the answer to those should be technical as well.


Hence my suggestion to say we have consensus to focus our efforts on the 
solution space that doesn't require a dpkg change -- that is purely an 
allocation of resources, and the remainder of the document is a very 
useful collection of the knowledge we have gathered so far, the avenues 
that have been investigated and possible blockers for those.


We can also try to get consensus or votes on expanding the solution space:

 - mitigations that require archive-side changes, like an upload check 
that a package doesn't trigger a known file loss condition

 - mitigations that require users to upgrade using a special procedure
 - mitigations that (further) violate policy by e.g. manipulating the 
dpkg database from the usrmerge package, or wrapping dpkg in a diversion
 - mitigations that require updating a stable release, such as shipping 
an updated debootstrap package


I'm not sure all of these would be unanimous, but if there is a 
technical solution that becomes viable through something we can vote on, 
that is completely valid.



In any case, the rough consensus on moving forward without major changes
to dpkg (for reasons we disagree about) paves the road for selecting a
subset of mitigations and proposing that as decision. The major missing
piece to perform this selection is an agreement on how we protect
aliasing symlinks from deletion (aka P9), because we still have quite
strong disagreement on the mitigations selected there.


The questions I see are

 - Can we get a full column of checkmarks by any combination of 
mitigations?

 - Are there procedural mitigations that can be useful?
 - Are these mitigations mutually compatible?

   Simon

[1] https://lists.debian.org/debian-dpkg/2023/06/msg00050.html

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Sam,

On Thu, Jul 06, 2023 at 01:51:04PM -0600, Sam Hartman wrote:
> BUT I don't think it matters.
> If we have a consensus we're unwilling to wait for a patch, it doesn't
> matter whether that's because:

Indeed, this is not how I looked at it. It also is a view that I don't
subscribe to, because I think commercial backing of the issue changes
the equation. So if I were to see changing dpkg as a viable way now (I
did earlier), I would be willing to wait for it, because we have
recently made significant progress on defining what those semantics
should be.  While you try to remove the reasoning for this point of view
from the consensus, the "willing to wait" language implies a reason of
"too slow" already.

> 1) Some of us think the patch would be a bad idea

Additionally, people disagree on why it is a bad idea.

> 2) Some of us think the patch will not happen because of politics
> 
> 3) Some of us think the patch won't happen because no one cares enough
> to write it
> 
> 4) Some of us think the patch will eventually get done
> 
> 5) Some of us think the problem is too constrained and if we really
> wanted to make progress we could incrementally move toward it.

We also have quite some disagreement on what "the patch" is in terms of
what semantics would help here.

> Helmut effectively asked us to agree with 1.

I disagree here. For reference, I am quoting the proposed consensus
item:

| The primary method for finalizing the /usr-merge transition is
| moving files to their canonical locations (i.e. from / to /usr)
| according to the dpkg fsys database (i.e. in data.tar of binary
| packages).  dpkg is not augmented with a general mechanism
| supporting aliasing nor do we encode specific aliases into dpkg.

I carefully avoided adding reasoning to the proposed consensus as I was
seeing our disagreement on reasoning. I now see how that second sentence
could be read as precluding a dpkg patch in general. Would it help to
add "+For the purpose of finalizing the transition,+ dpkg is not
augmented ..." to leave open the possibility to add such a change but
still implying that we are not waiting for it to happen?

> And I don't think there is a consensus on this.

Even though I disagree on that's what I asked for, I agree that we don't
have consensus on patching dpkg being a bad idea in general.

So how do we get towards an agreeable consensus item? Evidently, the one
I proposed does not work out and the "willing to wait" variant you
proposed garners rough consensus, but not more. Would someone else be
able to propose wording that passes muster?

> 
> 
> I have reviewed the  DEP 17 draft at
> https://subdivi.de/~helmut/dep17.html
> 
> 
> Helmut asked for consensus on   the problems and mitigations or at least
> I think he did.
> I think we don't need that.
> I think we need consensus on decisions and confirmation that everyone
> feels heard.

Heh. I see how you get there. I agree with that latter part and tried
to use the agreement on problems and mitigations as a vehicle for
ensuring that everyone feels heard. Evidently, that does not work out
either.

In any case, the rough consensus on moving forward without major changes
to dpkg (for reasons we disagree about) paves the road for selecting a
subset of mitigations and proposing that as decision. The major missing
piece to perform this selection is an agreement on how we protect
aliasing symlinks from deletion (aka P9), because we still have quite
strong disagreement on the mitigations selected there.

> WRT the problems, I confirm that the list of problems does (in my
> reading) accurately describe the problems people have brought up.

Thank you!

> I don't think we have (or should try to get) a consensus on which
> problems need to be fixed except in so far as that affects our consensus
> on a proposal.

I was trying to imply that we need to address (more or less) all of
these nine problems. I say address rather than fix, because we may
choose to only fix them in certain environments and skip others (e.g.
derivatives, addon repositories, backports, skip upgrades etc.).

> I will admit that even though I've followed the discussion fairly
> closely, I don't have a good feeling about the mitigations.
> 
> I think that once a reasonable set of the mitigations have been applied,
> we'll be in a reasonably good place.
> 
> My concern is about upgrades and about unstable.
> I would like to see a set of instructions that I could follow for moving
> files in my packages in the data.tar to their canonical locations.

To me, that set of instructions is a later step, because those
instructions strongly depend on the selection of the mitigations and the
selection varies wildly with our disagreement of symlink protection. If
I were to present instructions now (one way or another), people would
rightfully disagree (different ones depending on the selection).

> I'd like instructions that clearly allowed me to 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

> "Russ" == Russ Allbery  writes:

Russ> Helmut Grohne  writes:

Russ> What Sam is trying to
Russ> say, I think, is that a different phrasing offers a way to
Russ> avoid that discussion and agree to disagree on the best
Russ> architecture in the abstract by pointing out an additional
Russ> constraint: how long we're willing to wait and how
Russ> uncomfortable we're willing to make things for ourselves to
Russ> try to force the dpkg changes to appear.


Exactly.
For example, in response to my rephrasing Helmut asserted (correctly of
course) that there is no patch ready today.
If we go down that road, we can repeat the discussion about whether the
politics are a significant factor in why we have no patches today.
I appreciate Helmut has a strong position on that issue,  but some of us
disagree with Helmut strongly on that point.

BUT I don't think it matters.
If we have a consensus we're unwilling to wait for a patch, it doesn't
matter whether that's because:

1) Some of us think the patch would be a bad idea

2) Some of us think the patch will not happen because of politics

3) Some of us think the patch won't happen because no one cares enough
to write it

4) Some of us think the patch will eventually get done

5) Some of us think the problem is too constrained and if we really
wanted to make progress we could incrementally move toward it.

Helmut effectively asked us to agree with 1.
And I don't think there is a consensus on this.



I have reviewed the  DEP 17 draft at
https://subdivi.de/~helmut/dep17.html


Helmut asked for consensus on   the problems and mitigations or at least
I think he did.
I think we don't need that.
I think we need consensus on decisions and confirmation that everyone
feels heard.

WRT the problems, I confirm that the list of problems does (in my
reading) accurately describe the problems people have brought up.
I don't think we have (or should try to get) a consensus on which
problems need to be fixed except in so far as that affects our consensus
on a proposal.

I will admit that even though I've followed the discussion fairly
closely, I don't have a good feeling about the mitigations.

I think that once a reasonable set of the mitigations have been applied,
we'll be in a reasonably good place.

My concern is about upgrades and about unstable.
I would like to see a set of instructions that I could follow for moving
files in my packages in the data.tar to their canonical locations.

I'd like instructions that clearly allowed me to reason about
upgrades, and about how my packages interact with other packages during
the transition.
Because several of the problems look kind of serious, and my reading of
the mitigations is that hey, by the time we get done, it'll all be okay.
Implicit is the idea that because of the moratorium these problems are
rare.
Except during the transition, there will be no moratorium, so perhaps
these problems will not be rare.

There are two many mitigations, and the interactions matter for me to
think about safety.

Proposed way to address this concern:

A) Develop a proposal assuming no dpkg changes for moving files too
canonical locations in data.tar.
Assume bootstrap protocol 3B (assume bootstrap creates the initial
symlinks).  I will talk about expanding to bootstrap protocol 3C in a bit.

B) Develop an argument for the safety of that proposal.

C) Let's review and agree to that.

D) Then think about bootstrap protocol 3C; see below.



Regarding bootstrapping.

I think Helmut and Josh have convinced me that my proposal for bootstrap
protocol 3B is viable.
We could do that if we want to.

I do not find having more complexity for bootstrapping pre-stretch, or
for having the bootstrap tools be required to have a bit of special
knowledge to be blocking bugs.
If we could make 3C work, I would not object to it.
I think Josh has argued that 3C is a nice to have--some day.

I do not see an argument that 3C is safe though.
Luca is right that if it breaks unstable or breaks upgrades, it will be
really bad.

So I think that to argue for 3C you need a specific proposal about what
happens.
And you need a really strong argument that implementing that proposal is
safe for upgrades and safe during the transition.
It sounds like part of the argument of safety during transitions is that
you will coordinate with ftpmaster to do some essential packages in
lock-step.
If you can show why that can be implemented and is safe, that's fine.

But because we have another option (3B) that is viable, the safety
argument needs to meet a high bar.  If people like Luca are saying they
are not convinced after reading it, even if they cannot articulate
specific concerns, I would consider that blocking.

Put another way.  We have to do something about moving most data to
canonical paths.  We need a safety argument for that of course.  The
review criteria there would be at 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Simon,

On Fri, Jun 30, 2023 at 08:06:15PM +0900, Simon Richter wrote:
> I think "backports" are missing as a user story.

I fully agree. What a serious omission. As a first step, I have updated
DEP17 to indicate which mitigations happen to work when being
backported. For instance, changing Replaces to Conflicts is something
that happens to just work for backports. Diverting dpkg-divert less so.

So in effect, the most likely outcome is that backports become very
fragile, because they essentially have to undo all the moves performed
in unstable. What we can do with relative ease is detect the problems
after they have been introduced.

I note that the quality of backports already leaves wishes. For
instance, kodi-addons-dev takes over /usr/bin/dh_kodiaddon_depends in
bullseye-backports from kodi-addons-dev-common in bullseye without
conflicts nor replaces. Likewise, nfs-ganesha-ceph takes over
/usr/lib/ganesha/libganesha_rados_*.so in bullseye-backports from
nfs-ganesha in bullseye.

Is making backports more fragile a reasonable trade-off for moving
forward?

> Most packages should be harmless, and the Contents file for
> bullseye-backports doesn't have too much in any of the affected directories.

Yeah, the number of affected cases should be relatively low, but when
things go wrong, it's bad. Is that ok if we can automatically detect it
(after it happened)?

> but the list of packages installing files into /lib is longer and includes
> all the kernel backports, so I guess that is another potential source of
> problems.

Without having looked, I'd expect the majority of practically affected
files below /lib to be systemd units and udev rules. These should not be
moved in backports, and as long as debhelper is being used, that might
just work.

> There might be an easy solution here, I have not investigated this very
> deeply because it is a workday and 11 hours out of every day are already
> spoken for.

I don't think there is an easy solution, but maybe it happens to work by
chance 90% (number made up) of the time and we shall be able to detect
the remaining 10%.

> > Stating a goal has been quite difficult, but I think that most of the
> > participants agree that we want to eliminate the file move moratorium
> > without getting problems in return.
> 
> I'd even widen that to "no more special handling needed in any place for the
> transition", with the moratorium being an example of that.

What other kind of special handling do you have in mind? I probably
agree.

> > When we get into mitigations, consensus is hard to come by. My
> > impression is that we have roughly two camps. One is in favour of major
> > changes to dpkg and the other is not.
> 
> It's difficult to summarize the situation in one sentence, because neither
> group is really objecting to dpkg changes, so I'd put the fault line at
> whether the transition should be facilitated through dpkg or not.

It's not clear to me, whether you'd consider M3 (a minor and revertable
mitigation in dpkg) to be covered by this or not.

> >   * Even if dpkg were fixed in trixie, trixie's libc6 would still be
> > unpacked using bookworm's dpkg. At least for this package, we cannot
> > rely on dpkg changes in trixie. Therefore we need workarounds or
> > spread the transition to forky. For other packages, even a
> > Pre-Depends on dpkg is not a guarantee that a changed dpkg will
> > perform the unpack.
> >   * Changes to dpkg will not fix bootstrapping.
> 
> The dpkg changes will fix bootstrapping, but we can't finish the transition
> until forky this way, because we need to be able to bootstrap with a libc6
> package that can be installed from bookworm.

Can you elaborate why? As far as I can see, debootstrap may perform the
initial unpack without help from dpkg. Then we invoke the unpakced dpkg
to configure essential packages. If dpkg plays any role in setting the
aliasing symbolic links, their creation will be late for running dpkg.
Maybe I'm missing something?

> We need to be careful here to not conflate the goal of the transition with
> the method for reaching it. We have consensus on the goal (basically,
> data.tar and filesystem matching is the definition of "done" I use for this
> transition).

I agree that we need to be careful about that conflation, because I
think you are conflating them. I disagree that the goal is having
data.tar and filesystem match up. You earlier argued that we are done
when special handling is no longer necessary and I see that as the goal.
Having all the files moved is one method to get there. We also seem to
have rough consensus that this is the preferre method.

> We do not have consensus on the technical implementation because there are
> people who believe the technical implementation proposed is not actually
> feasible. In my opinion, it is a 95% solution, which is very tempting but we
> need the remaining 5% as well. To a large extent, we are having this
> discussion because the usrmerge package 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Simon Richter]

Hi Helmut,

On 6/29/23 04:37, Helmut Grohne wrote:


Consensus proposal #1:



 This updated DEP17 is a complete representation of the known and
 relevant problems and known mitigations under discussion at the time
 of this writing.



Do you miss a related problem important to you? Do you miss your
preferred mitigation? Please speak up, so we can record it.


I think "backports" are missing as a user story.

Most packages should be harmless, and the Contents file for 
bullseye-backports doesn't have too much in any of the affected directories.


For /bin and /sbin, the list is short:

admin/brltty
admin/btrfs-progs
admin/e2fsck-static
admin/e2fsprogs
admin/glusterfs-client
admin/logsave
admin/mdadm
admin/moosefs-client
admin/systemd
admin/systemd,admin/systemd-standalone-sysusers
admin/systemd,admin/systemd-standalone-tmpfiles
admin/systemd-container
admin/systemd-resolved
admin/systemd-sysv
admin/udev
net/iproute2
net/wpasupplicant

but the list of packages installing files into /lib is longer and 
includes all the kernel backports, so I guess that is another potential 
source of problems.


There might be an easy solution here, I have not investigated this very 
deeply because it is a workday and 11 hours out of every day are already 
spoken for.



Stating a goal has been quite difficult, but I think that most of the
participants agree that we want to eliminate the file move moratorium
without getting problems in return.


I'd even widen that to "no more special handling needed in any place for 
the transition", with the moratorium being an example of that.



When we get into mitigations, consensus is hard to come by. My
impression is that we have roughly two camps. One is in favour of major
changes to dpkg and the other is not.


It's difficult to summarize the situation in one sentence, because 
neither group is really objecting to dpkg changes, so I'd put the fault 
line at whether the transition should be facilitated through dpkg or not.



  * Even if dpkg were fixed in trixie, trixie's libc6 would still be
unpacked using bookworm's dpkg. At least for this package, we cannot
rely on dpkg changes in trixie. Therefore we need workarounds or
spread the transition to forky. For other packages, even a
Pre-Depends on dpkg is not a guarantee that a changed dpkg will
perform the unpack.
  * Changes to dpkg will not fix bootstrapping.


The dpkg changes will fix bootstrapping, but we can't finish the 
transition until forky this way, because we need to be able to bootstrap 
with a libc6 package that can be installed from bookworm.


The subset of affected packages is rather small though.


There also is a minority arguing in favour of doing both. I've kinda
ruled out that option already as we get the downsides of both without
any further benefit in return.


I think that was me being pessimistic and expecting that we're going to 
end up doing both anyway. The downsides of the combination approach are 
more severe than those of either solution in isolation, because dpkg 
incurs a permanent and significant performance penalty to verify its 
database against reality, and gains additional error states.


So unless there is someone else, I'd call it a stretch to say that 
someone is "in favour" of this solution.



Consensus proposal #2:



 The primary method for finalizing the /usr-merge transition is
 moving files to their canonical locations (i.e. from / to /usr)
 according to the dpkg fsys database (i.e. in data.tar of binary
 packages).  dpkg is not augmented with a general mechanism
 supporting aliasing nor do we encode specific aliases into dpkg.



I recognize that this is not unanimous, but I think we still have
sufficient consensus on this. I suspect that maybe Simon Richter and a
few more would disagree here. If consensus fails, we may have to put
this to a vote.


We need to be careful here to not conflate the goal of the transition 
with the method for reaching it. We have consensus on the goal 
(basically, data.tar and filesystem matching is the definition of "done" 
I use for this transition).


We do not have consensus on the technical implementation because there 
are people who believe the technical implementation proposed is not 
actually feasible. In my opinion, it is a 95% solution, which is very 
tempting but we need the remaining 5% as well. To a large extent, we are 
having this discussion because the usrmerge package left us with a 90% 
solution.


All a vote can achieve in that situation is assign responsibility, and 
authorize more drastic measures such as bypassing regular procedures 
(e.g. the "flag day" transition proposal for bootstrapping). Unless the 
vote clarifies which of those it is, we will have disagreement on the 
scope, which would not be much of an improvement over the status quo 
where there is disagreement on the implications of the TC decision.


The minimal scope would be one where no actions are authorized, 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Fri, 30 Jun 2023 at 06:31, Simon Richter  wrote:
>
> Hi,
>
> On 6/30/23 02:32, Helmut Grohne wrote:
>
> >> If the dpkg maintainer were to merge aliasing support, I haven't seen
> >> anyone objecting strong enough to try and override that maintainer
> >> action for example.
>
> Correct. The proponents of the "work around dpkg" approach quite often
> insinuate that it would be impossible to get changes merged for "social
> reasons", but there is no opposition from either camp to changing dpkg.

And yet, ~10 years on and not only there are zero actual results to
show for it, but every single time the topic is brought up on the dpkg
mailing list, the maintainer's answer is some variation of "no, get
lost". Of course it's your time, so you are entitled to use it as you
see fit.

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Simon Richter]

Hi,

On 6/30/23 02:32, Helmut Grohne wrote:


If the dpkg maintainer were to merge aliasing support, I haven't seen
anyone objecting strong enough to try and override that maintainer
action for example.


Correct. The proponents of the "work around dpkg" approach quite often 
insinuate that it would be impossible to get changes merged for "social 
reasons", but there is no opposition from either camp to changing dpkg.



I think this is a misrepresentation. There is no readily mergable patch
for aliasing support. The most complete one is the tree from Simon
Richter and he considers that work incomplete still. At this time, it's
no a question of merging or not really.


Indeed. I'd place progress at 10%, with 80% being "call for testers", 
90% "mergeable", 95% "translations done" and 100% "Policy updated."


I feel confident that this approach works, but there is a lot of boxes 
to tick, like diversions and alternatives, it will require a massive 
test suite, and it will require documentation as there is a new admin tool.


   Simon

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Josh Triplett]

Russ Allbery wrote:
> This is more of a high-level design intuition that stems from some basic
> architectural principles, such as "dpkg should be the authority for what
> Debian installs on the file system so that it can ensure global
> consistency."
> 
> But to give you a concrete answer, here are the sorts of things that come
> to mind even after this transition is complete in Debian:
> 
> * Old packages that still use aliased paths installed on current systems.
> * Locally-built packages with /bin paths in data.tar.gz.
> * Weird local symlinks to move files around in the file system.
> * Some unforseen future transition where we want to merge two directories.

Indeed. In the history of Debian, we've had
- The /usr/doc -> /usr/share/doc transition
- The /usr merge
- Various package-specific transitions from an old directory to a new,
  where it'd be convenient for the package to install a transitional
  symlink rather than support both paths internally
- Local sysadmins who might want to move something to a different
  directory (whether for space constraints or local package
  organization) and have dpkg actually understand what they're doing
- In the future, I wouldn't be surprised if someone *attempts* to merge
  /usr/sbin -> /usr/bin one day.

It'd be incredible if those were as simple as building and installing a
package that installs a symlink in its data.tar where other packages
install a directory. (And possibly including some metadata that says
"yes, I really do mean to install a symlink redirect for a directory,
this isn't just a file conflict mistake".)

> I think we have an entire project full of people with expert skills,
> including some truly impressive C programmers, and I find our inability to
> muster those resources to improve *the* foundational program on which the
> work of the entire distribution rests on, in a way that meets a very high
> quality bar and is sustainable for the project, to be disappointing.  I'm
> not assigning blame; I'm simply saying that it makes me sad that the
> assumption is that only Guillem alone is able to work on this project and
> therefore its success is constrained by his available time.

I agree with you that this is disappointing.

> I have some concerns that Guillem is trying to boil the ocean and he's
> missing some viable incremental approach, but this is only a vague concern
> and I absolutely do not have the expertise in dpkg to argue that case.
> Clearly Guillem has convinced himself that it is the correct approach and
> he is an expert in this area, so he is probably correct and I am probably
> wrong.

I think this paragraph very much captures why a project with many expert
developers in a variety of different areas believes (whether correctly
or incorrectly) that their help would not be welcomed. (And to be clear,
I really do mean "correctly or incorrectly" here; please don't take this
as presupposing the answer to that question.)

dpkg is trying to solve an *incredibly* complex problem, with very
demanding constraints, and decades of historical expectations and
compatibility requirements, in addition to added constraints from
possible maintainer preferences and tastes. And as a project, we have
not solved the problem of either documenting or simplifying enough of
those to the point of inviting contribution.

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Russ Allbery]

Helmut Grohne  writes:

> To me, this leaves more question marks than earlier. What applications
> of aliasing do you envision that would benefit here? Anything concrete?

So, I do want to stay focused on Sam's point, which is that we should
avoid this specific argument by noting that we're not going to get dpkg
aliasing support within a workable length of time anyway, given where
we're at today.  I'm therefore not willing to put in the effort required
to deeply flesh out use cases, and I'm sure my initial thoughts are both
incomplete and inaccurate.

This is more of a high-level design intuition that stems from some basic
architectural principles, such as "dpkg should be the authority for what
Debian installs on the file system so that it can ensure global
consistency."

But to give you a concrete answer, here are the sorts of things that come
to mind even after this transition is complete in Debian:

* Old packages that still use aliased paths installed on current systems.
* Locally-built packages with /bin paths in data.tar.gz.
* Weird local symlinks to move files around in the file system.
* Some unforseen future transition where we want to merge two directories.

More generally, the way I am thinking about it from a high level is that
this transition has uncovered problems with dpkg's understanding of
symlinks.  (Well, uncovered for some of us who weren't paying close
atention; I'm sure Guillem has been aware of them for eons.)  Its model of
the world is too simple; there are POSIX file system semantics that it
does not understand and therefore mishandles.  Ideally we would fix this.

> Why did you see waiting for a dpkg patch as a reasonable approach
> earlier?

Because I thought it would be easier than it turns out to be.

> I think we have roughly three categories of dpkg changes on the table:
>  * Minimal hacks that would paper over some of the effects without
>causing a generally applicable aliasing support
>  * A mechanism where dpkg is being told about aliasing symlinks
>explicitly and thus would work with them
>  * An extension of the dpkg filesystem database wherein it would be able
>to determine the filetype for every filesystem object. In particular,
>this would allow it to tell directories from symlinks apart and
>handle the overlap in a meaningful way.

> Which of these do you consider "aliasing support"?

The second and the third.

> The third category is quite involved. Since it changes the internal
> layout of dpkg's database, a prerequisite for doing this is removing
> direct accesses to the database by other tools. Guillem has been doing
> that work and you can find more information about it at
> https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking. Note that this
> work goes beyond what is needed for aliasing as it also considers
> extending the metadata format in binary packages to be able to represent
> e.g. extended attributes.

Indeed, this is the point.  This is exactly what Sam and I are both
saying: a good architectural solution is going to take long enough that we
can't block this transition on it, and therefore debating whether or not
that solution is desirable is beside the point and is just creating more
things to argue about without resolving the problem in front of us.

> Did you expect Guillem to implement this faster?

I think we have an entire project full of people with expert skills,
including some truly impressive C programmers, and I find our inability to
muster those resources to improve *the* foundational program on which the
work of the entire distribution rests on, in a way that meets a very high
quality bar and is sustainable for the project, to be disappointing.  I'm
not assigning blame; I'm simply saying that it makes me sad that the
assumption is that only Guillem alone is able to work on this project and
therefore its success is constrained by his available time.

You will note that I am not ramping up on dpkg development despite being
an expert C programmer.  I really am not assigning blame.  I'm sad about a
lot of things that I could be doing and currently am not.

> The second category seems fairly narrow and tailored to the /usr-merge
> application. Do you consider it to support aliasing in the generic way
> that you had in mind?

I didn't have a specific generic way in mind.  I think it would be a
cleaner architecture than what we're going to do instead, but not as clean
as what Guillem is working on.

I have some concerns that Guillem is trying to boil the ocean and he's
missing some viable incremental approach, but this is only a vague concern
and I absolutely do not have the expertise in dpkg to argue that case.
Clearly Guillem has convinced himself that it is the correct approach and
he is an expert in this area, so he is probably correct and I am probably
wrong.

> Since Guillem is working on the third category, whom did you imagine to
> be working on the second one?

The answer to that question that I have been pushing 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Thu, 29 Jun 2023 at 13:34, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Thu, Jun 29, 2023 at 12:49:16PM +0100, Luca Boccassi wrote:
> > Essentially, this boils down to risks vs benefits. The risk of going
> > 3c is that every single Debian installation in existence breaks in
> > some interesting ways, as fixing the bootstrapping corner case is
> > delegated to the package upgrade workflow. The sole benefit is that
> > one of the two bootstrapping tools in widespread use keeps its
> > internal code a bit 'cleaner' from the point of view of some
> > technically unnecessary and self-imposed design constraints (yes
> > there's 2 more tools as pointed out yesterday, but they appear to be
> > at least under-maintained judging from tracker.d.o).
>
> I'm not sure you understand what 3c is about. I think it is safe to say
> that you are in favour of moving all of the files to their canonical
> location (i.e. from / to /usr). This is half of the picture for 3c. The
> other half is shipping the symbolic links in base-files rather than
> having them created in some way not tracked by dpkg. If you plug these
> two together, you have made /usr-merged bootstrapping work without
> having changed the protocol.
>
> So what is the risk involved here? I think there are three main risk
> categories at play:
>
> 1. The risk of effects from moving files from / to /usr. This is a risk
>that you clearly see as worth taking regardless of the bootstrap
>case.
>
> 2. The risk of effects from shipping the symbolic links in base-files. I
>see that you'd rather not do this, but not shipping them in any
>package poses a deletion risk of its own, so shipping them
>effectively is a risk mitigation and is what allows us to drop
>protective diversions eventually. It stills risks breaking
>debootstrap's behind-the-back approach of merging, so we'll likely
>have to do a stable upload of debootstrap.
>
> 3. The risk of unstable becoming temporarily non-bootstrappable. This is
>where I see the main fragility of the approach. As is evident from
>your next paragraph, you don't really care about this either.
>
> Given this, it seems rather evident that you have a different risk in
> mind that I do not see at this time. Can you elaborate?

The risk to unstable is not just to become unbootstrappable, but
completely broken, for every installation, because of the required
song with the essential set in order to allow for base-files to
ship symlinks, that needs to happen perfectly and flawlessly,
everywhere, always, for all times. And not just in unstable, but for
all upgrades from bookworm to trixie too, as they'll have to perform
the same song too. This is way worse than the usrmerge package
transition: at least there we can do some pre-flight checks, and abort
if unworkable conditions are found, without touching anything. As far
as I can see such a failsafe just isn't there with this approach.
The opposite side of the coin is that unstable (and only unstable) new
chroots cannot be created for a day or two.

It seems to me the category and magnitude of risk of these two options
are not even remotely comparable.

> Then, software maintainers tend to say "no" when a feature poses a
> non-trivial cost to permanent maintenance. We see this all the time and
> you shrug it off, because it's not your package. However, when people do
> the reverse (e.g. diverting systemd's units poses a non-trivial
> maintenance cost to systemd), you take it for granted that you can
> unilaterally say "no". Why is it ok for you to say "no", but not for
> other maintainers to say "no"?

The difference is that what you are mentioning here is just broken and
unworkable, it's not just "maintenance cost", although that's a
factor. On the other hand having a bootstrap protocol is not broken,
it's how things work and how they will always work. How much work said
protocol needs to do is a design choice, and opinions differ, but
neither of the two options can be categorized as outright broken.

> > I don't see how it's worth the risk. This is essentially a problem in
> > the bootstrapping tools, so solving it in the bootstrapping tools is
> > not only the safest approach - worst case scenario, creating a new sid
> > chroot might not work for a couple days, not a big deal given it
> > happens all the time for various reasons as we've seen this week -
> > it's also the right approach.
>
> You seem to have missed Johannes reasoning entirely. He sees Debian as a
> component-based system. He argues that this is not a problem in the
> bootstrapping tools, but a problem in the components being bootstrapped.
> In effect, the usrmerge binary package currently implements it in a
> component-oriented way. Since it is a problem with that component,
> solving it there makes most sense, no? That alone makes it obvious, that
> this is not a problem limited to bootstrapping. We have now duplicated
> this mechanism to usrmerge and debootstrap and you are 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Russ,

On Thu, Jun 29, 2023 at 11:51:57AM -0700, Russ Allbery wrote:
> I think I fall into the category that Sam is thinking of.  I don't agree
> that aliasing support in dpkg is useful only for this transition.  I think
> there are other cases where two directories could end up being aliases for
> the same directory.  However, I have been convinced that changing dpkg to
> properly support this will take longer than I'm willing to wait given the
> problems that the /usr-merge transition is causing right now, and
> therefore I agree with a consensus that we shouldn't wait for dpkg
> aliasing support (even though I disagree, I think, about the long-term
> usefulness of that support).

To me, this leaves more question marks than earlier. What applications
of aliasing do you envision that would benefit here? Anything concrete?

Why did you see waiting for a dpkg patch as a reasonable approach
earlier? I think we have roughly three categories of dpkg changes on the
table:
 * Minimal hacks that would paper over some of the effects without
   causing a generally applicable aliasing support
 * A mechanism where dpkg is being told about aliasing symlinks
   explicitly and thus would work with them
 * An extension of the dpkg filesystem database wherein it would be able
   to determine the filetype for every filesystem object. In particular,
   this would allow it to tell directories from symlinks apart and
   handle the overlap in a meaningful way.

Which of these do you consider "aliasing support"? I expect that you
would not consider the first category. The third category is quite
involved. Since it changes the internal layout of dpkg's database, a
prerequisite for doing this is removing direct accesses to the database
by other tools. Guillem has been doing that work and you can find more
information about it at
https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking. Note that this
work goes beyond what is needed for aliasing as it also considers
extending the metadata format in binary packages to be able to represent
e.g. extended attributes. Did you expect Guillem to implement this
faster? The second category seems fairly narrow and tailored to the
/usr-merge application. Do you consider it to support aliasing in the
generic way that you had in mind? Since Guillem is working on the third
category, whom did you imagine to be working on the second one?

> I am very disappointed that we have not successfully added aliasing
> support to dpkg, which to me is the obviously correct way of addressing
> this problem architecturally.  To me, this says sad and depressing things
> about the state of the project that we have not been able to do this.
> What Sam is trying to say, I think, is that a different phrasing offers a
> way to avoid that discussion and agree to disagree on the best
> architecture in the abstract by pointing out an additional constraint: how
> long we're willing to wait and how uncomfortable we're willing to make
> things for ourselves to try to force the dpkg changes to appear.

Please bear in mind that a general form of aliasing support entails all
sorts of crazy corner cases. In the earlier thread, Simon Richter
highlighted some of them. We'd need to come up with reasonable semantics
for the case where a package wants to install a symbolic link to a
location that is already occupied with a directory for instance. We also
need to figure out what happens when we remove that package. What
happens when the symbolic link is diverted? What happens if the target
of an aliasing symlink itself is an aliasing symlink? In assuming that
files have one and only one canonical location, we are saved from this
complexity on multiple levels. Why do you see adding such complexity as
obviously correct?

The more I read your and Sam's mails, the more I have the impression
that I miss some important aspect.

Helmut

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Thu, 29 Jun 2023 at 20:39, Simon McVittie  wrote:
>
> On Thu, 29 Jun 2023 at 19:32:15 +0200, Helmut Grohne wrote:
> > I think
> > that we will have to touch debootstrap in any case. If you specify
> > --variant=buildd, you get an unmerged chroot even when you do it on
> > trixie or unstable.
> ...
> > our buildds, which are still unmerged and get
> > debootstrapped as unmerged once a week
>
> This certainly seems obviously wrong, because unmerged /usr was
> special-cased to be "almost supported" for bookworm-based environments
> (unsupported as an end state for user systems that will get upgraded
> to trixie, but still required to work well enough for autobuilders,
> for QA tools and during the upgrade from bullseye), but in trixie that
> no longer applies and a package that only works on merged-/usr systems
> would no longer be considered to have a bug.
>
> (At least, that's my reading of what the technical committee has said,
> and certainly it's what I intended us to be saying when drafting #994388!)
>
> I think we probably want buildd chroots to stay unmerged-/usr for:
>
> - bullseye and older (including -security, -proposed-updates etc.)
> - bullseye-backports
> - bookworm (including -security, -proposed-updates etc.?)
>
> but transition to merged-/usr for:
>
> - bookworm-backports
> - trixie and newer
> - unstable
> - experimental
>
> smcv
> (speaking here as an individual DD and not on behalf of the TC)

The agreement last year was that we would wait for the tech committee
indication before changing the buildds. If there is such a decision, I
can start working on that.

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Simon McVittie]

On Thu, 29 Jun 2023 at 19:32:15 +0200, Helmut Grohne wrote:
> I think
> that we will have to touch debootstrap in any case. If you specify
> --variant=buildd, you get an unmerged chroot even when you do it on
> trixie or unstable.
...
> our buildds, which are still unmerged and get
> debootstrapped as unmerged once a week

This certainly seems obviously wrong, because unmerged /usr was
special-cased to be "almost supported" for bookworm-based environments
(unsupported as an end state for user systems that will get upgraded
to trixie, but still required to work well enough for autobuilders,
for QA tools and during the upgrade from bullseye), but in trixie that
no longer applies and a package that only works on merged-/usr systems
would no longer be considered to have a bug.

(At least, that's my reading of what the technical committee has said,
and certainly it's what I intended us to be saying when drafting #994388!)

I think we probably want buildd chroots to stay unmerged-/usr for:

- bullseye and older (including -security, -proposed-updates etc.)
- bullseye-backports
- bookworm (including -security, -proposed-updates etc.?)

but transition to merged-/usr for:

- bookworm-backports
- trixie and newer
- unstable
- experimental

smcv
(speaking here as an individual DD and not on behalf of the TC)

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Russ Allbery]

Helmut Grohne  writes:
> On Wed, Jun 28, 2023 at 02:55:28PM -0600, Sam Hartman wrote:

>> I think you might get a more clear consensus if you phrase that in
>> terms of whether people are willing to wait for major changes in dpkg.
>> If the dpkg maintainer were to merge aliasing support, I haven't seen
>> anyone objecting strong enough to try and override that maintainer
>> action for example.

> I think this is a misrepresentation. There is no readily mergable patch
> for aliasing support. The most complete one is the tree from Simon
> Richter and he considers that work incomplete still. At this time, it's
> no a question of merging or not really.

> From my point of view, the question really is whether we want to
> permanently extend dpkg's features in a way that we only really see
> useful for one transition while still having to maintain it for
> backwards compatibility.

I think I fall into the category that Sam is thinking of.  I don't agree
that aliasing support in dpkg is useful only for this transition.  I think
there are other cases where two directories could end up being aliases for
the same directory.  However, I have been convinced that changing dpkg to
properly support this will take longer than I'm willing to wait given the
problems that the /usr-merge transition is causing right now, and
therefore I agree with a consensus that we shouldn't wait for dpkg
aliasing support (even though I disagree, I think, about the long-term
usefulness of that support).

I am very disappointed that we have not successfully added aliasing
support to dpkg, which to me is the obviously correct way of addressing
this problem architecturally.  To me, this says sad and depressing things
about the state of the project that we have not been able to do this.
What Sam is trying to say, I think, is that a different phrasing offers a
way to avoid that discussion and agree to disagree on the best
architecture in the abstract by pointing out an additional constraint: how
long we're willing to wait and how uncomfortable we're willing to make
things for ourselves to try to force the dpkg changes to appear.

> Can we maybe turn the question upside down and phrase it in terms of how
> to place the aliasing symlinks?

> Option #3d:

> A binary package (such as base-files) shall contain the aliasing
> symlinks in its data.tar.

> Option #3e:

> No package shall contain the aliasing symbolic links in a data.tar
> and dpkg shall not track them in its filesystem database.

I really would like us to work towards #3d in the longer term.  I'm
convinced by josch's argument here.  But I would be willing to go with a
short-term consensus around doing #3e for now, as long as we're not
foreclosing on doing this properly via #3d in the future.

(I think there are some reasons to believe that once most of the file
moves in data.tar.gz have happened, the ones that are currently blocked by
not having a solution for this transition, it will be easier to get to #3d
from #3e than it is right now.)

-- 
Russ Allbery (r...@debian.org)  

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Sam,

On Wed, Jun 28, 2023 at 02:55:28PM -0600, Sam Hartman wrote:
> I have read the mail, not the full updated DEP, so I cannot yet ack
> this.

Hmm. Do you intend to do that? If you are short on time, I think the
problem section is more important than the mitigation section.

> Helmut> When we get into mitigations, consensus is hard to come
> Helmut> by. My impression is that we have roughly two camps. One is
> Helmut> in favour of major changes to dpkg and the other is not.
> 
> I think you might get a more clear consensus if you phrase that in terms
> of whether people are willing to wait for major changes in dpkg.
> If the dpkg maintainer were to merge aliasing support, I haven't seen
> anyone objecting strong enough to try and override that maintainer
> action for example.

I think this is a misrepresentation. There is no readily mergable patch
for aliasing support. The most complete one is the tree from Simon
Richter and he considers that work incomplete still. At this time, it's
no a question of merging or not really.

>From my point of view, the question really is whether we want to
permanently extend dpkg's features in a way that we only really see
useful for one transition while still having to maintain it for
backwards compatibility.

In any case, your reply demonstrates why it is so difficult to get to a
good consensus item on this.

> Ah, this was really helpful, because it allowed me to understand that at
> least you and I haven't even been thinking about the problem space the
> same.

Cool. I've also learned something about how debootstrap works
differently than I originally thought from Ansgar. Both of these are
indications that this discussion is constructive.

> Let's explore whether debootstrapping tools need to have release
> knowledge at all.

Yes, that's an important question. I implied it in #3a vs #3b/#3c.

> Support for creating a merged /usr installation was first added in
> debootstrap 1.0.83 in September of 2016.
> It was enabled by default in  1.0.85 in October of 2016.
> So, everything since stretch has supported merged /usr.

In essence, you are proposing to permanently make the setup of symbolic
links part of the bootstrap protocol. Given that installing usrmerge has
even worked on older releases, we could simply treat them as
retroactively merged.

> 1) Do we actually still need to support boostrapping things older than
> stretch at all using modern bootstrap tools?

A customer of mine (the one that makes me work on /usr-merge :) still
works with jessie and developing updates for jessie still tends to
happen on unstable systems. So I argue that around 10 DDs probably still
have a need for doing this.

> If you're really installing something that old, can't you use a
> container image to use an older bootstrapping tool?

So after having answered the question of whether some people still need
this, I think installing an older debootstrap would be far more annoying
than the workaround you propose:

> 2) Even if we do, I think it's okay to say that you need to specify
> --no-merged-usr when installing something older than stretch, just as
> you need to specify that if you want a buster, stretch, or bullseye
> version that is not merged /usr.

While that workaround seems simple enough, plugging an option into
debootstrap can be difficult at times. I have also been working on
setting up autopkgtests and learned that debci wraps the debootstrap
call in quite some layers. Stuffing --no-merged-usr through those layers
is a non-trivial effort. I know this, because I recently sent multiple
MRs for debci to get --keyring through those layers.

Quite obviously, I am in a really special situation of actually working
with jessie and working on autopkgtests for jessie. No sane person would
do that and I cannot expect that Debian goes through extra hoops just
for me. That said, it's not like your strategy is without cost. It just
happens elsewhere.

> So my proposal is to modify the bootstrap protocol, and unless an
> administrator specifically requests a non merged /usr system, then merge
> /usr.

This sounds as if we'd just have to patch mmdebstrap and cdebootstrap
(and remove multistrap) while keeping debootstrap the way it is. I think
that we will have to touch debootstrap in any case. If you specify
--variant=buildd, you get an unmerged chroot even when you do it on
trixie or unstable. This already surprised some users and probably needs
changing. So debootstrap is the thing that definitely needs an update
(even in stable) while the others may not need an update if we end up
picking #3c.

> My initial analysis is that you're making this more complicated than it
> needs to be.

I disagree with my personal and Freexian collaborator hats, but I see
how you get here and that my use cases are all but representative. I
find it plausible that we'd get a majority for the opinion that having
to specify --no-merged-usr for old releases would be an acceptable
workaround.

> My 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Andreas Metzler]

On 2023-06-28 Helmut Grohne  wrote:
> The category of generic changes includes
> imposing an ordering on initial unpacks (e.g. base-files first).

Hello,

I have not dug deeply into this but in the back of my mind a voice is
vaguely remebering that we already had multiple times wished we had this
in our toolset for handling other upgrade issues.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Wookey]

On 2023-06-29 12:53 +0200, Johannes Schauer Marin Rodrigues wrote:
> Choosing #3c gets us more than just a simple and clean design. Encoding the
> information of how a chroot should look like in the packages instead of the
> bootstrapping tool allows creating chroots for Debian unstable all the way 
> back
> to 2006-08-10 using debbisect or debootsnap. Yes, creating old chroots via
> intermediary chroots is possible but it wastes processor cycles, adds
> complexity and requires hardcoding timestamps in the tools doing the job
> automatically. Letting the Essential:yes packages and their dependencies 
> decide
> how a chroot is supposed to look like is also friendly to our derivatives as
> they then no longer need to maintain their custom setup in a tool like
> debootstrap. Choosing a component-based view on the bootstrapping problem does
> not only give is a clean design but also desirable properties for creating
> either old chroots from snapshot.d.o or chroots for derivatives without
> requiring hardcoding things over and over again in several tools.
> 
> I fear that we are sacrificing the benefits we get from using the component
> based approach to software engineering. We are tempted by a quick-to-implement
> solution to get things done now without having to think much more about it and
> silently accept the long term costs for all tools in the bootstrapping space.

Thanks for writing this mail Josch. You saved me writing a long mail
making many of the same points.

It has long been a fundamental feature of Debian that it was just the
sum of its parts (packages). Breaking that feature was a mistake and I
was very disappointed that we chose to do that for a while as it
didn't seem in line with our general favour of rigour over
expediency. 

I too strongly favour 3c as a way to get back that fundamental
property of non-hackiness, where the OS is just the unpacked packages,
with ordering controlled by the dependency metadata.

And kudos to Helmut for turning that massive thread into a cogent list
of issues and options. It is much appreciated.

I agree with the two consensus proposals Helmut put in the email at
the top of this thread (that DEP17 is a good representation of the
issues and that putting the files in the canonical locations is the
long-term endpoint of this transition).

Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi Luca,

On Thu, Jun 29, 2023 at 12:49:16PM +0100, Luca Boccassi wrote:
> Essentially, this boils down to risks vs benefits. The risk of going
> 3c is that every single Debian installation in existence breaks in
> some interesting ways, as fixing the bootstrapping corner case is
> delegated to the package upgrade workflow. The sole benefit is that
> one of the two bootstrapping tools in widespread use keeps its
> internal code a bit 'cleaner' from the point of view of some
> technically unnecessary and self-imposed design constraints (yes
> there's 2 more tools as pointed out yesterday, but they appear to be
> at least under-maintained judging from tracker.d.o).

I'm not sure you understand what 3c is about. I think it is safe to say
that you are in favour of moving all of the files to their canonical
location (i.e. from / to /usr). This is half of the picture for 3c. The
other half is shipping the symbolic links in base-files rather than
having them created in some way not tracked by dpkg. If you plug these
two together, you have made /usr-merged bootstrapping work without
having changed the protocol.

So what is the risk involved here? I think there are three main risk
categories at play:

1. The risk of effects from moving files from / to /usr. This is a risk
   that you clearly see as worth taking regardless of the bootstrap
   case.

2. The risk of effects from shipping the symbolic links in base-files. I
   see that you'd rather not do this, but not shipping them in any
   package poses a deletion risk of its own, so shipping them
   effectively is a risk mitigation and is what allows us to drop
   protective diversions eventually. It stills risks breaking
   debootstrap's behind-the-back approach of merging, so we'll likely
   have to do a stable upload of debootstrap.

3. The risk of unstable becoming temporarily non-bootstrappable. This is
   where I see the main fragility of the approach. As is evident from
   your next paragraph, you don't really care about this either.

Given this, it seems rather evident that you have a different risk in
mind that I do not see at this time. Can you elaborate?

Then, software maintainers tend to say "no" when a feature poses a
non-trivial cost to permanent maintenance. We see this all the time and
you shrug it off, because it's not your package. However, when people do
the reverse (e.g. diverting systemd's units poses a non-trivial
maintenance cost to systemd), you take it for granted that you can
unilaterally say "no". Why is it ok for you to say "no", but not for
other maintainers to say "no"?

> I don't see how it's worth the risk. This is essentially a problem in
> the bootstrapping tools, so solving it in the bootstrapping tools is
> not only the safest approach - worst case scenario, creating a new sid
> chroot might not work for a couple days, not a big deal given it
> happens all the time for various reasons as we've seen this week -
> it's also the right approach.

You seem to have missed Johannes reasoning entirely. He sees Debian as a
component-based system. He argues that this is not a problem in the
bootstrapping tools, but a problem in the components being bootstrapped.
In effect, the usrmerge binary package currently implements it in a
component-oriented way. Since it is a problem with that component,
solving it there makes most sense, no? That alone makes it obvious, that
this is not a problem limited to bootstrapping. We have now duplicated
this mechanism to usrmerge and debootstrap and you are proposing to
duplicate it again. I argue that a maintainable implementation should
centralize this aspect into (preferably only) one component.

Helmut

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Timo Röhling]

Hi Luca,

* Luca Boccassi  [2023-06-29 12:49]:

The sole benefit is that one of the two bootstrapping tools in
widespread use keeps its internal code a bit 'cleaner' from the
point of view of some technically unnecessary and self-imposed
design constraints (yes there's 2 more tools as pointed out
yesterday, but they appear to be at least under-maintained judging
from tracker.d.o).

I believe you are narrowing the focus too much on an implementation
instead of taking the concept into consideration.

If I understand Josch correctly, he is aiming for a fully
declarative way to describe how to setup a Debian system, and I
believe that the idea has a lot of merit. It is also consistent with
a general trend in Debian: just look at d/rules and see how little
actual code a modern debhelper rule script needs; with dh-sequence-*
and other declarative files in debian/, it is often little more than

%:
dh $@

It makes packaging simpler, avoids code duplication and also makes
it much easier for tools like lintian to reason about package
behavior. For instance, good luck coming up with a reliable check to
see if a hand-written d/rules without dh_auto_test properly obeys
the "nocheck" profile.


I don't see how it's worth the risk. This is essentially a problem in
the bootstrapping tools, so solving it in the bootstrapping tools is
not only the safest approach - worst case scenario, creating a new sid
chroot might not work for a couple days, not a big deal given it
happens all the time for various reasons as we've seen this week -
it's also the right approach.

Josch already made a good analogy that this is "essentially a
bootstrapping tools problem" in the same way that dependency
resolution is "esssentially a dpkg problem", yet nobody patches dpkg
if someone omits "Depends: libbar-dev" in a libfoo-dev which happens
to include stuff from bar in its headers.

I do concede that we need not limit ourselves to just two options,
either have a bootstrapping that works with package dependencies
as-is, or have a free-for-all with random and arbitrarily complex
code in bootstrapping tools. We could also declare a bootstrapping
protocol that always unpacks a specific package (let's call it
"first-package"), which may contain additional configuration options
to guide the bootstrapping process. We could also teach dpkg to
always upgrade that particular package first, as a way to get
something like a dist-upgrade script.

My point is, Josch raises some interesting points, which we should
at least take into consideration and not dismiss them out of hand
because bootstrapping is not important for us.


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Thu, 29 Jun 2023 at 11:53, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> mmdebstrap author here. This is the other bootstrapping tool which is 
> currently
> sitting at ~17% of the popcon value of debootstrap.
>
> Quoting Helmut Grohne (2023-06-28 21:37:44)
> > Once that is settled, the next big question is how to handle bootstrapping.
> > We had a number of people arguing in favour of changing the bootstrap
> > protocol. Such changes can be classified into generic changes and
> > release-dependent changes. A release-dependent change enhances bootstrapping
> > tools with knowledge of available releases and adapts behaviour in
> > release-specific ways that are encoded into the bootstrapping tool. As it
> > stands, the only bootstrapping tool that has been enhanced in this way is
> > debootstrap and that support is limited to Debian and two derivatives. The
> > category of generic changes includes imposing an ordering on initial unpacks
> > (e.g. base-files first). Others are in favour of not changing the bootstrap
> > protocol. In that view, some data.tar will have to ship the symbolic links
> > and all other essential packages need to have their files canonicalized.
> >
> > Among these options, the first has a working prototype (debootstrap),
> > but it is unclear how that extends to use of snapshot.d.o and how to
> > make it work with debootsnap and debbisect as those tend to use a suite
> > name rather than a codename. The last option has a prototype and relies
> > on uploading a number of essential packages in a short window of time.
> > (What could possibly go wrong?)
> >
> > It is not clear to me how we can get to a consensus on these, so the
> > best I can do here is summarize options.
> >
> > Option #3a:
> >
> > The bootstrap protocol shall be changed to contain a task for
> > merging /usr as is done in debootstrap in a release-dependent way.
> >
> > This is an instance of M16 from DEP17.
> >
> > Option #3b:
> >
> > The bootstrap protocol shall be changed in unspecified ways to
> > support the /usr-merged systems in a way that does not depend on
> > matching the codename or suitename.
> >
> > This is an instance of M16 from DEP17.
> >
> > Option #3c:
> >
> > The bootstrap protocol shall remain unchanged. Therefore, all
> > essential packages need to move their files out of aliased locations
> > and the aliasing symlinks are to be installed from a data.tar of a
> > binary package such as base-files.

<...>

> I think this now comes down to what we as a project care about. Which 
> use-cases
> are important to us? Which properties do we want to preserve?
>
> What do you think?

Essentially, this boils down to risks vs benefits. The risk of going
3c is that every single Debian installation in existence breaks in
some interesting ways, as fixing the bootstrapping corner case is
delegated to the package upgrade workflow. The sole benefit is that
one of the two bootstrapping tools in widespread use keeps its
internal code a bit 'cleaner' from the point of view of some
technically unnecessary and self-imposed design constraints (yes
there's 2 more tools as pointed out yesterday, but they appear to be
at least under-maintained judging from tracker.d.o).

I don't see how it's worth the risk. This is essentially a problem in
the bootstrapping tools, so solving it in the bootstrapping tools is
not only the safest approach - worst case scenario, creating a new sid
chroot might not work for a couple days, not a big deal given it
happens all the time for various reasons as we've seen this week -
it's also the right approach.

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Johannes Schauer Marin Rodrigues]

Hi,

mmdebstrap author here. This is the other bootstrapping tool which is currently
sitting at ~17% of the popcon value of debootstrap.

Quoting Helmut Grohne (2023-06-28 21:37:44)
> Once that is settled, the next big question is how to handle bootstrapping.
> We had a number of people arguing in favour of changing the bootstrap
> protocol. Such changes can be classified into generic changes and
> release-dependent changes. A release-dependent change enhances bootstrapping
> tools with knowledge of available releases and adapts behaviour in
> release-specific ways that are encoded into the bootstrapping tool. As it
> stands, the only bootstrapping tool that has been enhanced in this way is
> debootstrap and that support is limited to Debian and two derivatives. The
> category of generic changes includes imposing an ordering on initial unpacks
> (e.g. base-files first). Others are in favour of not changing the bootstrap
> protocol. In that view, some data.tar will have to ship the symbolic links
> and all other essential packages need to have their files canonicalized.
> 
> Among these options, the first has a working prototype (debootstrap),
> but it is unclear how that extends to use of snapshot.d.o and how to
> make it work with debootsnap and debbisect as those tend to use a suite
> name rather than a codename. The last option has a prototype and relies
> on uploading a number of essential packages in a short window of time.
> (What could possibly go wrong?)
> 
> It is not clear to me how we can get to a consensus on these, so the
> best I can do here is summarize options.
> 
> Option #3a:
> 
> The bootstrap protocol shall be changed to contain a task for
> merging /usr as is done in debootstrap in a release-dependent way.
> 
> This is an instance of M16 from DEP17.
> 
> Option #3b:
> 
> The bootstrap protocol shall be changed in unspecified ways to
> support the /usr-merged systems in a way that does not depend on
> matching the codename or suitename.
> 
> This is an instance of M16 from DEP17.
> 
> Option #3c:
> 
> The bootstrap protocol shall remain unchanged. Therefore, all
> essential packages need to move their files out of aliased locations
> and the aliasing symlinks are to be installed from a data.tar of a
> binary package such as base-files.
> 
> This is M2+M11 from DEP17.
> 
> While a few people including Marco d'Itri and Sam Hartman have argued in
> favour of exploring the space of #3b, no proposals have emerged in the
> interim. The proposal in #3a has three significant limitations:
>  * It creates compatibility issues when combining old a new suites
>unless changes to bootstrapping tools are backported to older
>releases.
>  * It becomes a whack-a-mole, since we need to add codenames of every
>derivative to every bootstrapping implementation.
>  * It breaks bootstrapping from snapshot.d.o and therefore breaks tools
>such as debbisect and debootsnap.
> 
> While the first of these limitations is shared with #3b, the others are
> not and that would make #3b more attractive to me if there was a
> concrete proposal to evaluate. The one about unpacking base-files first
> seemed the most concrete to me, but it has the downside of imposing a
> permanent cost on bootstrapping tools even though we only need that
> behaviour temporarily, which seems like too bad of a trade-off to start
> with in my opinion. Did I miss a relevant proposal for modifying the
> bootstrap protocol?
> 
> On the flip side, there is a demo for #3c showing that we can move most
> of the things except for a hand full of packages and then flip the
> switch (for bootstrapping) in unstable by uploading those packages
> simultaneously. The biggest downside of this probably is the inherent
> fragility of this approach. Even if this is extensively tested before
> uploading chances are good that we still break something unforeseen in
> the process.
> 
> Can I get more feedback from those who rather not have #3c implemented as to
> how you see things moving forward?

there was some feedback of people who'd rather not have #3c implemented. I
agree with them that changing bootstrapping tools is a quick and easy fix that
only requires a few lines of easily testable code, can be deployed quickly and
is unlikely to break unrelated things in a serious way.

In this email I'd like to explain why I think that #3a and #3b are not good
ways forward for the distribution. In summary, what I fear is, that choosing
either #3a or #3b would mean choosing a simple fix just because it's simple and
ignoring the fact that those solutions mean a permanent cost for the
distribution (even beyond debootstrap) for many years to come. What I'd like
Debian to choose would be the technically and architecturally proper solution
even if it means more work but results in multiple long-term benefits. To
expand further on why I think this way, let me go back in time a bit.

Around 7 years ago I started being 

Re: amount of bootstrapping tools in Debian (Re: Second take at DEP17 - consensus call on /usr-merge matters)

[Holger Levsen]

On Wed, Jun 28, 2023 at 08:59:18PM +, Holger Levsen wrote:
> however it's author also said on #-devel just now:
[...]
<  josch> | h01ger: i'm not multistraps author though

(josch AFAICS is the last maintainer of it, maintaining it from 2016
to 2018.)

my point is: it's more than two tools and let's do this properly.


--
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

The system isn't broken. It was built this way.


signature.asc
Description: PGP signature

Re: amount of bootstrapping tools in Debian (Re: Second take at DEP17 - consensus call on /usr-merge matters)

[Holger Levsen]

On Wed, Jun 28, 2023 at 08:28:28PM +, Holger Levsen wrote:
> On Wed, Jun 28, 2023 at 09:15:30PM +0100, Luca Boccassi wrote:
> > > to how you see things moving forward?
> > Changing the bootstrap tools seems much safer. It is just two tools,
> three: debootstrap, mmdebstrap and cdebootstrap.

four: https://tracker.debian.org/multistrap

however it's author also said on #-devel just now:

 | h01ger: i tried to fix multistrap but failed 
https://gitlab.mister-muffin.de/josch/multistrap/commit/cd5dfbbbf2435bae8fc34ac32ee7d716c24bada8
< josch> i very much dislike removing stuff that still works -- if you feel 
differently, feel free to go ahead. i'll not stand in anybodies way :)

(quoted here with permission.)


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Religion has been more harmful to humanity than cigarettes.


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

I believe I'm up on this discussion to at least comment on the consensus
calls you discuss in the mail.
Except where noted below, I support your reading of consensus.


Helmut> Consensus proposal #1:

Helmut> This updated DEP17 is a complete representation of the
Helmut> known and relevant problems and known mitigations under
Helmut> discussion at the time of this writing.

I have read the mail, not the full updated DEP, so I cannot yet ack
this.

Helmut> When we get into mitigations, consensus is hard to come
Helmut> by. My impression is that we have roughly two camps. One is
Helmut> in favour of major changes to dpkg and the other is not.

I think you might get a more clear consensus if you phrase that in terms
of whether people are willing to wait for major changes in dpkg.
If the dpkg maintainer were to merge aliasing support, I haven't seen
anyone objecting strong enough to try and override that maintainer
action for example.


Helmut> There also is a minority arguing in favour of doing
Helmut> both. I've kinda ruled out that option already as we get the
Helmut> downsides of both without any further benefit in return.

I agree there is not sufficient support to try and force this.
I don't think there is a strong enough consensus to push back against
anything reasonable the dpkg maintainer did in this space.

Helmut> My impression is that the (vocal) majority falls into the
Helmut> latter category. The major alternative here is getting into
Helmut> a state where all paths shipped in binary packages are not
Helmut> aliased (dubbed M2 in DEP17).  Therefore, I propose a second
Helmut> consensus call.

Helmut> Consensus proposal #2:

Helmut> The primary method for finalizing the /usr-merge
Helmut> transition is moving files to their canonical locations
Helmut> (i.e. from / to /usr) according to the dpkg fsys database
Helmut> (i.e. in data.tar of binary packages).  dpkg is not
Helmut> augmented with a general mechanism supporting aliasing nor
Helmut> do we encode specific aliases into dpkg.

Helmut> I recognize that this is not unanimous, but I think we still
Helmut> have sufficient consensus on this. I suspect that maybe
Helmut> Simon Richter and a few more would disagree here. If
Helmut> consensus fails, we may have to put this to a vote.

I agree, there is a rough consensus on this.

Helmut> Once that is settled, the next big question is how to handle
Helmut> bootstrapping. We had a number of people arguing in favour
Helmut> of changing the bootstrap protocol. Such changes can be
Helmut> classified into generic changes and release-dependent
Helmut> changes. A release-dependent change enhances bootstrapping
Helmut> tools with knowledge of available releases and adapts
Helmut> behaviour in release-specific ways that are encoded into the
Helmut> bootstrapping tool. As it stands, the only bootstrapping
Helmut> tool that has been enhanced in this way is debootstrap and
Helmut> that support is limited to Debian and two derivatives. The
Helmut> category of generic changes includes imposing an ordering on
Helmut> initial unpacks (e.g. base-files first). Others are in
Helmut> favour of not changing the bootstrap protocol. In that view,
Helmut> some data.tar will have to ship the symbolic links and all
Helmut> other essential packages need to have their files
Helmut> canonicalized.

Ah, this was really helpful, because it allowed me to understand that at
least you and I haven't even been thinking about the problem space the
same.

Let's explore whether debootstrapping tools need to have release
knowledge at all.
Support for creating a merged /usr installation was first added in
debootstrap 1.0.83 in September of 2016.
It was enabled by default in  1.0.85 in October of 2016.
So, everything since stretch has supported merged /usr.

1) Do we actually still need to support boostrapping things older than
stretch at all using modern bootstrap tools?
If you're really installing something that old, can't you use a
container image to use an older bootstrapping tool?

2) Even if we do, I think it's okay to say that you need to specify
--no-merged-usr when installing something older than stretch, just as
you need to specify that if you want a buster, stretch, or bullseye
version that is not merged /usr.

So my proposal is to modify the bootstrap protocol, and unless an
administrator specifically requests a non merged /usr system, then merge
/usr.

My initial analysis is that you're making this more complicated than it
needs to be.

My assumption here is also that the change needed to the bootstrap
protocol is the same as the change that debootstrap has been doing for
years.
If there are additional changes that would be harmful when bootstrapping
stretch, buster, bullseye, or bookworm,
I agree it gets more 

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Ansgar]

Hi,

On Wed, 2023-06-28 at 21:37 +0200, Helmut Grohne wrote:
> Among these options, the first has a working prototype (debootstrap),
> but it is unclear how that extends to use of snapshot.d.o and how to
> make it work with debootsnap and debbisect as those tend to use a
> suite name rather than a codename.

debootstrap allows using suite names as well. As their meaning changed
over time, the changes for conditional merged-usr also made debootstrap
extract the Codename from the Release file in case a user specified a
suite name.  These days there are a few other changes that might depend
on the release (identified by the codename).

As pointed out on IRC, this might not work for the combination of
"unstable" and using snapshot.d.o, but in principle that could be
handled by looking at the Release's Date field as well (but, meh, not
sure if we need that).

>  * It becomes a whack-a-mole, since we need to add codenames of every
>derivative to every bootstrapping implementation.

Well, shouldn't it already have become whack-a-mole then?  debootstrap
has the code for many years and is widely used after all... It doesn't
seem to have been a problem in practice as there haven't been
complaints since this was implemented AFAIK.  (Possibly users just
specify --no-merged-usr manually if needed; no idea.)

> Introduction
> 
[...]
> Regardless of what strategy we end up choosing here, we will likely
> keep some of the temporary changes even in the `forky` release to
> accommodate external repositories and derivatives.

There should be no additional work for derivatives.  merged-/usr is not
really something a Debian-based derivative can opt-out of and if some
distribution did so (say because of warnings in dpkg or for other
reasons), it's their own problem...

Ansgar

amount of bootstrapping tools in Debian (Re: Second take at DEP17 - consensus call on /usr-merge matters)

[Holger Levsen]

On Wed, Jun 28, 2023 at 09:15:30PM +0100, Luca Boccassi wrote:
> > to how you see things moving forward?
> Changing the bootstrap tools seems much safer. It is just two tools,

three: debootstrap, mmdebstrap and cdebootstrap.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

“I'll tell you what freedom is to me No fear.” (Nina Simone)


signature.asc
Description: PGP signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Wed, 28 Jun 2023 at 20:38, Helmut Grohne  wrote:
>
> Hi,
>
> thanks for all the valuable feedback on the huge DEP17 thread. As
> promised, I looked into condensing that discussion into something
> shorter. That shorter thing still has more than 3000 words and is
> available as source at

After seeing the 3k words, I now regret having asked you for an update
a couple days ago :-D

> Consensus proposal #1:
>
> This updated DEP17 is a complete representation of the known and
> relevant problems and known mitigations under discussion at the time
> of this writing.

+1

> Consensus proposal #2:
>
> The primary method for finalizing the /usr-merge transition is
> moving files to their canonical locations (i.e. from / to /usr)
> according to the dpkg fsys database (i.e. in data.tar of binary
> packages).  dpkg is not augmented with a general mechanism
> supporting aliasing nor do we encode specific aliases into dpkg.

+1

> Option #3a:
>
> The bootstrap protocol shall be changed to contain a task for
> merging /usr as is done in debootstrap in a release-dependent way.
>
> This is an instance of M16 from DEP17.
>
> Option #3b:
>
> The bootstrap protocol shall be changed in unspecified ways to
> support the /usr-merged systems in a way that does not depend on
> matching the codename or suitename.
>
> This is an instance of M16 from DEP17.
>
> Option #3c:
>
> The bootstrap protocol shall remain unchanged. Therefore, all
> essential packages need to move their files out of aliased locations
> and the aliasing symlinks are to be installed from a data.tar of a
> binary package such as base-files.
>
> This is M2+M11 from DEP17.
>
> While a few people including Marco d'Itri and Sam Hartman have argued in
> favour of exploring the space of #3b, no proposals have emerged in the
> interim. The proposal in #3a has three significant limitations:
>  * It creates compatibility issues when combining old a new suites
>unless changes to bootstrapping tools are backported to older
>releases.

I have mentioned this a few times already, but: we already did this
last year, I backported a bunch of changes in debootstrap to bullseye
and even buster, to add support for usr-is-merged. I do not see this
as a "significant" limitation, merely some additional busywork that
I'm happy to take care of.

>  * It becomes a whack-a-mole, since we need to add codenames of every
>derivative to every bootstrapping implementation.

As we have learned recently, the project is fine if know-broken
changes percolate down to derivatives, it is fine to just document it,
and it is fine to let said derivatives do their own downstream changes
to deal with it as they see fit. I don't see why this case would be
any different. Sauce for the goose is sauce for the gander.

>  * It breaks bootstrapping from snapshot.d.o and therefore breaks tools
>such as debbisect and debootsnap.

Sure, some combinations might be broken, but do we even guarantee that
any combination from any point of snapshot.d.o would always work?
Seeing that just today there was an unintentional (and quickly fixed)
regression in an essential package that broke installation altogether,
it seems to me the answer is no. Hence as above, sauce for the goose
etc etc.

> While the first of these limitations is shared with #3b, the others are
> not and that would make #3b more attractive to me if there was a
> concrete proposal to evaluate. The one about unpacking base-files first
> seemed the most concrete to me, but it has the downside of imposing a
> permanent cost on bootstrapping tools even though we only need that
> behaviour temporarily, which seems like too bad of a trade-off to start
> with in my opinion. Did I miss a relevant proposal for modifying the
> bootstrap protocol?
>
> On the flip side, there is a demo for #3c showing that we can move most
> of the things except for a hand full of packages and then flip the
> switch (for bootstrapping) in unstable by uploading those packages
> simultaneously. The biggest downside of this probably is the inherent
> fragility of this approach. Even if this is extensively tested before
> uploading chances are good that we still break something unforeseen in
> the process.
>
> Can I get more feedback from those who rather not have #3c implemented as
> to how you see things moving forward?

Changing the bootstrap tools seems much safer. It is just two tools,
and we need backports to [[[old]old]old]stable just for debootstrap as
that's what is used on the buildd infrastructure, and the other will
be up to the maintainer how far back to go. The risk is contained to
the bootstrapping case, which _usually_ is a relatively 'safe'
environment, in the sense that there is some expectation that it might
fail and the runner is expected to deal with it. Moving the risk to
the upgrade phase of every single Debian instance in existence seems
inherently more risky to me.

> Option 

Second take at DEP17 - consensus call on /usr-merge matters

[Helmut Grohne]

Hi,

thanks for all the valuable feedback on the huge DEP17 thread. As
promised, I looked into condensing that discussion into something
shorter. That shorter thing still has more than 3000 words and is
available as source at

https://salsa.debian.org/dep-team/deps/-/merge_requests/5

and I also put up a rendered version at

https://subdivi.de/~helmut/dep17.html

and for those preferring to read offline, I'm also appending the main
file to this mail.

This still is a draft and it fundamentally still lacks the "proposal"
part of a DEP. What it does however is collecting all the problems we've
encountered in the discussion and most of the proposed mitigations and
it has gathered a few reviews already.

Consensus proposal #1:

This updated DEP17 is a complete representation of the known and
relevant problems and known mitigations under discussion at the time
of this writing.

Do you miss a related problem important to you? Do you miss your
preferred mitigation? Please speak up, so we can record it.

Stating a goal has been quite difficult, but I think that most of the
participants agree that we want to eliminate the file move moratorium
without getting problems in return. And I also hope that my listing of 9
problems is agreeable by the project. We may need changes still and we
may encounter new problems, but this tries to capture the best we have
right now.

When we get into mitigations, consensus is hard to come by. My
impression is that we have roughly two camps. One is in favour of major
changes to dpkg and the other is not.

Arguments in favour of changing dpkg (dubbed M1 in DEP17):
 * We change one piece and doing that fixes a whole host of problems.
 * dpkg learns to understand the current situation and avoids us
   applying a lot of workarounds.
 * A solution may be more generally applicable.
 * There is no urgency with moving files to their canonical locations if
   dpkg knows to identify aliased and canonical locations.

Arguments in favour of not changing dpkg:
 * We'd implement changes for one transition, but they pose a permanent
   maintenance cost that does not become easily removable later.
 * Even if dpkg were fixed in trixie, trixie's libc6 would still be
   unpacked using bookworm's dpkg. At least for this package, we cannot
   rely on dpkg changes in trixie. Therefore we need workarounds or
   spread the transition to forky. For other packages, even a
   Pre-Depends on dpkg is not a guarantee that a changed dpkg will
   perform the unpack.
 * Changes to dpkg will not fix bootstrapping.

There also is a minority arguing in favour of doing both. I've kinda
ruled out that option already as we get the downsides of both without
any further benefit in return.

My impression is that the (vocal) majority falls into the latter
category. The major alternative here is getting into a state where all
paths shipped in binary packages are not aliased (dubbed M2 in DEP17).
Therefore, I propose a second consensus call.

Consensus proposal #2:

The primary method for finalizing the /usr-merge transition is
moving files to their canonical locations (i.e. from / to /usr)
according to the dpkg fsys database (i.e. in data.tar of binary
packages).  dpkg is not augmented with a general mechanism
supporting aliasing nor do we encode specific aliases into dpkg.

I recognize that this is not unanimous, but I think we still have
sufficient consensus on this. I suspect that maybe Simon Richter and a
few more would disagree here. If consensus fails, we may have to put
this to a vote.

Once that is settled, the next big question is how to handle
bootstrapping. We had a number of people arguing in favour of changing
the bootstrap protocol. Such changes can be classified into generic
changes and release-dependent changes. A release-dependent change
enhances bootstrapping tools with knowledge of available releases and
adapts behaviour in release-specific ways that are encoded into the
bootstrapping tool. As it stands, the only bootstrapping tool that has
been enhanced in this way is debootstrap and that support is limited to
Debian and two derivatives. The category of generic changes includes
imposing an ordering on initial unpacks (e.g. base-files first). Others
are in favour of not changing the bootstrap protocol. In that view, some
data.tar will have to ship the symbolic links and all other essential
packages need to have their files canonicalized.

Among these options, the first has a working prototype (debootstrap),
but it is unclear how that extends to use of snapshot.d.o and how to
make it work with debootsnap and debbisect as those tend to use a suite
name rather than a codename. The last option has a prototype and relies
on uploading a number of essential packages in a short window of time.
(What could possibly go wrong?)

It is not clear to me how we can get to a consensus on these, so the
best I can do here is summarize options.

Option #3a:

The bootstrap protocol