Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Russ Allbery]

Zack Weinberg  writes:

> That does not clearly say that a *pair* of packages, where one installs
> files in /path and the other one installs files in /usr/path, is not 
> allowed.  I'm guessing it was the *intent*, but the example, at least,
> makes it sound like it's only talking about within a single package.

Thanks, good catch.  We've been dealing with this elsewhere as the other
replies pointed out, but we should update the wording in Policy to make
this explicit.

I'll open a separate bug for that.

-- 
Russ Allbery (r...@debian.org)  

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On 2022-09-28 2:16 PM, Russ Allbery wrote:

"Zack Weinberg"  writes:


1. Is there already a rule (or multiple rules) somewhere that forbids
the existence of pairs of packages where one ships /X/Y and the
other ships /usr/X/Y, where X is a directory on non-merged-/usr
systems and a symlink on merged-/usr systems, and Y is any name?


Policy 10.1:

 To support merged-/usr systems, packages must not install files in
 both /path and /usr/path. For example, a package must not install both
 /bin/example and /usr/bin/example.


That does not clearly say that a *pair* of packages, where one installs 
files in /path and the other one installs files in /usr/path, is not 
allowed.  I'm guessing it was the *intent*, but the example, at least, 
makes it sound like it's only talking about within a single package. 
Suggest maybe the editorial addition of


For another example, if one package installs /bin/example, then
no package which is coinstallable with that package may install
/usr/bin/example.

(the "coinstallable with" clause means to clarify that the existing pair 
discussed upthread is fine because there's a package-level Conflicts)


zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Russ Allbery]

"Zack Weinberg"  writes:

> 1. Is there already a rule (or multiple rules) somewhere that forbids
>the existence of pairs of packages where one ships /X/Y and the
>other ships /usr/X/Y, where X is a directory on non-merged-/usr
>systems and a symlink on merged-/usr systems, and Y is any name?

Policy 10.1:

To support merged-/usr systems, packages must not install files in
both /path and /usr/path. For example, a package must not install both
/bin/example and /usr/bin/example.

If a file is moved between /path and /usr/path in revisions of a
Debian package, and a compatibility symlink at the old path is needed,
the symlink must be managed in a way that will not break when /path
and /usr/path are the same underlying directory due to symlinks or
other mechanisms.

We've had that rule in Policy since May of 2017.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Svante Signell]

Hi,

I would really appreciate if you quote your reply properly:
It was not Andreas Metzler who sent the below:

> On Tue, 2022-09-27 at 18:25 +0200, Andreas Metzler wrote:
> > On 2022-09-27 Zack Weinberg  wrote:
> > [...]

On Wed, 2022-09-28 at 11:55 +0100, Luca Boccassi wrote:

> > 
> > > This number is of usrmerged systems is so large that we cannot mark
> > > them as unsupported ("Please reinstall"). Whether this percentage
> > > is 25% or 90% does not matter.
> > 
> > You can easily revert any system having usrmerge installed with dpkg-
> > fsys-usrunmess. This should be known by all Debian users, by some
> > suitable channel.
> > 
> > And for example the latest init-system-helpers release should add
> > this to the package description (if not reverted). This applies to
> > other present and future packages having usrmerge as a dependency
> > too.
> 
> Please note that this will result in an unsupported system, as per CTTE
> decision. It is important to note this, for the record. So no, that
> will not be added anywhere, package description or otherwise, and in
> fact the last time it was "made known by all Debian users" it caused
> quite a lot of damage, and was forcefully reverted.

Please learn how to properly quote/reply to peoples postings :(

Thanks!

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Wed, Sep 28, 2022, at 5:08 AM, Svante Signell wrote:
>
> You can easily revert any system having usrmerge installed with dpkg-
> fsys-usrunmess. This should be known by all Debian users, by some
> suitable channel.

Having used it myself a couple of times, I would question "easily".  If all 
goes well, yes, you run it and you reboot and you're done, but if all *doesn't* 
go well you're left having to manually repair a system with important files not 
existing in *either* their unmerged or their merged location, which may require 
booting from recovery media.

I'd say that if Debian were going to widely advertise the availability of 
dpkg-fsys-usrunmess, first it ought to be revamped to ensure that it's fully 
restartable, idempotent, and never, not even transiently, places the system in 
a state where it cannot boot at least as far as single user mode (in systemd 
terms, rescue.target, *not* just emergency.target).

Of course the exact same criticism applies to convert-usrmerge -- skimming its 
code just now, it appears to be idempotent and restartable in principle, but if 
"the system crashes at a really bad time" (to quote its own comments) I think 
it _could_ leave the system in a state where it cannot boot as far as 
rescue.target.  In fact, see #1020920.

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Wed, Sep 28, 2022, at 1:40 PM, Helmut Grohne wrote:
> Hi Zack,
>
> On Wed, Sep 28, 2022 at 12:29:19PM -0400, Zack Weinberg wrote:
>> I thought about this a bunch yesterday evening and I believe I see a
>> concrete scenario that can cause problems but is not covered by the
>> moratorium: Suppose there exist two packages, one of which ships
>> /bin/foo, and the other ships /usr/bin/foo.  On a non-merged system,
>> these packages can coexist.  On a merged system, they have a file
>> conflict that dpkg will not detect.
>
> $ ssh delfin.debian.org sqlite3 
> /srv/dedup.debian.org/database/dedup.sqlite3 '"'"SELECT * FROM content 
> AS ca JOIN content AS cb ON ca.filename = './usr' || 
> substr(cb.filename, 2) WHERE cb.filename NOT LIKE './usr/%';"'"'
> 166447797|96615|./usr/bin/systemctl|201531|221889683|4004|./bin/systemctl|1173136
> 210029518|32365|./usr/sbin/update-service|2917|223295080|31077|./sbin/update-service|4573
> $
>
> So we have systemctl vs systemd and daemontools-run vs runit, both of
> which declare Conflicts.

*nod* I don't think we need to worry about this when there's a declared 
package-level conflict.

> Depends on whether you consider that one-liner above tooling I guess.

Good enough for now, probably -- it would be nice to have some automation 
searching for such overlaps in packages that *don't* declare Conflicts, and 
filing bugs, but I won't call that a blocker.

> Do you see any other issues?

Not at present, but I'm not the person you should be asking!  The only person 
who could possibly say "yeah, the rules in place are sufficient to prevent 
problems post-bookworm until the bugs are actually fixed", with the level of 
confidence we need before proceeding with this transition, is ... the dpkg 
maintainer.

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Helmut Grohne]

Hi Zack,

On Wed, Sep 28, 2022 at 12:29:19PM -0400, Zack Weinberg wrote:
> I thought about this a bunch yesterday evening and I believe I see a
> concrete scenario that can cause problems but is not covered by the
> moratorium: Suppose there exist two packages, one of which ships
> /bin/foo, and the other ships /usr/bin/foo.  On a non-merged system,
> these packages can coexist.  On a merged system, they have a file
> conflict that dpkg will not detect.

$ ssh delfin.debian.org sqlite3 /srv/dedup.debian.org/database/dedup.sqlite3 
'"'"SELECT * FROM content AS ca JOIN content AS cb ON ca.filename = './usr' || 
substr(cb.filename, 2) WHERE cb.filename NOT LIKE './usr/%';"'"'
166447797|96615|./usr/bin/systemctl|201531|221889683|4004|./bin/systemctl|1173136
210029518|32365|./usr/sbin/update-service|2917|223295080|31077|./sbin/update-service|4573
$

So we have systemctl vs systemd and daemontools-run vs runit, both of
which declare Conflicts.

> So questions for you and everyone else reading this:
> 
> 1. Is there already a rule (or multiple rules) somewhere that forbids
>the existence of pairs of packages where one ships /X/Y and the
>other ships /usr/X/Y, where X is a directory on non-merged-/usr
>systems and a symlink on merged-/usr systems, and Y is any name?

I think Marco et al have been filing bugs about these kind of issues and
NMUing the remainders a very long time ago way before debootstrap
defaulted to merged /usr.

> 2. Does Debian already have tooling that can *find* pairs of such
>packages?  (This is a fully independent question from 1.  If
>there's a rule but no automation to enforce it then we don't *know*
>nobody's breaking the rule.  If there is no rule then, before we
>consider adding such a rule, we need to know whether any packages
>exist that break it.)

Depends on whether you consider that one-liner above tooling I guess.

Do you see any other issues?

Helmut

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Ansgar]

On Wed, 2022-09-28 at 12:29 -0400, Zack Weinberg wrote:
> 1. Is there already a rule (or multiple rules) somewhere that forbids
>    the existence of pairs of packages where one ships /X/Y and the
>    other ships /usr/X/Y, where X is a directory on non-merged-/usr
>    systems and a symlink on merged-/usr systems, and Y is any name?

*sigh*

There has been such a rule for many, many years already.

I really feel you lack investigating the issue before filing yet
another ctte bug about it.

Ansgar

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Tue, Sep 27, 2022, at 4:12 PM, Sebastian Ramacher wrote:
> On 2022-09-27 10:26:36 -0400, Zack Weinberg wrote:
>> On Tue, Sep 27, 2022, at 5:15 AM, Sebastian Ramacher wrote:
>> >> I'd like to make sure that the bug submitter has not identified
>> >> something new here.
>> >
>> > I've not seen any new issues appearing since the last round I file bugs.
>>
>> I wasn’t aware that you have been filing bugs related to the
>> transition.  What criteria are you using to find and file those bugs?
>
> I only checked for packages violating the moratorium. Thankfully a check
> for that was implemented by Luca in piuparts. If we have additional
> patterns that could cause issues for upgrades, the check would ideally
> be extended with that information.

I thought about this a bunch yesterday evening and I believe I see a
concrete scenario that can cause problems but is not covered by the
moratorium: Suppose there exist two packages, one of which ships
/bin/foo, and the other ships /usr/bin/foo.  On a non-merged system,
these packages can coexist.  On a merged system, they have a file
conflict that dpkg will not detect.

For practical reasons I doubt that two such packages actually exist --
nobody wants the concrete implementation of "foo" to be selected by
what order the user happened to list /usr/bin and /bin in their PATH,
this is what alternatives are for -- but, I don't see anything in
Policy that *forbids* it outright.  (I could have missed something.)
Also, the undetected file conflict can happen in *any* directory that
is converted to a symlink on a merged system, and  it's at least
vaguely plausible to me that there might be packages shipping
small variations on the same library as /lib/foo.so and /usr/lib/foo.so
(perhaps one has fewer dependencies, to be used during early boot).

So questions for you and everyone else reading this:

1. Is there already a rule (or multiple rules) somewhere that forbids
   the existence of pairs of packages where one ships /X/Y and the
   other ships /usr/X/Y, where X is a directory on non-merged-/usr
   systems and a symlink on merged-/usr systems, and Y is any name?

2. Does Debian already have tooling that can *find* pairs of such
   packages?  (This is a fully independent question from 1.  If
   there's a rule but no automation to enforce it then we don't *know*
   nobody's breaking the rule.  If there is no rule then, before we
   consider adding such a rule, we need to know whether any packages
   exist that break it.)

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Tue, Sep 27, 2022, at 12:25 PM, Andreas Metzler wrote:
> On 2022-09-27 Zack Weinberg  wrote:
> [...]
>> What I am asking for is a schedule change: specifically, that the
>> merged /usr transition not be allowed to proceed past the status quo
>> as of two weeks ago (i.e. *before* init-system-helpers added a
>> dependency on usrmerge|usr-is-merged) until after the dpkg bugs are
>> fixed to the satisfaction of the dpkg maintainers.
> [...]
>
> Hello Zack,
>
> Afaiui the only thing the change two weeks caused is an increased
> percentage of usrmerged Debian installations.

It is *conceivable* that a system that has been *converted* to
merged-/usr, in the way usrmerge does it, is different from a system
that was originally installed with merged /usr, in a way that matters
to whether the dpkg bugs can be triggered on that system.  I thought I
had come up with just such a scenario yesterday, in fact.  On further
consideration I was wrong -- but that doesn't mean there are no such
scenarios.

However:

> Afaict the problem is unchanged: There is a very large number of
> usrmerged systems (every system installed with bullseye installer or
> newer unless some very specific steps were taken to avoid this) which
> are prone to bugs due to dpkg not having been changed *first*. This
> number is of usrmerged systems is so large that we cannot mark them as
> unsupported ("Please reinstall"). Whether this percentage is 25% or 90%
> does not matter.

I basically agree with this assertion.  For instance, I think it's not
realistic to roll back every such system to an un-merged state, and
then restart the entire transition, this time following the procedure
that was used when /usr/man was moved to /usr/share/man, which is, it
appears to me, what Guillem would ideally like to have happen.

(I will point out, though, that if we had done it that way starting in
2015 or even 2017, *we would be done by now*, since that process takes
exactly three releases to complete.)

If I had had the authority to make the relevant decision at the time,
I probably would have insisted on getting the dpkg bugs fixed before
the installer's default could be changed.  But that ship has sailed.

However, I think there's still value from a process-management
perspective in declaring that non-merged systems may not be converted,
and are to remain supported, until all critical components of the
distribution -- in particular dpkg -- fully support the merged state.
For instance, it means that the proponents of the transition have a
concrete incentive to get *all* of the remaining work done, and not
just the piece that matters to them.

zw

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Luca Boccassi]

On Wed, 2022-09-28 at 09:51 +0200, Helmut Grohne wrote:
> Hi Luca,
> 
> As much as I agree with you on other matters...
> 
> On Tue, Sep 27, 2022 at 09:11:18PM +0100, Luca Boccassi wrote:
> > baseless, patently false statements - I frankly find it quite upsetting
> > to see claimed that "we have refused to fix any bug" as a self-evident
> > fact, when even a cursory look at the distribution packages/bugs
> > trackers in the past couple of months tells a very different story.
> 
> This is not as clear cut as it may seem and we have disagreed on this
> before. In earlier days, we had multiple disagreements about whether
> something actually is a bug. And when it came to fix dpkg-shlibdeps,
> there clearly was a lack of action until Raphael and Guillem handled the
> matter in despair. I agree that the broad accusation is distant from the
> truth.  Recent history has a lot of quick responses and fixes even in
> the face of inappropriate communication styles used by others.  However,
> when the thing to touch is dpkg, I think refusal is the right term.
> Admittedly, you do have reasons to refuse, but that's still refusal. We
> keep these bugs (whose severity we disagree about) and have mitigations
> and workarounds for them in place.
>
> This is a niche aspect of the whole matter. I think it deserves
> correction, but it doesn't change the broad picture that there are
> little news in this report that would change the CTTE evaluation of the
> matter.

Adding mitigations, workarounds and enhancing our test suites to detect
possible issues in my book does count as working on it. It might not be
some people's preferred form of contribution, but on the other hand
there's no universal law of physics demanding that everything needs to
be fixed to the complete satisfaction of any and all passerbys. Also,
none of you are paying my salary ;-)

Could the mentioned issues happen before? Maybe. Can they happen now,
after the work that has been done? No (again to the best of my
knowledge, reproducers that can fly undetected are always welcome).
Ignoring this does not strike me as a good way to start a conversation,
much less to make demands (not referring to you, obviously).

> There also is one more recent disagreement that keeps popping up here
> and there. We used to have a bootstrap sequence that did not require
> auxiliary setup code. Initially, the transition was deployed by usrmerge
> and all was fine from a bootstrap point of view. Now with usr-is-merged,
> this interface is broken. From my point of view, this is a significant
> regression and when I've been discussing this with proponents, the
> reaction could reasonably be described as refusal to recognize the
> existence of a problem. I fear we need to revisit this at some point.
> The transition is also not complete until that auxiliary code is gone.

This will sound like a long-winded rant but please bear with me. In
modern OS design the end goal has to be that vendors ship under /usr
and optionally /etc (see the concept of hermetic /usr and/or /etc).
Everything else happens at the moment a system is instantiated. And
that does not only include directory layout like /var, /root, /home and
so on - which cannot and must not be shipped by packages to make this
work - but it _must_ include setting up security mechanisms such as
encryption and attestation.

Linux is woefully and embarassingly behind Windows and OSX on the
security front these days, and it's a shame because for a long time we
were so far ahead. For starters, we need to move toward transparent and
automated encryption (and much more) by default or we will keep falling
behind, and we can't live forever off the meme that "Linux is more
secure by default" when that is patently not true anymore, because at
some point the zeitgeist will catch up.

The key point I want to put across is that the idea that you can get
from zero to a working, finalized system _exclusively_ by installing
packages is not something that is fit for the modern world of
computing.

You should of course be able to create an atomic, hermetic vendor tree
(/usr) just by installing packages. My hope is that with bookworm+1 we
will be in a place where no packages install anything under /bin, /sbin
and /lib*, and everything has moved, so that we will be in that place
again. How we will get there, well, there's a few options - one thing
at a time.

> In the end, we probably agree to disagree on what it means to have this
> transition finished. I expect that from your point of view, the
> transition is now finished given that init-system-helper forces the
> migration. I very much disagree that we're done. The question now is who
> will do the work to finish it and who will refuse to do that work?

At the cost of sounding like management - depends on your definition of
"finished". My top-priority goal is that from bookworm onward we can
assume and take for granted that all systems are converted and we do
not have to keep supporting both 

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Luca Boccassi]

> On Tue, 2022-09-27 at 18:25 +0200, Andreas Metzler wrote:
> > On 2022-09-27 Zack Weinberg  wrote:
> > [...]
> > > What I am asking for is a schedule change: specifically, that the
> > > merged /usr transition not be allowed to proceed past the status
> > > quo as of two weeks ago (i.e. *before* init-system-helpers added
> a
> > > dependency on usrmerge|usr-is-merged) until after the dpkg bugs
> are
> > > fixed to the satisfaction of the dpkg maintainers.
> > [...]
> > 
> > Hello Zack,
> > 
> > Afaiui the only thing the change two weeks caused is an increased
> > percentage of usrmerged Debian installations.
> > 
> > Afaict the problem is unchanged: There is a very large number of
> > usrmerged systems (every system installed with bullseye installer
> or
> > newer unless some very specific steps were taken to avoid this)
> which
> > are prone to bugs due to dpkg not having been changed *first*. This
> > number is of usrmerged systems is so large that we cannot mark them
> > as unsupported ("Please reinstall"). Whether this percentage is 25%
> > or 90% does not matter.
> 
> You can easily revert any system having usrmerge installed with dpkg-
> fsys-usrunmess. This should be known by all Debian users, by some
> suitable channel.
> 
> And for example the latest init-system-helpers release should add
> this
> to the package description (if not reverted). This applies to other
> present and future packages having usrmerge as a dependency too.

Please note that this will result in an unsupported system, as per CTTE
decision. It is important to note this, for the record. So no, that
will not be added anywhere, package description or otherwise, and in
fact the last time it was "made known by all Debian users" it caused
quite a lot of damage, and was forcefully reverted.

To elaborate, in practice this means that after bookworm, things can
start assuming that all systems are merged-usr, and it also means that
changes of any kind will very likely not be held back on the off-chance
that they might not work on unmerged systems (and no testing will be
required to detect that either). There are already taints in place to
detect unmerged systems.

In other words, one is of course free to do as they wish on their
systems, but they also get to keep the pieces.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Svante Signell]

On Tue, 2022-09-27 at 18:25 +0200, Andreas Metzler wrote:
> On 2022-09-27 Zack Weinberg  wrote:
> [...]
> > What I am asking for is a schedule change: specifically, that the
> > merged /usr transition not be allowed to proceed past the status
> > quo as of two weeks ago (i.e. *before* init-system-helpers added a
> > dependency on usrmerge|usr-is-merged) until after the dpkg bugs are
> > fixed to the satisfaction of the dpkg maintainers.
> [...]
> 
> Hello Zack,
> 
> Afaiui the only thing the change two weeks caused is an increased
> percentage of usrmerged Debian installations.
> 
> Afaict the problem is unchanged: There is a very large number of
> usrmerged systems (every system installed with bullseye installer or
> newer unless some very specific steps were taken to avoid this) which
> are prone to bugs due to dpkg not having been changed *first*. This
> number is of usrmerged systems is so large that we cannot mark them
> as unsupported ("Please reinstall"). Whether this percentage is 25%
> or 90% does not matter.

You can easily revert any system having usrmerge installed with dpkg-
fsys-usrunmess. This should be known by all Debian users, by some
suitable channel.

And for example the latest init-system-helpers release should add this
to the package description (if not reverted). This applies to other
present and future packages having usrmerge as a dependency too.

Thanks!

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Helmut Grohne]

Hi Luca,

As much as I agree with you on other matters...

On Tue, Sep 27, 2022 at 09:11:18PM +0100, Luca Boccassi wrote:
> baseless, patently false statements - I frankly find it quite upsetting
> to see claimed that "we have refused to fix any bug" as a self-evident
> fact, when even a cursory look at the distribution packages/bugs
> trackers in the past couple of months tells a very different story.

This is not as clear cut as it may seem and we have disagreed on this
before. In earlier days, we had multiple disagreements about whether
something actually is a bug. And when it came to fix dpkg-shlibdeps,
there clearly was a lack of action until Raphael and Guillem handled the
matter in despair. I agree that the broad accusation is distant from the
truth.  Recent history has a lot of quick responses and fixes even in
the face of inappropriate communication styles used by others.  However,
when the thing to touch is dpkg, I think refusal is the right term.
Admittedly, you do have reasons to refuse, but that's still refusal. We
keep these bugs (whose severity we disagree about) and have mitigations
and workarounds for them in place.

This is a niche aspect of the whole matter. I think it deserves
correction, but it doesn't change the broad picture that there are
little news in this report that would change the CTTE evaluation of the
matter.

There also is one more recent disagreement that keeps popping up here
and there. We used to have a bootstrap sequence that did not require
auxiliary setup code. Initially, the transition was deployed by usrmerge
and all was fine from a bootstrap point of view. Now with usr-is-merged,
this interface is broken. From my point of view, this is a significant
regression and when I've been discussing this with proponents, the
reaction could reasonably be described as refusal to recognize the
existence of a problem. I fear we need to revisit this at some point.
The transition is also not complete until that auxiliary code is gone.

In the end, we probably agree to disagree on what it means to have this
transition finished. I expect that from your point of view, the
transition is now finished given that init-system-helper forces the
migration. I very much disagree that we're done. The question now is who
will do the work to finish it and who will refuse to do that work?

Helmut

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sean Whitton]

Hello,

On Tue 27 Sep 2022 at 09:23AM +01, Matthew Vernon wrote:

> As Sean says, though, questions of what are and aren't RC bugs are typically
> the domain of the release team, not the TC.

I didn't intend to say that -- I think that Policy+TC decides bug
severities, in general.  But the RT decide which ones actually count for
an impeding release.

(One might then argue that that means that the RT do in fact determine
which are the RC bugs, but I think that unhelpfully simplifies how
Debian decision-making works.)

> I don't think you're asking us to revisit our decision on the approach
> taken to merged-/usr; we don't generally return to a decision once
> made (and a GR would normally be the approach to overturning a TC
> decision). Personally, I think there are circumstances where we might
> (e.g. a convincing argument that we missed something critical in our
> decision-making, or that circumstances have changed sufficiently to
> warrant another look), but I don't think we are in that situation here
> at the moment.
>
> I think the best way to proceed would be to open a bug describing the problem
> that Simon outlines with RC severity; the relevant maintainers and release
> team can then discuss how to resolve the issues and if they warrant delaying
> the release or adjusting when we complete the transition. Obviously those
> people might want to ask the TC for advice, but I think that would be up to
> them at least in the first instance.

Thanks for this helpful perspective.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sean Whitton]

Hello,

On Tue 27 Sep 2022 at 10:07AM -04, Zack Weinberg wrote:

> I may have misunderstood the TC recommendation here.  I was under the
> impression that the “no migration of file paths” rule was *only* in
> effect until the release of bookworm, and that it was motivated by the
> need to continue supporting non-merged systems up till that point, not
> by the dpkg bugs.

No, it remains in place beyond that, and a decision on lifting it has
not been made by anyone.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Gunnar Wolf]

Christoph Berg dijo [Tue, Sep 27, 2022 at 10:23:32PM +0200]:
> Re: Sean Whitton
> > Therefore, we will likely just close this bug, I'm afraid.
> 
> +1 on closing from me.

I agree this bug should be closed. I won't comment more, as there is
not much more to add without going in circles back to
already-discussed issues.


signature.asc
Description: PGP signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sean Whitton]

Hello,

On Mon 26 Sep 2022 at 04:48PM -04, Zack Weinberg wrote:

> I'm surprised to hear you say that, given that, in the past, the TC
> has required bugs of various severities to be filed, and has required
> maintainers not to alter bug severities.  Almost all of what I'm
> asking for would follow "by operation of Policy", as it were, from the
> requested s:critical bugs on usrmerge and usr-is-merged; I only
> spelled them out for explicitness's sake.  And I didn't file the bugs
> myself because they would certainly be rejected by the maintainers,
> and then I'd have to escalate _that_ to you, so I'm trying to save time
> by skipping that step.

This isn't about Policy, though, it's about timetabling, as you say
downthread, and that's basically why we think the RT is the most
relevant decision-making body -- they're the team with the timetables.

> In my opinion, a "suitable transition mechanism" _must_ include a fix
> for the dpkg bugs,

Many TC and RT members basically agree with you, including myself, and
lament the lack of integration of merged-/usr with dpkg.  That's behind
the idea in the RT's d-d-a e-mail where they said (paraphrasing) "the
transition is not complete until the work on dpkg is complete."  But
they decided that less was to be done for bookworm: the transition is to
be left incomplete in just this way.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Christoph Berg]

Re: Sean Whitton
> Therefore, we will likely just close this bug, I'm afraid.

+1 on closing from me.

Christoph

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sebastian Ramacher]

On 2022-09-27 10:26:36 -0400, Zack Weinberg wrote:
> On Tue, Sep 27, 2022, at 5:15 AM, Sebastian Ramacher wrote:
> >> I'd like to make sure that the bug submitter has not identified
> >> something new here.
> >
> > I've not seen any new issues appearing since the last round I file bugs.
> 
> I wasn’t aware that you have been filing bugs related to the
> transition.  What criteria are you using to find and file those bugs?

I only checked for packages violating the moratorium. Thankfully a check
for that was implemented by Luca in piuparts. If we have additional
patterns that could cause issues for upgrades, the check would ideally
be extended with that information.

Cheers
-- 
Sebastian Ramacher

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Luca Boccassi]

> This particular issue in dpkg is very much known and nothing new (a
> short recap: dpkg can lose files if files are moved between packages
> *and* symlinked directores, such as / and /usr, at the same time).
> 
> 
> To mitigate it, bluca added a piuparts check which rejects packages
> that move files from / to /usr (for bookworm/sid). This is overly
> conservative as strictly speaking, we'd only have to be careful of
> packages that move files from / to /usr and between packages within
> one release cycle. I.e. it would be perfectly fine to move files for
> / to /usr if during a release cycle those files aren't moved between
> packages. I suspect this will be a rare case, that said definitely
> something to be aware of if we don't get a fixed dpkg.
> 
> Fwiw, once all files are moved to /usr, the dpkg bug is no longer
> really relevant.
> 
> So, I think there isn't any new information here in #1020792 which
> would warrant a halt.

Indeed, there is nothing new reported here, it's a mix of old news -
none of the failure modes mentioned can actually happen given the
piuparts check that has been in place for a few months (and if someone
can prove the opposite please let us know and we'll update it), and
baseless, patently false statements - I frankly find it quite upsetting
to see claimed that "we have refused to fix any bug" as a self-evident
fact, when even a cursory look at the distribution packages/bugs
trackers in the past couple of months tells a very different story.
Also there were several status updates sent over time, with detailed
progress reports, that were linked in the d-d-a mail - pretty hard to
miss. In short tons of work has happened, and continue to happen, and
seeing it casually dismissed with a shrug is honestly quite
disheartening.

Finally, extraordinary claims require extraordinary evidence: given it
was claimed that "all systems will be rendered unbootable", it should
be trivial to show it. Provide the log of an upgrade bullseye ->
bookworm that fails to boot. Should be easy enough, given it
*allegedly* affects all systems (despite of course nobody ever having
seen anything remotely like it, ever, over the course of several
years), no? We'll be eagerly waiting for a detailed and evidence-based
report.

In the meanwhile, I'd humbly suggest close+wontfix.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Andreas Metzler]

On 2022-09-27 Zack Weinberg  wrote:
[...]
> What I am asking for is a schedule change: specifically, that the
> merged /usr transition not be allowed to proceed past the status quo
> as of two weeks ago (i.e. *before* init-system-helpers added a
> dependency on usrmerge|usr-is-merged) until after the dpkg bugs are
> fixed to the satisfaction of the dpkg maintainers.
[...]

Hello Zack,

Afaiui the only thing the change two weeks caused is an increased
percentage of usrmerged Debian installations.

Afaict the problem is unchanged: There is a very large number of
usrmerged systems (every system installed with bullseye installer or
newer unless some very specific steps were taken to avoid this) which
are prone to bugs due to dpkg not having been changed *first*. This
number is of usrmerged systems is so large that we cannot mark them as
unsupported ("Please reinstall"). Whether this percentage is 25% or 90%
does not matter.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Tue, Sep 27, 2022, at 5:15 AM, Sebastian Ramacher wrote:
>> I'd like to make sure that the bug submitter has not identified
>> something new here.
>
> I've not seen any new issues appearing since the last round I file bugs.

I wasn’t aware that you have been filing bugs related to the
transition.  What criteria are you using to find and file those bugs?

If you have time to put into actively *looking* for bugs *other than*
the one identified by Simon, that would be hugely helpful and I think
a good way to go about it would be to write a fuzzer for package
upgrades — try to generate the weirdest possible scenarios of packages
splitting, recombining, replacing each other, conflicting with each
other, transferring files among each other, diverting each other’s
files, etc. etc.  Using that, see if you can come up with concrete
reproducers for all the data loss scenarios listed by Guillem at
, or for *other* data
loss scenarios, hitherto unsuspected by anyone.

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Tue, Sep 27, 2022, at 4:23 AM, Matthew Vernon wrote:
> Thanks for bringing this to the committee; even if Sean is correct that
> we won't act on this report, you've described the issues clearly and I
> think it was worth bringing to our attention.

Thank you for saying so.

> As Sean says, though, questions of what are and aren't RC bugs are
> typically the domain of the release team, not the TC.
>
> I don't think you're asking us to revisit our decision on the approach
> taken to merged-/usr

No, I am not.  To be excruciatingly specific, I do not want to
re-litigate the decision on “merged /usr via aliased dirs” vs
“merged /usr via moves and symlink farms”.

What I am asking for is a schedule change: specifically, that the
merged /usr transition not be allowed to proceed past the status quo
as of two weeks ago (i.e. *before* init-system-helpers added a
dependency on usrmerge|usr-is-merged) until after the dpkg bugs are
fixed to the satisfaction of the dpkg maintainers.

I do not think that this is “returning to a decision once made” and I
*do* think that this is within the ambit of the TC’s responsibilities.
(Concretely, the TC would be overruling Luca Bonassi on the subject of
whether init-system-helpers may depend on usrmerge|usr-is-merged, and
issuing more project-wide advice on the timing and the blockers for
the transition.)

> I think the best way to proceed would be to open a bug describing the
> problem that Simon outlines with RC severity; the relevant maintainers
> and release team can then discuss how to resolve the issues and if they
> warrant delaying the release or adjusting when we complete the
> transition. Obviously those people might want to ask the TC for advice,
> but I think that would be up to them at least in the first instance.

This gets into interpersonal issues.  Supposing I filed a critical-
severity bug on any or all of dpkg, usrmerge, or init-system-helpers,
I fully expect that the respective maintainer would immediately close
it as either not their problem or not actually a problem at all.
I would then have to come back here and request that you override
their determination of the severity and/or appropriate package to
carry the bug.  Having the TC act first would save everyone involved
some steps, wouldn’t it?

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

[Procedural note: I’m very busy with my day job this week, so I will
be responding to messages related to this report in batch mode, once a
day.]

On Mon, Sep 26, 2022, at 4:49 PM, Sam Hartman wrote:
>> "Sean" == Sean Whitton  writes:
>
> Sean> - you might be lacking the full context of TC-involving
> Sean> discussions over the past few months, but so far as I can see,
> Sean> you are asking for us to undo a decision that we only just
> Sean> made, which doesn't make sense.
>
> Sean, as someone who has tried to follow this for a while, I'm confused
> by a third thing.
> Doesn't the TC recommendation given additional weight by the RT that we
> not migrate file paths until the dpkg bug has been fixed  make it
> impossible for the pre-conditions for disappearing files to happen?

I may have misunderstood the TC recommendation here.  I was under the
impression that the “no migration of file paths” rule was *only* in
effect until the release of bookworm, and that it was motivated by the
need to continue supporting non-merged systems up till that point, not
by the dpkg bugs.

If the rule is in effect until the specific dpkg bug identified by
Simon is fixed, then that does make the situation *somewhat* less
dire, but I still stand on the request I made, because of:

>>Several other equally dire scenarios
>>are listed in https://wiki.debian.org/Teams/Dpkg/MergedUsr under the
>>“merged-/usr-via-aliased-dirs” subheading, albeit only tersely.)
>
> I think that if you are going to ask for drastic action like you are
> doing, a turse mention on a wiki page is not sufficient documentation.
> Simon's mail got significant attention because he actually worked
> through and documented the situation.
> I and I believe several people who have looked at this situation believe
> that the RT guidance will prevent the issue Simon raised from happening.

I think this may get at the heart of the disconnect between my
perspective and yours.  You are apparently content to think that the
issue Simon identified is the only issue that actually needs a remedy.
I, on the other hand, am assuming that this is only one of potentially
dozens of similar data-loss scenarios (not necessarily dozens of
*bugs*; perhaps just several other scenarios that tickle the same bug)
and that most of them are *not* addressed by the “no migration of file
paths” rule.

I believe this, fundamentally, because I believe Guillem Jover.
You say that “if you are going to ask for drastic action like
you are doing, a terse mention on a wiki page is not sufficient
documentation”, and that would be fair if it was me who wrote
the wiki page, but it was Guillem who wrote it, and *he’s the
dpkg maintainer*.  If he says there’s a situation where, quote,
“dpkg and dpkg-divert are currently broken by this approach, will
fail to notice file conflicts and will overwrite files” then I
take that seriously, and what I want here is fundamentally for
the project as a whole to take Guillem’s concerns seriously.
This is why I said “to the satisfaction of the dpkg maintainers”
in my original request.

In particular, it should not be on me, or Guillem, to demonstrate
how each of the problems listed on the wiki page can be triggered.
It should be *the proponents of usrmerge* who are held responsible
for fully investigating just how many data loss scenarios there are,
and for proposing solutions.  Given that they have spent the last five
years, possibly more, *refusing* to do so, I think it is entirely
reasonable for the project to demand a halt to their proposed change
until they have done all of the due diligence that they *should* have
done five years ago.

I know that usrmerge has been planned for rather *longer* than five
years now, but here’s the thing: No properly-written Unix program
has any reason to *care* whether /bin is a symlink to /usr/bin.
Therefore there is no justification for pursuing usrmerge with any
kind of urgency.  I can see the value from a theoretical system
integration design perspective, and I don’t mind it happening
*eventually*, but I do very much care that it not render my personal
workstation (which has tracked unstable for 25 years now with no
serious issues) unbootable.

(Postscript: Let me be clear that I’m just a user of the distribution.
I’ve never even considered becoming a DD.  But as a longtime contributor
to several key upstream projects for *all* Linux distributions, and as
someone who *has* been running Debian unstable on his personal workstation
for 25 years, I’d like to think that I know what I’m talking about when
I say things like “no properly-written Unix program has any reason to care
whether /bin is a symlink to /usr/bin”.)

zw

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Michael Biebl]

This particular issue in dpkg is very much known and nothing new (a 
short recap: dpkg can lose files if files are moved between packages 
*and*  symlinked directores, such as / and /usr, at the same time).



To mitigate it, bluca added a piuparts check which rejects packages that 
move files from / to /usr (for bookworm/sid).
This is overly conservative as strictly speaking, we'd only have to be 
careful of packages that move files from / to /usr and between packages 
within one release cycle.
I.e. it would be perfectly fine to move files for / to /usr if during a 
release cycle those files aren't moved between packages.
I suspect this will be a rare case, that said definitely something to be 
aware of if we don't get a fixed dpkg.


Fwiw, once all files are moved to /usr, the dpkg bug is no longer really 
relevant.


So, I think there isn't any new information here in #1020792 which would 
warrant a halt.


Regards,
Michael



OpenPGP_signature
Description: OpenPGP digital signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sebastian Ramacher]

On 2022-09-26 14:49:15 -0600, Sam Hartman wrote:
> > "Sean" == Sean Whitton  writes:
> 
> Sean> - you might be lacking the full context of TC-involving
> Sean> discussions over the past few months, but so far as I can see,
> Sean> you are asking for us to undo a decision that we only just
> Sean> made, which doesn't make sense.
> 
> Sean, as someone who has tried to follow this for a while, I'm confused
> by a third thing.
> Doesn't the TC recommendation given additional weight by the RT that we
> not migrate file paths until the dpkg bug has been fixed  make it
> impossible for the pre-conditions for disappearing files to happen?

Indeed, there is a moratorium in place for this case.

> I'd like to make sure that the bug submitter has not identified
> something new here.

I've not seen any new issues appearing since the last round I file bugs.

Cheers

> 
> >Also, to the bug submitter:
> >needs to be fixed urgently.  Several other equally dire scenarios
> >are listed in https://wiki.debian.org/Teams/Dpkg/MergedUsr under the
> >“merged-/usr-via-aliased-dirs” subheading, albeit only tersely.)
> 
> I think that if you are going to ask for drastic action like you are
> doing, a turse mention on a wiki page is not sufficient documentation.
> Simon's mail got significant attention because he actually worked
> through and documented the situation.
> I and I believe several people who have looked at this situation believe
> that the RT guidance will prevent the issue Simon raised from happening.
> 
> I suspect that if someone were to clearly document a different situation
> that had not previously been considered, people would look at it.
> I'd hope the TC would be open to considering newly documented technical
> problems as an example.
> But because the decision has been made, the bar for blocking things is a
> lot higher at this point than turse descriptions on a wiki page.
> 

-- 
Sebastian Ramacher

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sebastian Ramacher]

On 2022-09-27 09:23:47 +0100, Matthew Vernon wrote:
> Hi Zack,
> 
> Thanks for bringing this to the committee; even if Sean is correct that we
> won't act on this report, you've described the issues clearly and I think it
> was worth bringing to our attention.
> 
> On 26/09/2022 20:28, Zack Weinberg wrote:
> 
> > It has been known for some time that dpkg has bugs which prevent it
> > from correctly handling merged-/usr systems.  #858331/#848622 is the
> > only such bug (that I can find) that has actually been recorded in the
> > BTS, and it *appears* to be a relatively harmless problem, affecting
> > only dpkg-query output.
> 
> This much is uncontroversial.
> 
> > However, Simon Richter  showed in
> > https://lists.debian.org/debian-devel/2021/08/msg00326.html that the > bad 
> > dpkg-query output is only the most obvious symptom of a much more
> > serious problem, and that, under conditions that will plausibly occur
> > in the archive after the release of bookworm (assuming all continues
> > as presently planned) upgrading packages on a merged-/usr system can
> > cause packaged files to disappear from the filesystem.  The files most
> > likely to be affected are the files that are currently packaged in
> > /bin, /sbin, and /lib, including, just to mention a few, /bin/bash,
> > /bin/systemd, and /lib/$ARCH/libc.so.6.  Thus, the dpkg bugs can
> > render systems unbootable on upgrade, and should be considered
> > critical severity.
> 
> This is a very useful message, and (at least to my mind) makes it clearer
> how more serious problems might well occur.
> 
> As Sean says, though, questions of what are and aren't RC bugs are typically
> the domain of the release team, not the TC.
> 
> I don't think you're asking us to revisit our decision on the approach taken
> to merged-/usr; we don't generally return to a decision once made (and a GR
> would normally be the approach to overturning a TC decision). Personally, I
> think there are circumstances where we might (e.g. a convincing argument
> that we missed something critical in our decision-making, or that
> circumstances have changed sufficiently to warrant another look), but I
> don't think we are in that situation here at the moment.
> 
> I think the best way to proceed would be to open a bug describing the
> problem that Simon outlines with RC severity; the relevant maintainers and
> release team can then discuss how to resolve the issues and if they warrant
> delaying the release or adjusting when we complete the transition. Obviously
> those people might want to ask the TC for advice, but I think that would be
> up to them at least in the first instance.

Is there a package in the archive that has this issue? If so, can you
point me to a bug report?

Cheers
-- 
Sebastian Ramacher

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Matthew Vernon]

Hi Zack,

Thanks for bringing this to the committee; even if Sean is correct that 
we won't act on this report, you've described the issues clearly and I 
think it was worth bringing to our attention.


On 26/09/2022 20:28, Zack Weinberg wrote:


It has been known for some time that dpkg has bugs which prevent it
from correctly handling merged-/usr systems.  #858331/#848622 is the
only such bug (that I can find) that has actually been recorded in the
BTS, and it *appears* to be a relatively harmless problem, affecting
only dpkg-query output.


This much is uncontroversial.


However, Simon Richter  showed in
https://lists.debian.org/debian-devel/2021/08/msg00326.html that the > bad 
dpkg-query output is only the most obvious symptom of a much more
serious problem, and that, under conditions that will plausibly occur
in the archive after the release of bookworm (assuming all continues
as presently planned) upgrading packages on a merged-/usr system can
cause packaged files to disappear from the filesystem.  The files most
likely to be affected are the files that are currently packaged in
/bin, /sbin, and /lib, including, just to mention a few, /bin/bash,
/bin/systemd, and /lib/$ARCH/libc.so.6.  Thus, the dpkg bugs can
render systems unbootable on upgrade, and should be considered
critical severity.


This is a very useful message, and (at least to my mind) makes it 
clearer how more serious problems might well occur.


As Sean says, though, questions of what are and aren't RC bugs are 
typically the domain of the release team, not the TC.


I don't think you're asking us to revisit our decision on the approach 
taken to merged-/usr; we don't generally return to a decision once made 
(and a GR would normally be the approach to overturning a TC decision). 
Personally, I think there are circumstances where we might (e.g. a 
convincing argument that we missed something critical in our 
decision-making, or that circumstances have changed sufficiently to 
warrant another look), but I don't think we are in that situation here 
at the moment.


I think the best way to proceed would be to open a bug describing the 
problem that Simon outlines with RC severity; the relevant maintainers 
and release team can then discuss how to resolve the issues and if they 
warrant delaying the release or adjusting when we complete the 
transition. Obviously those people might want to ask the TC for advice, 
but I think that would be up to them at least in the first instance.


Thanks,

Matthew

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Mon, Sep 26, 2022, at 4:30 PM, Sean Whitton wrote:
> I believe that this request is invalid, for two reasons:
>
> - the specific things you ask for are all or mostly things that we think
>   are currently up to the Release Team, and the TC cannot override
>   delegates

I'm surprised to hear you say that, given that, in the past, the TC
has required bugs of various severities to be filed, and has required
maintainers not to alter bug severities.  Almost all of what I'm
asking for would follow "by operation of Policy", as it were, from the
requested s:critical bugs on usrmerge and usr-is-merged; I only
spelled them out for explicitness's sake.  And I didn't file the bugs
myself because they would certainly be rejected by the maintainers,
and then I'd have to escalate _that_ to you, so I'm trying to save time
by skipping that step.

> - you might be lacking the full context of TC-involving discussions over
>   the past few months, but so far as I can see, you are asking for us to
>   undo a decision that we only just made, which doesn't make sense.

As far as _I_ am aware, I'm acting on this bit of the "more specific
advice re #978636", issued last year:

# If a suitable transition mechanism is not available by the time the
# Debian 12 freeze is reached, the Technical Committee will rescind our
# advice in #978636 and modify our advice in this resolution to reflect
# the situation that has been reached. We hope that this will not be
# necessary.

In my opinion, a "suitable transition mechanism" _must_ include a fix
for the dpkg bugs, and it is almost certain at this point that the
dpkg bugs will _not_ be fixed in time for the freeze, hence it is
appropriate at this point for the TC to "rescind our advice in #978636
and modify our advice in this resolution", along the lines I requested.

If there has been substantive discussion or progress on the dpkg bugs
"over the past few months" I have not seen it either on -ctte or on
-devel.  I would appreciate a concrete summary of what has happened so
far and what is planned to happen.

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sam Hartman]

> "Sean" == Sean Whitton  writes:

Sean> - you might be lacking the full context of TC-involving
Sean> discussions over the past few months, but so far as I can see,
Sean> you are asking for us to undo a decision that we only just
Sean> made, which doesn't make sense.

Sean, as someone who has tried to follow this for a while, I'm confused
by a third thing.
Doesn't the TC recommendation given additional weight by the RT that we
not migrate file paths until the dpkg bug has been fixed  make it
impossible for the pre-conditions for disappearing files to happen?

I'd like to make sure that the bug submitter has not identified
something new here.

>Also, to the bug submitter:
>needs to be fixed urgently.  Several other equally dire scenarios
>are listed in https://wiki.debian.org/Teams/Dpkg/MergedUsr under the
>“merged-/usr-via-aliased-dirs” subheading, albeit only tersely.)

I think that if you are going to ask for drastic action like you are
doing, a turse mention on a wiki page is not sufficient documentation.
Simon's mail got significant attention because he actually worked
through and documented the situation.
I and I believe several people who have looked at this situation believe
that the RT guidance will prevent the issue Simon raised from happening.

I suspect that if someone were to clearly document a different situation
that had not previously been considered, people would look at it.
I'd hope the TC would be open to considering newly documented technical
problems as an example.
But because the decision has been made, the bar for blocking things is a
lot higher at this point than turse descriptions on a wiki page.

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Sean Whitton]

Hello,

On Mon 26 Sep 2022 at 03:28PM -04, Zack Weinberg wrote:

> Package: tech-ctte
> Severity: normal
> X-Debbugs-Cc: z...@owlfolio.org
>
> I formally request that the Technical Committee call a halt to the
> merged-/usr transition until such time as all of the bugs in dpkg that
> can, on a merged-/usr system, cause damage to the contents of the
> filesystem (e.g. packaged files being silently deleted on upgrade)
> have been fixed to the satisfaction of the dpkg maintainers.

I believe that this request is invalid, for two reasons:

- the specific things you ask for are all or mostly things that we think
  are currently up to the Release Team, and the TC cannot override
  delegates

- you might be lacking the full context of TC-involving discussions over
  the past few months, but so far as I can see, you are asking for us to
  undo a decision that we only just made, which doesn't make sense.

Therefore, we will likely just close this bug, I'm afraid.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

Package: tech-ctte
Severity: normal
X-Debbugs-Cc: z...@owlfolio.org

I formally request that the Technical Committee call a halt to the
merged-/usr transition until such time as all of the bugs in dpkg that
can, on a merged-/usr system, cause damage to the contents of the
filesystem (e.g. packaged files being silently deleted on upgrade)
have been fixed to the satisfaction of the dpkg maintainers.

## Background

It has been known for some time that dpkg has bugs which prevent it
from correctly handling merged-/usr systems.  #858331/#848622 is the
only such bug (that I can find) that has actually been recorded in the
BTS, and it *appears* to be a relatively harmless problem, affecting
only dpkg-query output.

However, Simon Richter  showed in
https://lists.debian.org/debian-devel/2021/08/msg00326.html that the
bad dpkg-query output is only the most obvious symptom of a much more
serious problem, and that, under conditions that will plausibly occur
in the archive after the release of bookworm (assuming all continues
as presently planned) upgrading packages on a merged-/usr system can
cause packaged files to disappear from the filesystem.  The files most
likely to be affected are the files that are currently packaged in
/bin, /sbin, and /lib, including, just to mention a few, /bin/bash,
/bin/systemd, and /lib/$ARCH/libc.so.6.  Thus, the dpkg bugs can
render systems unbootable on upgrade, and should be considered
critical severity.

(N.B. Simon’s message is the only thing I can find that gives
step-by-step instructions to reproduce a problem that could plausibly
cause catastrophic filesystem damage during package upgrades, but the
scenario it describes should _not_ be considered the only problem that
needs to be fixed urgently.  Several other equally dire scenarios
are listed in https://wiki.debian.org/Teams/Dpkg/MergedUsr under the
“merged-/usr-via-aliased-dirs” subheading, albeit only tersely.)

Despite the severity of the dpkg bugs having been known since
_at least_ August of 2021, the people working on the merged-/usr
transition have made almost no effort to fix them.  A patch exists
(https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=858331;filename=file_move_moratorium.patch;msg=46)
that at least partially addresses the problem, but only one desultory
attempt was made to get it merged.

The people working on the merged-/usr transition have repeatedly
claimed that the several years of experience Debian now has with
systems that were _originally installed_ with merged /usr, and the
somewhat longer baseline for Ubuntu doing the same thing, indicates
the dpkg bugs are not a problem in practice.  That claim is incorrect.
The dpkg bugs are *latent* right now because, until the actual release
of bookworm, packages have to continue supporting non-merged systems.
Therefore they cannot move files from /{bin,sbin,lib} to
/usr/{bin,sbin,lib}, which is one of the conditions required to
trigger the most serious known issue (disappearance of packaged files).

I anticipate that if the merged-/usr transition proceeds as planned,
*all systems* that track either testing or unstable will be rendered
unbootable at least once in the bookworm+1 development cycle.  This
level of fallout is not acceptable, even in potentiality.  Since it
seems clear that the people working on the merged-/usr transition have
no intention of getting the bugs fixed, project-level action is
required.

## Specific actions requested

I ask the Committee to require all of the following, partially
reversing the decision taken in #978636, and overriding maintainers
as necessary:

 - The usrmerge and usr-is-merged packages are to be removed from
   testing immediately, and severity:critical (justification: “makes
   unrelated software break” and/or “breaks the entire system”) bugs
   are to be filed to prevent them from re-entering testing.

 - Those bugs may not be closed, nor may their severity be lowered,
   until *the dpkg maintainers agree* that dpkg can now fully support
   merged-/usr systems.

 - No package may declare a dependency on either usrmerge or
   usr-is-merged until the respective bug is closed.  Any packages in
   the archive that currently declare such a dependency must drop it
   immediately (or else be removed from testing as well).

 - No *other* mechanism which converts existing non-merged-/usr
   installations to merged-/usr, as a side-effect of package upgrade
   or installation, may enter testing, until at least the usr-is-merged
   bug is closed.  [I can imagine at least one possible resolution
   which would make it safe to use dpkg on a system where /usr has
   been merged ever since installation, but *not* on a system that was
   converted the way the usrmerge package does it.]

 - If merged-/usr conversion fails to reach testing before the release
   freeze for bookworm, then bookworm shall continue to support
   non-merged /usr for its lifetime, and similarly for future
   (post-bookworm) releases.

 - Optionally: