Re: [PATCH] schroot: created merged-/usr chroots for trixie and beyond

[Aurelien Jarno]

Hi Helmut,

On 2023-10-07 12:47, Helmut Grohne wrote:
> The CTTE has ruled that from trixie onward, maintainers may rely on
> systems being merged-/usr. This includes the build environment.
> ---
>  modules/schroot/files/setup-dchroot | 12 +++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> Hi DSA,
> 
> would you consider applying the patch to dsa-puppet.git? It is meant to
> convert trixie and newer chroots to merged-/usr. We also meant to
> implement this change by updating debootstrap in bullseye and bookworm
> via -pu, but the presence of --no-merged-usr spoiled this plan. As such
> I propose moving forward with this.
> 
> d-devel Cced to let people know what's happening and to keep a public
> record of this patch.

I have just applied your patch. The chroots are going to get
automatically regenerated tonight at 22:13 UTC.

Regards
Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://aurel32.net

Re: [PATCH] schroot: created merged-/usr chroots for trixie and beyond

[Simon McVittie]

On Sat, 07 Oct 2023 at 12:47:04 +0200, Helmut Grohne wrote:
> would you consider applying the patch to dsa-puppet.git? It is meant to
> convert trixie and newer chroots to merged-/usr.

This partially reverts my earlier change
<https://salsa.debian.org/dsa-team/mirror/dsa-puppet/-/commit/b54f52d2899c5785923c804fdfbba0782c147da4>.

I confirm that the change Helmut proposes is now appropriate.

Thanks,
smcv

[PATCH] schroot: created merged-/usr chroots for trixie and beyond

[Helmut Grohne]

The CTTE has ruled that from trixie onward, maintainers may rely on
systems being merged-/usr. This includes the build environment.
---
 modules/schroot/files/setup-dchroot | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

Hi DSA,

would you consider applying the patch to dsa-puppet.git? It is meant to
convert trixie and newer chroots to merged-/usr. We also meant to
implement this change by updating debootstrap in bullseye and bookworm
via -pu, but the presence of --no-merged-usr spoiled this plan. As such
I propose moving forward with this.

d-devel Cced to let people know what's happening and to keep a public
record of this patch.

Helmut

diff --git a/modules/schroot/files/setup-dchroot 
b/modules/schroot/files/setup-dchroot
index d6e61f5a8..0f2330506 100755
--- a/modules/schroot/files/setup-dchroot
+++ b/modules/schroot/files/setup-dchroot
@@ -342,6 +342,16 @@ case "$suite" in
 ;;
 esac
 
+mergedusr=--merged-usr
+case "$suite" in
+  jessie|stretch|buster|bullseye|bookworm)
+mergedusr=--no-merged-usr
+;;
+  trusty|utopic|vivid|wily|xenial|yakkety|zesty|artful|bionic|cosmic)
+mergedusr=--no-merged-usr
+;;
+esac
+
 bindir=$(mktemp -d)
 cleanup+=("rm -r $bindir")
 cat > "$bindir/wget" << 'EOF'
@@ -357,7 +367,7 @@ PATH="$bindir:$PATH" \
 --include="$include" \
 --variant=buildd \
 --arch="$arch" \
---no-merged-usr \
+"$mergedusr" \
 "$suite" "$rootdir" "$mirror" "$script"
 echo "$tuple" > "$rootdir/etc/debian_chroot"
 echo "force-unsafe-io" > "$rootdir/etc/dpkg/dpkg.cfg.d/force-unsafe-io"
-- 
2.42.0

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Nikolaus Rath]

On May 17 2023, Andrea Pappacoda  wrote:
> Hi all,
>
> first of all thank you for this great thread. While I could feel some tension 
> while
> reading it, it's completely normal and I've learned a lot.
>
> I have a question though: if /lib64/ld-linux-x86-64.so.2 is already a symlink 
> on
> non-merged-/usr systems, pointing to 
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, why would
> it be an issue to have /lib64/ld-linux-x86-64.so.2 point to 
> /usr/lib64/ld-linux-x86-64.so.2?

I don't think it would be, and I don't think anyone else is saying it would be.

> Why do we want binaries to look for the loader in
> /usr/lib64/ld-linux-x86-64.so.2 if that would still be a symlink to
> /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2?

My understanding is that there is a desire for the /lib64 symlink not to
be needed, because it would simplify bootstrapping new systems.


Best,
Nikolaus
-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:
> On Wed, 17 May 2023 at 01:05, Russ Allbery  wrote:

>> I do think the industry is moving away (well, has already moved away)
>> from Linux Standards Base pre-compiled C binaries without wrappers like
>> snap or flatpak, although there are some very notable exceptions, such
>> as Steam's special use cases explained elsewhere on the thread.  I
>> don't believe the industry is moving away from Go and Rust binaries, so
>> this is, I think, a more complicated picture than your description
>> indicates.

> I know absolutely nothing about Go, and preciously little about Rust,
> but my understanding from news article and so on was that both of those
> ecosystems pretty openly avoid "traditional" distributions, in favour of
> their own self-contained ecosystems.

I think apt-cache search will show that's a significant overstatement and
quite a lot of Go and Rust code is packaged for Debian.  Lots of Debian
maintainers put a lot of hard work into that.

> So would that use case be affected, even by an hypothetical and at this
> point extremely unlikely archive-wide change?

If we only modified specific core packages to use a different PT_INTERP,
obviously only those packages would be affected and execution of general
Go and Rust binaries would not be affected.  I suspect none of those core
packages today are Go or Rust.  (I do think it's likely that we'll have
critical bootstrapping packages written in Rust within the next decade,
though.)

If all Debian packages were built with a different PT_INTERP but the
toolchain itself was not modified, then Go and Rust binaries packaged for
Debian could not be copied to and run on non-Debian Linux systems (unless
it happened to work by accident because those Linux systems were also
/usr-merged and thus the non-canonical PT_INTERP also was present on those
systems).  This would be unfortunate and surprising and I'm sure that it
would break use cases that currently work, but it probably wouldn't be
catastrophic.  Still, it would obviously be a regression.

If the toolchain shipped with Debian were modified to build binaries with
a different PT_INTERP, then Go and Rust binaries built with that toolchain
would not work on other Linux distributions.  If every other major Linux
distribution were /usr-merged and thus the modified PT_INTERP worked
somewhat by accident on most other distributions, this likewise would
still break some things for some people, but similar to the previous point
it would be a regression but probably wouldn't be catastrophic.  It would
be more serious than the previous problem, though, since now we're
affecting user-built binaries and not just packaged binaries.  Many more
of those will be built with the intent to copy them to other systems.

If changes caused Go and Rust binaries built on Debian to not work on
numerous major non-Debian Linux distributions, I would consider that a
serious release-critical bug.  But that doesn't seem all that likely from
this specific change assuming that the modified PT_INTERP would also be
present on other major Linux distributions such as Red Hat, Fedora, SuSE,
Arch, Gentoo, etc.  I am *assuming* those are all /usr-merging, and thus
it might be, but I have not tracked this and do not know if that's true.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Wed, 17 May 2023 at 01:05, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > It does say something interesting. When we started, the assertion was
> > that packages not relying on the symlink being present was fundamental
> > for portability and cross-compatibility. Then, it shrinked to
> > portability and cross-compatibility of a subset of packages. Now it
> > further shrank to portability and cross-compatibility of a subset of
> > packages of a subset of vintages.
>
> I think it's pretty clear that I'm totally failing to communicate what I
> was getting at with this example and the process is becoming incredibly
> frustrating, so I'm going to stop trying here.
>
> > But does it really work, or do we just hope it does? I personally see a
> > great deal of signals that say it doesn't. For one, in 2023, "build one
> > binary and run it everywhere" doesn't really seem to me to be the
> > principal choice in terms of portability.
>
> Well, believe what you believe, but I literally do that daily, as does
> anyone else who regularly uses software from a Rust or Go ecosystem.  Not
> a single work day goes by without me running, on some random Ubuntu or Red
> Hat or Debian system, binaries that were compiled on some random other
> Linux distribution (often I have no idea which one).
>
> > It might have been once, and there might be a number of relevant use
> > cases still, but it seems to me the industry has largely gone in a very
> > different direction nowadays.
>
> I do think the industry is moving away (well, has already moved away) from
> Linux Standards Base pre-compiled C binaries without wrappers like snap or
> flatpak, although there are some very notable exceptions, such as Steam's
> special use cases explained elsewhere on the thread.  I don't believe the
> industry is moving away from Go and Rust binaries, so this is, I think, a
> more complicated picture than your description indicates.

I know absolutely nothing about Go, and preciously little about Rust,
but my understanding from news article and so on was that both of
those ecosystems pretty openly avoid "traditional" distributions, in
favour of their own self-contained ecosystems. So would that use case
be affected, even by an hypothetical and at this point extremely
unlikely archive-wide change? Meaning, are those randomly compiled
Go/Rust binaries you run for work taken from the Debian archive, or
locally/externally/self compiled? I do know we do ship a number of
those, although I am very oblivious to the number and nature of that
subset. And conversely, if the hypothetical change was just to the
essential set, I assume there's no Go/Rust there, right?

> > Not everything of course, but wouldn't it be worth it to specify core
> > items such as cross-compatibility requirements? Not in terms of
> > implementation details, but in terms of what the goals and the minimum
> > expectations are. There are obviously some, and they are obviously of
> > great importance to many, and yet to me it feels like they shift every
> > time I ask a different question.
>
> I am certainly never going to say no to more maintained documentation.
> That would be great.  If you're feeling motivated and you're willing to
> really listen to what people are saying and accept and understand a lot of
> ambiguity and a lot of assumptions that don't match your assumptions, go
> for it!
>
> I personally am not even doing justice to Policy in its existing scope, so
> this isn't something I will personally be tackling.  Honestly, I should
> have spent all of the time I spent on this thread working on Policy
> instead.  :)

Sure, I can add that to my todo list. But I have already two policy
items open for review, so certainly won't get to it before those are
sorted, hint hint :-)
(probably it would have been best not to mention it, lol)

> > But if you prefer to focus on first principles, that's great! I've been
> > asking this a lot: is cross-distribution harmonization a core
> > requirement? Is it important enough to trump other goals and
> > distro-local benefits? If so, isn't it worth to discuss it and then
> > pencil it down?
>
> I think some parts of it are a core requirement.  It would be very
> surprising, and very bad, if we couldn't run Go and Rust binaries built on
> another distribution, for example, or if Go or Rust binaries built on
> Debian couldn't be used on other distributions, both subject to the normal
> constraints of glibc cross-compatibility that everyone building binaries
> already knows about.  I think other parts of it are not a core
> requirement, but still something that is nice to have and that we
> shouldn't break without a really good reason.
>
> I think the specific details of the Linux Standards Base process mostly
> didn't turn into something the Linux world wanted to support going
> forward, and thus LSB harmonization, while an interesting idea, is no
> longer a requirement in general.  However, we still follow some pieces of
> it that 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Roger Lynn]

On 15/05/2023 19:00, Simon McVittie wrote:
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
>> People build things on Debian that are not Debian packages. People
>> compile binaries on Debian, and expect them to work on any system that
>> has sufficiently new libraries.
> 
> *raises hand*
> 
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.

As another much smaller example, which I hope is still relevant, I have
taken binaries from Debian stable or oldstable armel and run them for
diagnostic purposes on embedded boards which only have Busybox installed and
for which I don't have a convenient build environment.

Regards,
Roger

PS Apologies if I've followed up incorrectly - I read debian-devel through
an NNTP gateway. - And I screwed up the reply so Simon and the bug got a
different copy.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Jeremy Stanley  writes:

> Throwing another common one on the heap, similar to the previous Steam
> example, Python wheels with compiled extensions are often distributed on
> PyPI for a fictional "manylinux" platform which indicates they're
> intended to be usable on most GNU/Linux distributions (though in their
> case, they compile on ~ancient RHEL versions to achieve that). One way
> that manulinux wheels are less like Rust or Go is that they typically
> don't use static compilation, but rather vendor in additional
> dynamically-linked libs which are unlikely to be present on the target
> installations.

Yeah, I've not been digging too deeply into cases like this since they
wouldn't be affected by the original proposal that started this off (it
wouldn't change binaries compiled with normal Debian tools or change what
ABI Debian can execute), but if we move into the more general question of
"does anyone care about building binaries on one Linux system and running
them on another," I think there are quite a lot of examples.

Another one just from the Python world is conda, which provides a parallel
set of pre-compiled libraries in addition to the Python modules and
expects to be able to install and run them on arbitrary systems outside of
a chroot or container, just by manipulating the dynamic loader search
path.  Conda is very widely used in scientific computing, where a lot of
use cases require Python libraries with substantial underlying C library
components that often do not have compatible or sufficiently optimized
versions packaged for the underlying distribution.

There are also, obviously, non-free software use cases galore, including I
suspect some in the Debian non-free archive, where binaries built on other
Linux distributions are being run on Debian systems.  I remember, in a
previous job, changing a few files in /etc to convince the Oracle
installer that a Debian system was actually a Red Hat system so that it
would run, and the resulting binaries then worked fine (and it would have
been a huge problem for us if they hadn't).  I think Oracle subsequently
added more support for Debian, or at least Ubuntu, but I'm sure there are
other cases like that still around today.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Jeremy Stanley]

On 2023-05-16 17:05:25 -0700 (-0700), Russ Allbery wrote:
[...]
> Well, believe what you believe, but I literally do that daily, as
> does anyone else who regularly uses software from a Rust or Go
> ecosystem.  Not a single work day goes by without me running, on
> some random Ubuntu or Red Hat or Debian system, binaries that were
> compiled on some random other Linux distribution (often I have no
> idea which one).
[...]

Throwing another common one on the heap, similar to the previous
Steam example, Python wheels with compiled extensions are often
distributed on PyPI for a fictional "manylinux" platform which
indicates they're intended to be usable on most GNU/Linux
distributions (though in their case, they compile on ~ancient RHEL
versions to achieve that). One way that manulinux wheels are less
like Rust or Go is that they typically don't use static compilation,
but rather vendor in additional dynamically-linked libs which are
unlikely to be present on the target installations.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Andrea Pappacoda]

Hi all,

first of all thank you for this great thread. While I could feel some 
tension while reading it, it's completely normal and I've learned a lot.


I have a question though: if /lib64/ld-linux-x86-64.so.2 is already a 
symlink on non-merged-/usr systems, pointing to 
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, why would it be an issue to 
have /lib64/ld-linux-x86-64.so.2 point to 
/usr/lib64/ld-linux-x86-64.so.2? Why do we want binaries to look for 
the loader in /usr/lib64/ld-linux-x86-64.so.2 if that would still be a 
symlink to /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2?


Sorry in advance if this sounds like a dumb question, and thanks again 
for your involvement in all of this!

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:

> It does say something interesting. When we started, the assertion was
> that packages not relying on the symlink being present was fundamental
> for portability and cross-compatibility. Then, it shrinked to
> portability and cross-compatibility of a subset of packages. Now it
> further shrank to portability and cross-compatibility of a subset of
> packages of a subset of vintages.

I think it's pretty clear that I'm totally failing to communicate what I
was getting at with this example and the process is becoming incredibly
frustrating, so I'm going to stop trying here.

> But does it really work, or do we just hope it does? I personally see a
> great deal of signals that say it doesn't. For one, in 2023, "build one
> binary and run it everywhere" doesn't really seem to me to be the
> principal choice in terms of portability.

Well, believe what you believe, but I literally do that daily, as does
anyone else who regularly uses software from a Rust or Go ecosystem.  Not
a single work day goes by without me running, on some random Ubuntu or Red
Hat or Debian system, binaries that were compiled on some random other
Linux distribution (often I have no idea which one).

> It might have been once, and there might be a number of relevant use
> cases still, but it seems to me the industry has largely gone in a very
> different direction nowadays.

I do think the industry is moving away (well, has already moved away) from
Linux Standards Base pre-compiled C binaries without wrappers like snap or
flatpak, although there are some very notable exceptions, such as Steam's
special use cases explained elsewhere on the thread.  I don't believe the
industry is moving away from Go and Rust binaries, so this is, I think, a
more complicated picture than your description indicates.

> Not everything of course, but wouldn't it be worth it to specify core
> items such as cross-compatibility requirements? Not in terms of
> implementation details, but in terms of what the goals and the minimum
> expectations are. There are obviously some, and they are obviously of
> great importance to many, and yet to me it feels like they shift every
> time I ask a different question.

I am certainly never going to say no to more maintained documentation.
That would be great.  If you're feeling motivated and you're willing to
really listen to what people are saying and accept and understand a lot of
ambiguity and a lot of assumptions that don't match your assumptions, go
for it!

I personally am not even doing justice to Policy in its existing scope, so
this isn't something I will personally be tackling.  Honestly, I should
have spent all of the time I spent on this thread working on Policy
instead.  :)

> But if you prefer to focus on first principles, that's great! I've been
> asking this a lot: is cross-distribution harmonization a core
> requirement? Is it important enough to trump other goals and
> distro-local benefits? If so, isn't it worth to discuss it and then
> pencil it down?

I think some parts of it are a core requirement.  It would be very
surprising, and very bad, if we couldn't run Go and Rust binaries built on
another distribution, for example, or if Go or Rust binaries built on
Debian couldn't be used on other distributions, both subject to the normal
constraints of glibc cross-compatibility that everyone building binaries
already knows about.  I think other parts of it are not a core
requirement, but still something that is nice to have and that we
shouldn't break without a really good reason.

I think the specific details of the Linux Standards Base process mostly
didn't turn into something the Linux world wanted to support going
forward, and thus LSB harmonization, while an interesting idea, is no
longer a requirement in general.  However, we still follow some pieces of
it that were properly implemented (like the FHS), and while we shouldn't
do that blindly forever (if for no other reason than the FHS is no longer
maintained), it's also valuable to not change that too fast and to only
break compatibility with now-widely-expected file system layout properties
when we have a really good reason.  Ideally, we would pick some smaller
subset of the LSB that still matters and agree with other major
distributions on some points of compatibility to, at the very least, help
ease the common problem of needing to administer multiple systems running
different Linux distributions.

There is no one answer to whether this trumps other goals and distro-local
benefits.  It depends on what those benefits are and what those goals are
and how important they are.  For Guix, obviously their immutable tree and
hard dependency pinning are more important to them than cross-distro
compatibility, and given their goals, that seems like an entirely
reasonable decision.  I would disagree vehemently with that decision for
Debian because Debian is not Guix.

In other words, it depends.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Tue, 16 May 2023 at 09:27, Simon McVittie  wrote:
>
> On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> > This sounds like a very interesting use case, and the first real one
> > mentioned, which is great to see - but I do not fully follow yet, from
> > what you are saying it seems to me that what you need is for your
> > binaries to use the usual pt_interp, that bit is clear. But why does
> > it matter if /usr/bin/ls on the host uses a different one?
>
> We don't need to run the ls from the host, but we do need to run
> glibc-related executables like ldconfig and localedef from either the
> host or the container runtime, whichever is newer. Because glibc is
> a single source package, executables and libraries within the glibc
> bubble sometimes make use of private symbols in libraries that are also
> within the glibc bubble (and IMO they have a right to do so), even though
> executables from outside glibc would be discouraged or disallowed from
> doing so. This means that when we have chosen a particular version of
> glibc (which, again, must be whichever one is newer), we try to use its
> matching version for *everything* originating in the glibc source package.
>
> In principle we could get exactly the same situation if we've imported a
> library from the host system (as a dependency of the graphics stack) that
> calls an executable as a subprocess and expects it to be >= the version
> it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

Thanks for the clarification, so if I understood correctly, your use
case is that sometimes (eg: when they are newer) you pull binaries
(eg: ldconfig) from the host, and run them from the container? So, in
case let's say ldconfig on the host points to /usr/lib/ld, but because
your container is not usr-merged, it wouldn't find the interpreter and
fail?

> The wider point than my specific use-case, though, is that when there's a
> standard, you can't predict what other software authors have looked at the
> statement "you can rely on this" and ... relied on it. See also Russ's
> (as ever, excellent) mails to the same thread.
>
> I appreciate that you are trying to explore the edges of the
> problem/constraint space and say "what if we did this, could that work?",
> and it's good that you are doing that; but part of that process is
> working with the other people on this list when they say "no, we can't
> do that because...", and respecting their input.

I respect and appreciate the input, but I want to understand it too,
hence the "because..." part is what I was looking for - so thanks for
providing it, it is really useful.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Tue, 16 May 2023 at 04:22, Russ Allbery  wrote:
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> >> Note that we're not talking about complicated packages with lots of
> >> runtime like, say, Emacs.  As I understand it your proposal wouldn't
> >> change PT_INTERP for that binary anyway.  We're presumably talking
> >> about the kind of binaries that you need to bootstrap a minimal system,
> >> so packages like coreutils or bash.  And I would indeed expect those
> >> binaries to be generally portable, as long as the same critical shared
> >> libraries are available on other systems (in this case, PCRE2 and
> >> ncurses).
>
> > Is that really the case? Let's test that hypothesis:
>
> I think you may have not read my whole message before replying, and also
> please assume that I know really basic facts about glibc compatibility and
> am not referring to that.
>
> I said "of a similar vintage" (farther down in my message) because of
> course we all know that binaries built against newer versions of glibc
> don't run on systems with older versions of glibc (and likewise for shared
> libraries in general and thus libselinux), and you tested a Debian
> unstable package on an Ubuntu system from 2020.  This says nothing
> interesting and has nothing to do with my point.

It does say something interesting. When we started, the assertion was
that packages not relying on the symlink being present was fundamental
for portability and cross-compatibility. Then, it shrinked to
portability and cross-compatibility of a subset of packages. Now it
further shrank to portability and cross-compatibility of a subset of
packages of a subset of vintages.
Why is the requirement that libselinux and glibc are not too recent
fine to have on coreutils, but the requirement that there's a symlink
is out of the question? If anything, the latter is trivial to add, so
the formers should feel more "disruptive" and game-breaking, no?

> > Whoops. Does this make coreutils rc-buggy now? ;-)
>
> You are the only person who is talking about RC bugs.  The bar here is not
> "prove to me that this is RC buggy," the bar is "you have to prove to a
> bunch of Debian core maintainers that they should break the ABI in their
> packages" (not to mention the additional small but permanent build
> complexity).  Demanding they prove to you that it's a bad idea is not how
> this works.

It's a tongue-in-cheek comment, I had hoped the emojii would give that away.

> The point of standards like an ABI is that a bunch of entirely unrelated
> people who never talk to each other and never look at each other's
> software are allowed to rely on them and assume no one else will break
> them.  This is how free software scales; without invariants that everyone
> can rely on without having to explain how they're relying on them, it is
> much more difficult to get an ecosystem to work together.  We don't just
> break things because they don't seem important; the space of people who
> may be relying on this standard is unknowable, which is the entire point.
> Opening those boxes is really expensive (in time, planning, communication,
> debugging, and yes, compatibility) and we should only do it when it
> really, really matters.

But does it really work, or do we just hope it does? I personally see
a great deal of signals that say it doesn't. For one, in 2023, "build
one binary and run it everywhere" doesn't really seem to me to be the
principal choice in terms of portability. It might have been once, and
there might be a number of relevant use cases still, but it seems to
me the industry has largely gone in a very different direction
nowadays.

> > It did look like a veto to me. More importantly, isn't relying on
> > passersby to spot alleged harmful changes dangerous, especially for
> > undocumented, uncodified and untested use cases, like unspecified and
> > vague cross-compatibility requirements?
>
> I'm honestly boggled.  This is a thread on debian-devel, which is
> literally how we do architecture vetting in this project.
>
> I absolutely do not think that we can write down everything of importance
> in designing a distribution so that we don't have to have conversations
> with other people in the project who have deep expertise when considering
> a significant architectural change like changing the PT_INTERP path of
> core binaries.

Not everything of course, but wouldn't it be worth it to specify core
items such as cross-compatibility requirements? Not in terms of
implementation details, but in terms of what the goals and the minimum
expectations are. There are obviously some, and they are obviously of
great importance to many, and yet to me it feels like they shift every
time I ask a different question.

> >> I mostly jumped in because it felt like you and Steve were just yelling
> >> at each other and I thought I might be able to explain some of where he
> >> was coming from in a way that may make more sense.
>
> > I don't 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Didier 'OdyX' Raboud  writes:

> This has existed in a (now distant) past as the "Linux Distribution
> Checker", in the context of the Linux Standard Base, that Debian and
> Ubuntu stopped caring about in late 2015.

Ah, yes, thank you, that makes sense.

> I'm not aware of more recent efforts in that direction; but it's an
> understatement to say the landscape has changed quite a bit since:
> containers, sandbox environments (and others) have forever changed the
> way we think about distributing binary executable. LSB had that
> ambition, and failed.

While that is certainly true, I feel like the pendulum may be swinging
back in a slightly different way with Go and Rust popularizing the idea
that you should be able to copy around a binary and run it on any Linux
system with a compatible architecture.  This is a much smaller problem
than LSB was trying to solve since LSB was trying to standardize things
like the shared library ABI and SONAMEs, which Go and Rust intentionally
avoid with static linking.  But they do rely very deeply on every system
being able to execute binaries built to the Linux ABI and glibc.  (I
realize that's a different question than the one discussed in this
thread.)

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Didier 'OdyX' Raboud]

Le mardi, 16 mai 2023, 17.06:38 h CEST Russ Allbery a écrit :
> I don't know if anyone has written an ABI compliance test for binaries.
> That sounds like something that would be in scope for the Linux Test
> Project, though, and it's possible their existing tests do some of this.

This has existed in a (now distant) past as the "Linux Distribution Checker", 
in the context of the Linux Standard Base, that Debian and Ubuntu stopped 
caring about in late 2015.

I'm not aware of more recent efforts in that direction; but it's an 
understatement to say the landscape has changed quite a bit since: containers, 
sandbox environments (and others) have forever changed the way we think about 
distributing binary executable. LSB had that ambition, and failed.

-- 
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

James Addison  writes:

> We've almost certainly all encountered limitations in upstream
> specifications and wondered when it's worth attempting a perceived
> improvement despite potential friction.

> If Debian did want/need to change the PT_INTERP path, is there a way to
> achieve that in both a standards-compliant and also a distro-compatible
> way?

My recollection is that we considered this when starting with multiarch
but none of the other distributions outside of Ubuntu were very enthused,
so we dropped it.  I saw those discussions on the glibc development lists,
which is not really the place for detailed x86_64 ABI discussion but which
would certainly be an interested party and has roughly the right people
present.  If I saw a problem that would need to be addressed with an ABI
change and didn't have anyone else to ask, that's where I personally would
start, but the Debian gcc and glibc maintainers would probably know more.

The bar for justifying a change will be fairly high, based on past
discussions.  I would expect it to require some sort of significant bug
that can't be worked around in another way.

> Although tests may not exist publicly in this case, the idea of using
> tests where possible to catch regressions seems relevant to me.  Tests
> can help people to identify when a compatibility boundary has been
> reached or overstepped, and, socially, they can provide a clear place to
> gather discussion if & when they become outdated (particularly if the
> tests are themselves are provided free and open source).  Copying
> binaries and running them seems like a form of testing, but perhaps we
> could find better ways.

I don't know if anyone has written an ABI compliance test for binaries.
That sounds like something that would be in scope for the Linux Test
Project, though, and it's possible their existing tests do some of this.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> This sounds like a very interesting use case, and the first real one
> mentioned, which is great to see - but I do not fully follow yet, from
> what you are saying it seems to me that what you need is for your
> binaries to use the usual pt_interp, that bit is clear. But why does
> it matter if /usr/bin/ls on the host uses a different one?

We don't need to run the ls from the host, but we do need to run
glibc-related executables like ldconfig and localedef from either the
host or the container runtime, whichever is newer. Because glibc is
a single source package, executables and libraries within the glibc
bubble sometimes make use of private symbols in libraries that are also
within the glibc bubble (and IMO they have a right to do so), even though
executables from outside glibc would be discouraged or disallowed from
doing so. This means that when we have chosen a particular version of
glibc (which, again, must be whichever one is newer), we try to use its
matching version for *everything* originating in the glibc source package.

In principle we could get exactly the same situation if we've imported a
library from the host system (as a dependency of the graphics stack) that
calls an executable as a subprocess and expects it to be >= the version
it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

The wider point than my specific use-case, though, is that when there's a
standard, you can't predict what other software authors have looked at the
statement "you can rely on this" and ... relied on it. See also Russ's
(as ever, excellent) mails to the same thread.

I appreciate that you are trying to explore the edges of the
problem/constraint space and say "what if we did this, could that work?",
and it's good that you are doing that; but part of that process is
working with the other people on this list when they say "no, we can't
do that because...", and respecting their input.

Thanks,
smcv

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[James Addison]

On Tue, 16 May 2023 at 04:22, Russ Allbery  wrote:
>
> > It did look like a veto to me. More importantly, isn't relying on
> > passersby to spot alleged harmful changes dangerous, especially for
> > undocumented, uncodified and untested use cases, like unspecified and
> > vague cross-compatibility requirements?
>
> I'm honestly boggled.  This is a thread on debian-devel, which is
> literally how we do architecture vetting in this project.
>
> I absolutely do not think that we can write down everything of importance
> in designing a distribution so that we don't have to have conversations
> with other people in the project who have deep expertise when considering
> a significant architectural change like changing the PT_INTERP path of
> core binaries.

We've almost certainly all encountered limitations in upstream
specifications and wondered when it's worth attempting a perceived
improvement despite potential friction.

If Debian did want/need to change the PT_INTERP path, is there a way
to achieve that in both a standards-compliant and also a
distro-compatible way?

And if there isn't: how would we resolve that?


This conversation is already lengthy, and based on recent experience
I'm definitely the kind of person who doesn't always read all the
details - so I don't know whether it's a good idea for anyone to
respond to my message.  But I haven't seen anyone discussing or
providing a safe migration path.

Although tests may not exist publicly in this case, the idea of using
tests where possible to catch regressions seems relevant to me.  Tests
can help people to identify when a compatibility boundary has been
reached or overstepped, and, socially, they can provide a clear place
to gather discussion if & when they become outdated (particularly if
the tests are themselves are provided free and open source).  Copying
binaries and running them seems like a form of testing, but perhaps we
could find better ways.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Bastian Blank]

Hi Steve

On Mon, May 15, 2023 at 02:01:15AM +0100, Steve McIntyre wrote:
> Russ has described copying *binaries* out of packages and running them
> elsewhere. I've done that too, from time to time. This is one of the
> things made possible by the ABI contract being followed.

And nothing in that requires that the interpreter matches.  Because
glibc is clever enough to allow running the interpreter standalone.  So
in any way you can just call:

/lib64/ld-linux-x86-64.so.2 /bin/ls

Bastian

-- 
Oh, that sound of male ego.  You travel halfway across the galaxy and
it's still the same song.
-- Eve McHuron, "Mudd's Women", stardate 1330.1

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

I'm dropping the TC bug from this thread, since I don't think it has
anything to do with that discussion at this point.  I probably should also
change the Subject line, but I'm keeping it to make it easier for the
people who want to tune out this thread, since I very much doubt they want
to tune back in at this point.

Luca Boccassi  writes:
> On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:

>> Note that we're not talking about complicated packages with lots of
>> runtime like, say, Emacs.  As I understand it your proposal wouldn't
>> change PT_INTERP for that binary anyway.  We're presumably talking
>> about the kind of binaries that you need to bootstrap a minimal system,
>> so packages like coreutils or bash.  And I would indeed expect those
>> binaries to be generally portable, as long as the same critical shared
>> libraries are available on other systems (in this case, PCRE2 and
>> ncurses).

> Is that really the case? Let's test that hypothesis:

I think you may have not read my whole message before replying, and also
please assume that I know really basic facts about glibc compatibility and
am not referring to that.

I said "of a similar vintage" (farther down in my message) because of
course we all know that binaries built against newer versions of glibc
don't run on systems with older versions of glibc (and likewise for shared
libraries in general and thus libselinux), and you tested a Debian
unstable package on an Ubuntu system from 2020.  This says nothing
interesting and has nothing to do with my point.

> Whoops. Does this make coreutils rc-buggy now? ;-)

You are the only person who is talking about RC bugs.  The bar here is not
"prove to me that this is RC buggy," the bar is "you have to prove to a
bunch of Debian core maintainers that they should break the ABI in their
packages" (not to mention the additional small but permanent build
complexity).  Demanding they prove to you that it's a bad idea is not how
this works.

The point of standards like an ABI is that a bunch of entirely unrelated
people who never talk to each other and never look at each other's
software are allowed to rely on them and assume no one else will break
them.  This is how free software scales; without invariants that everyone
can rely on without having to explain how they're relying on them, it is
much more difficult to get an ecosystem to work together.  We don't just
break things because they don't seem important; the space of people who
may be relying on this standard is unknowable, which is the entire point.
Opening those boxes is really expensive (in time, planning, communication,
debugging, and yes, compatibility) and we should only do it when it
really, really matters.

> It did look like a veto to me. More importantly, isn't relying on
> passersby to spot alleged harmful changes dangerous, especially for
> undocumented, uncodified and untested use cases, like unspecified and
> vague cross-compatibility requirements?

I'm honestly boggled.  This is a thread on debian-devel, which is
literally how we do architecture vetting in this project.

I absolutely do not think that we can write down everything of importance
in designing a distribution so that we don't have to have conversations
with other people in the project who have deep expertise when considering
a significant architectural change like changing the PT_INTERP path of
core binaries.

>> I mostly jumped in because it felt like you and Steve were just yelling
>> at each other and I thought I might be able to explain some of where he
>> was coming from in a way that may make more sense.

> I don't believe I've done any yelling here.

I'm using yelling pretty broadly and probably rather inaccurately here.
Maybe a better way of putting it is that Steve was yelling and you didn't
appear to be listening or understanding why he was yelling and were
responding in ways that were guaranteed to make him yell more.

You *are* coming across as kind of contemptuous of other people's
arguments, but then it's entirely possible that I am as well, so I'm
trying to ignore it.

> What if "setting the parking brake" is not enough, as the wheels are
> already off and rolling downhill, as shown above, because while
> everybody was looking at the parking brakes lever somebody ran off with
> the bolts that kept the wheels attached? Why is it worth worrying about
> compatibility with something that is already not compatible, and it's
> not expected to be compatible for almost all other aspects?

You can certainly decide to try to make the argument that ABI
compatibility between Linux distributions is a lost cause and we should
give up and not care about it any more.  That is a valid architectural
argument.  (To be clear, I don't think this is the argument you've been
making up to this point, which is about a very limited ABI break in
specific packages to achieve a specific purpose.)  I don't agree with that
argument, which doubtless doesn't come as a surprise, and your

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 18:54, Simon McVittie  wrote:
>
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
>
> *raises hand*
>
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.
>
> At the moment those binaries are built in ancient environments (based
> on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
> ubiquitous, we'll want to start relying on newer versions of Debian
> (which will still be very old versions *at the time*, but I'm sure that
> by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
> are not released yet). In this use-case, yes we do need to be using the
> interoperable ELF interpreter path.
>
> We were able to use (Ubuntu and) Debian for this *because* Debian is
> a relatively "ordinary" distribution that tends to follow cross-distro
> standards. The major counterexample is multiarch library paths, but
> multiarch library paths have some compelling technical advantages, and
> because there's no ambiguity about which architecture uses which directory,
> they're actually better for interop in some ways than /usr/lib (which
> is ambiguous, because the Red Hat family of distros expects to find 32-bit
> libraries there, but the Arch family expects 64-bit libraries, and it's
> not possible to construct a tree where both get what they want).
>
> I've spent a lot of the last 5 years working on putting Steam games in
> containers so that they'll work more reliably on all distros, including
> Debian, with less reliance on weird library search paths (although we
> still have to use binaries built on an ancient Debian derivative with a
> non-trivial RPATH to bootstrap the container environment). Because of
> constraints around recent GPUs needing current graphics drivers even
> if running on an otherwise ancient library stack, Steam's container
> runtime has to construct a hybrid environment where glibc is either the
> one from the host or the one from the container runtime environment,
> whichever is newer; and while doing that, we have experienced some
> broken situations that were caused by distributions diverging from the
> interoperable ELF interpreter. One concrete example is that Arch Linux
> uses the interoperable ELF interpreter for *almost* all executables,
> but uses a different ELF interpreter for executables from the glibc
> source package, for whatever reason; we were able to work around this,
> but for a while it caused Arch to fail to launch these containers where
> Debian/Fedora/etc. could.

This sounds like a very interesting use case, and the first real one
mentioned, which is great to see - but I do not fully follow yet, from
what you are saying it seems to me that what you need is for your
binaries to use the usual pt_interp, that bit is clear. But why does
it matter if /usr/bin/ls on the host uses a different one? It's your
executables that you ship as part of that runtime that are the entry
points that need the usual loader path for your chroot-on-steroids,
no? The loader would still be reachable as it always was in this
theoretical exercise. I am probably missing something in how this
works in details.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> >> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
> >> major player in Linux distributions, and I'm not sure how much they
> >> care about compatibility with anyone else.)
>
> > This is a counter-example to confute the assertion that *everybody* does
> > the same thing, which has been made multiple times. I'm not sure whether
> > it's an experiment or not, I mean if you ask their users/developers I
> > think they'd tell you it's very much production ready, but it is largely
> > irrelevant: it exists, and that's the only reason why it was mentioned,
> > as it shows that it is _possible_ to do that and be a working
> > distribution. Doesn't imply it's automatically desirable, but that was
> > not the intention.
>
> Ah, okay, I'm happy to agree with that point: you can violate the ABI and
> continue to be a working distribution.  (There are a lot of parts of the
> ABI that if you violated them you would not have a working distribution,
> but this is not one of them so far as I can tell.)
>
> > Not quite: my argument is that binaries from these packages are not
> > intended and not required to be ran on non-Debian systems, so there's no
> > incompatibility introduced in the first place - everything still works
> > where it is supposed to, exactly as it was before.
>
> I think we're saying the same thing but quibbling over phrasing.  I'd put
> that as saying that it's fine for the binaries of certain core Debian
> packages to be incompatible with the ABI because they're not intended to
> be used outside of Debian.  (In other words, I'm talking about
> incompatibility as a concrete, testable property of a binary, and I think
> you're talking about incompatibility as a more abstract concept of a
> distribution.)
>
> > No aggression intended whatsoever, sorry if it appeared that way. I am
> > trying to understand what the rules are.
>
> Well, the rule that I'd ideally set is don't break the ABI, even if it's
> not obvious why breaking the ABI is a bad idea or you can't see any bad
> consequences that could come from it, unless the reason for breaking the
> ABI is absolutely central to the mission and purpose of Debian.
>
> That said, it's not like we've never shipped a binary in Debian with a
> different PT_INTERP.  (I vaguely remember that some programming language
> uses PT_INTERP tricks for some sort of private binary scheme?  Objective
> CAML or something?  I ran across it once years ago and can't remember the
> details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
> situations that I don't remember the details of, although I don't think it
> does that with general binaries.)  So I do see your point that you would
> prefer the rule to be more pragmatic than that.
>
> My counterargument is that this proposal seems to mostly be about avoiding
> having to create a symlink at a critical point in the bootstrap process,
> and while it's tricky to get the timing right (and therefore kind of
> annoying), the resulting usable system has to have that symlink anyway (I
> think there's no disagreement about that).  Not following the ABI for core
> binaries seems like a scary change with unknown consquences to a bunch of
> core packages to solve what looks like a relatively minor (if admittedly
> annoying!) problem.
>
> Note that the target of PT_INTERP on Debian is *already* a symlink, to
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
> that we actually want to use.  We're already ensuring compatibility with a
> symlink and I think we should just keep doing that and not venture into
> these waters.

It's not even a proposal, it's a discussion to see "what would
actually break if X happened". That's why I'm asking for concrete use
cases, rather than theoretical points. Also, I am interested in what
the rules are. For example, glibc compatibility is just as important
if not more, why does that follow a different standard? If anything,
adding a missing symlink is trivial and a single command. Adding a
missing symbol to a shared object... not quite. There is an endless
list of downstream-only debianisms that break cross-compatibility in
just the same way (if not worse), and yet nobody cares about those.
Why?

> >> The world does not divide neatly into supported and unsupported use
> >> cases.  There are a lot of things I do to computers that I expect to
> >> work in some situations but not in others.  That includes, say, having
> >> a Debian chroot on some other OS and running binaries from that chroot
> >> without going into the chroot.  Often that will absolutely not work.
> >> Sometimes it will work, and it's convenient that it will work for some
> >> obscure recovery situations or other weird one-off use cases.  I've
> >> also copied files from working systems to broken systems running a
> >> different distribution before, 

Standards compliance (Was: Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited))

[Jeroen Dekkers]

On Mon, 15 May 2023 02:42:27 +0200,
Luca Boccassi wrote:
>
> On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
> > An obvious specific example of such a system would be one that didn't
> > merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> > path, but that's just one obvious example.  There may be others; the whole
> > point of an ABI is that you do not change things like this, not even if
> > you can't personally imagine why your change wouldn't be harmful.  There's
> > a whole process for changing an ABI that involves everyone else agreeing
> > as well, and unless one goes through that process, the ABI is what it is.
> > Debian not building ABI-compliant binaries would be highly surprising.
>
> That's self-evidently not true, as there are other distributions where
> that already happens, it's been already mentioned. Besides, we are not
> talking about sacred religious texts - the point is making things
> work. If they do, is it _really_ non-compliant/incompatible?

The point of a standard is definitely not that people ignore it and do things
that are in direct contradiction with the standard. The point of a standard is
that people agree on a specification and will also follow that specification.

Something working and something being compliant are different concepts. Software
can be compliant and not working and software can also be non-compliant and
working. Something that is in direct contradiction with a standard is definitely
non-compliant. Whether it works or not is not relevant for that.

Kind regards,

Jeroen Dekkers

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Mon, 15 May 2023 at 06:48:04 +0200, Johannes Schauer Marin Rodrigues wrote:
> Obviously, with Luca's proposal, binaries from packages built with a different
> dynamic linker path in them would not work on distributions without 
> merged-/usr
> symlinks. But if the property of stuff from Debian being able to run on
> non-Debian non-merged-/usr systems is an important one, then why was it okay 
> to
> have merged-/usr as the default? Because with merged-/usr we already changed
> the interface contract for a lot of things because now binaries and libraries
> can also be found at other locations than on non-merged-/usr systems. A script
> with a /usr/bin/bash shebang built on and for Debian will not work on a system
> without the symlinks.

pre-bookworm gcc writes /lib64/ld-linux-x86-64.so.2 into the ELF header
of amd64 binaries and I think post-bookworm gcc should continue to do so
even though that has never been the physical path to the ELF interpreter
on Debian (it's really /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, or
historically a version-numbered path). People who are only testing on
Debian systems, even in pre-merged-/usr releases like Debian 9, could
already have been relying on /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
existing; and now that we're saying merged-/usr is mandatory,
people who are only testing on Debian >= 12 could also start
relying on /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 or
/usr/lib64/ld-linux-x86-64.so.2; but we arrange for both/all paths
to exist, and we advise developers that they should only rely on the
interoperable /lib64/ld-linux-x86-64.so.2, treating all other paths that
lead to the same binary as an implementation detail.

I don't think it's any different to say that both /usr/bin/sh and /bin/sh
exist and will work, but ask that everyone should continue to use /bin/sh
and treat /usr/bin/sh as an implementation detail.

The way I think about merged /usr is a variant of Postel's principle:
be conservative in what you require, but be liberal about what will work.
A merged-/usr system can run software that assumes merged-/usr, and can
also run software that doesn't (because of the compatibility symlinks);
a non-merged-/usr system can run software that doesn't assume merged-/usr,
but software that makes that assumption will sometimes fail to work on it.

Consistent with that, despite being in favour of merged-/usr myself,
I didn't switch my development laptop or my autopkgtest VM images
to the merged-/usr setup until it became effectively mandatory
during the bookworm cycle - because if a package was doing something
not-maximally-portable, I wanted my development laptop or my test VM to
be one of the places it would fail to work, so that I would notice and
report that bug.

Conversely, I *did* switch non-development computers (servers and family
laptops) to merged-/usr, because on those systems the important thing is
for software to work, even if in theory it "doesn't deserve" to work.

On post-bookworm, merged-/usr Debian systems, if we keep using #!/bin/sh
and #!/usr/bin/perl scripts, and avoiding #!/usr/bin/sh and #!/bin/perl
scripts, then I think that's a positive thing for cross-distro interop -
and using the interoperable path to the ELF interpreter (dynamic linker)
is completely consistent with that.

As far as I can see, post-bookworm Lintian will continue to warn about
the non-interoperable spellings like #!/usr/bin/sh and #!/bin/perl,
unless changed to special-case them.

Note that the paths that are canonical from the point of view
of cross-distro interoperability (like #!/bin/sh) are not always
the same as the paths that are canonical from the point of view of
realpath(). *At the moment*, they are usually canonical from dpkg's
point of view, but that won't be the case forever, and I think that's
fine. /lib64/ld-linux-x86-64.so.2 isn't considered to be canonical by
either realpath() or dpkg either (dpkg knows it's a symlink, even on
non-merged-/usr systems where /lib64 is a real directory), but it's
canonical for the x86_64-linux-gnu ABI and that's what we say is most
important.

> With merged-/usr we already *did* "change fundamental things around" for
> reasons that are really not clear to me (but which i do not want to discuss
> here) and as a result did not "care about interoperability" (with those who do
> not also adopt it).

Didn't we? With merged /usr, the "theoretically correct" #!/bin/sh
scripts that have always worked will continue to work, and additionally,
"incorrect" #!/usr/bin/sh scripts will *also* work. That sounds like
caring about interoperability to me!

The one piece of interop we lose with merged-/usr becoming mandatory is
that if a developer has only tested on Debian (and other merged-/usr distros
like Fedora), if they have used a less-interoperable spelling of the path
like #!/usr/bin/sh or #!/bin/perl, their testing will not highligh

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.

*raises hand*

Hello, I represent an example of those people. With my $work hat on,
I'm involved in maintaining a family of Debian derivatives (the Steam
Runtime) that is used to run Steam games on arbitrary distributions,
including but not limited to Debian. Some of these binaries are built
on a Debian derivative, but run on an arbitrary other distribution,
using a RPATH[1] to find their non-glibc dependencies.

At the moment those binaries are built in ancient environments (based
on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
ubiquitous, we'll want to start relying on newer versions of Debian
(which will still be very old versions *at the time*, but I'm sure that
by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
are not released yet). In this use-case, yes we do need to be using the
interoperable ELF interpreter path.

We were able to use (Ubuntu and) Debian for this *because* Debian is
a relatively "ordinary" distribution that tends to follow cross-distro
standards. The major counterexample is multiarch library paths, but
multiarch library paths have some compelling technical advantages, and
because there's no ambiguity about which architecture uses which directory,
they're actually better for interop in some ways than /usr/lib (which
is ambiguous, because the Red Hat family of distros expects to find 32-bit
libraries there, but the Arch family expects 64-bit libraries, and it's
not possible to construct a tree where both get what they want).

I've spent a lot of the last 5 years working on putting Steam games in
containers so that they'll work more reliably on all distros, including
Debian, with less reliance on weird library search paths (although we
still have to use binaries built on an ancient Debian derivative with a
non-trivial RPATH to bootstrap the container environment). Because of
constraints around recent GPUs needing current graphics drivers even
if running on an otherwise ancient library stack, Steam's container
runtime has to construct a hybrid environment where glibc is either the
one from the host or the one from the container runtime environment,
whichever is newer; and while doing that, we have experienced some
broken situations that were caused by distributions diverging from the
interoperable ELF interpreter. One concrete example is that Arch Linux
uses the interoperable ELF interpreter for *almost* all executables,
but uses a different ELF interpreter for executables from the glibc
source package, for whatever reason; we were able to work around this,
but for a while it caused Arch to fail to launch these containers where
Debian/Fedora/etc. could.

This is not something that any of the distributions involved is
going to formally support, so the overall system consists of various
unsupported-but-works-in-practice things happening; but there is
absolutely no chance of Valve/Steam shipping separate binaries for
each distro (there are too many distros for that, even if you don't
count source-based distros like Gentoo that have an almost unlimited
number of concrete instantiations as binaries). However loudly we
might insist that our small subset-of-a-subset is the one they should
be targeting, what we're going to get is binaries that work on "mostly
interoperable" distros. As a result, if we want games on Linux (and
anything else requiring binary interop) to continue to be a thing,
pragmatically we should aim to be one of those "mostly interoperable"
distros, and encourage our friends and/or rivals in other distros to
do the same. Otherwise, the most likely outcome is for developers to go
back to an attitude of "I'm not going to support Linux, too many random
differences" and releasing for Windows only, and we all lose out.

As a result of all this, I would strongly prefer our compiler to
default to hard-coding the same interoperable ELF interpreter that
glibc upstream and the various major distros have agreed on (for example
/lib64/ld-linux-x86-64.so.2 on amd64), and I would also prefer it if we
can use that interoperable path in all the binaries we ship, including
src:glibc and the rest of the transitively Essential set.

One way that I like to think about this sort of thing is that we have a
finite number of "weird Debianism" tokens available, and we should aim
to spend them on the things that give us the best cost/benefit ratio.
We've never considered changing the ELF interpreter to be one of those,
to the extent of having a /lib64 solely for its benefit (on amd64)
even though by policy we don't generally use lib64.

(Incidentally, Steam's container runtime is always a merged-/usr
environment, to provide an environment that maximizes the 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:
> On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:

>> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
>> major player in Linux distributions, and I'm not sure how much they
>> care about compatibility with anyone else.)

> This is a counter-example to confute the assertion that *everybody* does
> the same thing, which has been made multiple times. I'm not sure whether
> it's an experiment or not, I mean if you ask their users/developers I
> think they'd tell you it's very much production ready, but it is largely
> irrelevant: it exists, and that's the only reason why it was mentioned,
> as it shows that it is _possible_ to do that and be a working
> distribution. Doesn't imply it's automatically desirable, but that was
> not the intention.

Ah, okay, I'm happy to agree with that point: you can violate the ABI and
continue to be a working distribution.  (There are a lot of parts of the
ABI that if you violated them you would not have a working distribution,
but this is not one of them so far as I can tell.)

> Not quite: my argument is that binaries from these packages are not
> intended and not required to be ran on non-Debian systems, so there's no
> incompatibility introduced in the first place - everything still works
> where it is supposed to, exactly as it was before.

I think we're saying the same thing but quibbling over phrasing.  I'd put
that as saying that it's fine for the binaries of certain core Debian
packages to be incompatible with the ABI because they're not intended to
be used outside of Debian.  (In other words, I'm talking about
incompatibility as a concrete, testable property of a binary, and I think
you're talking about incompatibility as a more abstract concept of a
distribution.)

> No aggression intended whatsoever, sorry if it appeared that way. I am
> trying to understand what the rules are.

Well, the rule that I'd ideally set is don't break the ABI, even if it's
not obvious why breaking the ABI is a bad idea or you can't see any bad
consequences that could come from it, unless the reason for breaking the
ABI is absolutely central to the mission and purpose of Debian.

That said, it's not like we've never shipped a binary in Debian with a
different PT_INTERP.  (I vaguely remember that some programming language
uses PT_INTERP tricks for some sort of private binary scheme?  Objective
CAML or something?  I ran across it once years ago and can't remember the
details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
situations that I don't remember the details of, although I don't think it
does that with general binaries.)  So I do see your point that you would
prefer the rule to be more pragmatic than that.

My counterargument is that this proposal seems to mostly be about avoiding
having to create a symlink at a critical point in the bootstrap process,
and while it's tricky to get the timing right (and therefore kind of
annoying), the resulting usable system has to have that symlink anyway (I
think there's no disagreement about that).  Not following the ABI for core
binaries seems like a scary change with unknown consquences to a bunch of
core packages to solve what looks like a relatively minor (if admittedly
annoying!) problem.

Note that the target of PT_INTERP on Debian is *already* a symlink, to
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
that we actually want to use.  We're already ensuring compatibility with a
symlink and I think we should just keep doing that and not venture into
these waters.

>> The world does not divide neatly into supported and unsupported use
>> cases.  There are a lot of things I do to computers that I expect to
>> work in some situations but not in others.  That includes, say, having
>> a Debian chroot on some other OS and running binaries from that chroot
>> without going into the chroot.  Often that will absolutely not work.
>> Sometimes it will work, and it's convenient that it will work for some
>> obscure recovery situations or other weird one-off use cases.  I've
>> also copied files from working systems to broken systems running a
>> different distribution before, and there's a list of caveats as long as
>> my arm, but sometimes it's a quick fix for something.

> Vast, vast majority of binaries from existing packages will already
> not work out of the box in that use case though.

I'm not sure why you think this is true and it makes me wonder if maybe my
intuition about cross-distribution compatibility is wrong.  I would expect
to be able to copy, say, most (all?) binaries from coreutils from a Debian
system to some other distribution and run it (and that's exactly the sort
of binary that is useful in this kind of cross-distribution rescue case).
Is this not true today?  What breaks?

Note that we're not talking about complicated packages with lots of
runtime like, say, Emacs.  As I understand it your proposal wouldn't
change PT_INTERP for that binary anyway. 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 14:36, Steve McIntyre  wrote:
>
> Hey Johannes,
>
> On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
> >So did we not years ago decide, that the result of the "cross- and
> >inter-project discussion" is, that everybody is going merged-/usr and that's
> >why we need it too and that's why it is okay to build a system where binaries
> >and scripts built for it just may not run on those other systems that do not 
> >do
> >it?  With merged-/usr we already *did* "change fundamental things around" for
> >reasons that are really not clear to me (but which i do not want to discuss
> >here) and as a result did not "care about interoperability" (with those who 
> >do
> >not also adopt it). In my own Debian work I so far only got extra work 
> >because
> >of merged-/usr and I do not see the benefits (yet) and I was hoping that
> >"changing fundamental things around Linux and (basically) not caring about
> >interoperability" was *not* Debian's attitude but alas here we are.
> >
> >So have we not already burned the bridges to the non-merged-/usr world? Why 
> >was
> >it okay back then to say "we can make this change because all other important
> >players are doing merged-/usr so we can/have to as well". And now in the
> >PT_INTERP discussion somehow we care again about those systems? I thought we
> >already had the "cross- and inter-project discussion" about merged-/usr and
> >because the result was "yes, go for it" we did it too. But if that is the 
> >case,
> >why do we now care for a subset of the interoperability problems caused by
> >merged-/usr for systems that don't have it?
>
> This change is absolutely *not* needed to make merged-/usr work; if
> anybody is claiming that it is, then they are not being 100% honest
> with us. All the other distros doing merged-/usr have done it without
> making this change, and it's also been working OK for us so far
> without this change.

That is absolutely true, it is not mandatory. It is one possible
solution (of many) to a particular use case being sounded out, that's
all. I don't think it was mentioned by anybody as needed, if it was,
happy to clarify.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

Hey Johannes,

On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
wrote:
>Quoting Steve McIntyre (2023-05-15 02:54:02)
>> 
>> Pointing at gentoo or nixos as examples of projects that have decided
>> to break compatibility doesn't cut it, I'm afraid. They're well known
>> for changing fundamental things around Linux and (basically) not caring about
>> interoperability. That attitude is *not* Debian's.
>
>me and Luca have different ideas about how bootstrapping a Debian chroot should
>look like and I don't want to make an argument *for* changing PT_INTERP here as
>I think that keeping compatibility with others by following ABI is a good thing
>and because I think (and hope -- but Helmut is doing that analysis right now)
>that the debootstrap problem can be solved in a way I envision without changing
>PT_INTERP. But what I do not understand about the argument against Luca's
>proposal is:
>
>Obviously, with Luca's proposal, binaries from packages built with a different
>dynamic linker path in them would not work on distributions without merged-/usr
>symlinks. But if the property of stuff from Debian being able to run on
>non-Debian non-merged-/usr systems is an important one, then why was it okay to
>have merged-/usr as the default? Because with merged-/usr we already changed
>the interface contract for a lot of things because now binaries and libraries
>can also be found at other locations than on non-merged-/usr systems. A script
>with a /usr/bin/bash shebang built on and for Debian will not work on a system
>without the symlinks.

Despite the massive upheavals of merged-/usr in *other* ways, it's
actually a *minor* change as far as compatibility is concerned
here. The runtime linker (aka interpreter) is responsible for
resolving symbols and finding needed libraries. So long as *that* bit
works OK, then ~everything else should follow OK. This is how it's
possible to have things work across distros: binaries don't actually
care where the libraries live, or whether they came from rpm or deb
packaging, etc.

The issue at hand here is that the interpreter path is the most basic
thing that matters for this compatibility. Break this and *nothing*
can work.

>So did we not years ago decide, that the result of the "cross- and
>inter-project discussion" is, that everybody is going merged-/usr and that's
>why we need it too and that's why it is okay to build a system where binaries
>and scripts built for it just may not run on those other systems that do not do
>it?  With merged-/usr we already *did* "change fundamental things around" for
>reasons that are really not clear to me (but which i do not want to discuss
>here) and as a result did not "care about interoperability" (with those who do
>not also adopt it). In my own Debian work I so far only got extra work because
>of merged-/usr and I do not see the benefits (yet) and I was hoping that
>"changing fundamental things around Linux and (basically) not caring about
>interoperability" was *not* Debian's attitude but alas here we are.
>
>So have we not already burned the bridges to the non-merged-/usr world? Why was
>it okay back then to say "we can make this change because all other important
>players are doing merged-/usr so we can/have to as well". And now in the
>PT_INTERP discussion somehow we care again about those systems? I thought we
>already had the "cross- and inter-project discussion" about merged-/usr and
>because the result was "yes, go for it" we did it too. But if that is the case,
>why do we now care for a subset of the interoperability problems caused by
>merged-/usr for systems that don't have it?

This change is absolutely *not* needed to make merged-/usr work; if
anybody is claiming that it is, then they are not being 100% honest
with us. All the other distros doing merged-/usr have done it without
making this change, and it's also been working OK for us so far
without this change.

Breaking an agreed interface contract like this is axiomatically
*wrong* and will hurt us and the rest of the Linux ecosystem.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"You can't barbecue lettuce!" -- Ellie Crane

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Steve McIntyre (2023-05-15 02:54:02)
> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> >On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> >
> >> The x86-64 ABI is set. Feel free to make the case to the next
> >> architecture designer that their new ABI should have the dynamic linker
> >> in `/usr/lib`. That would *not* have the same downsides, as long as
> >> everyone agrees on a path.
> >
> >In practice it is not, though. There are other distributions that
> >change PT_INTERP for their own purposes, they've already been listed
> >in this thread. And I am still not hearing any concrete, factual use
> >case that would be impaired by such a change. I'm beginning to
> >seriously think there aren't any? Is that really the case?
> 
> The ABI has been agreed and set down in documentation that *just
> about* everybody has been following since its inception. This includes
> the most basic set of definitions of what an x86-64 program must look
> like, including the interpreter path. If this path is changed, then
> *at the most basic level* we'd be making programs that are not valid
> by the ABI we've agreed to. This is an *external interface contract*,
> not something we should ever consider changing without significant
> cross- and inter-project discussion.
> 
> Pointing at gentoo or nixos as examples of projects that have decided
> to break compatibility doesn't cut it, I'm afraid. They're well known
> for changing fundamental things around Linux and (basically) not caring about
> interoperability. That attitude is *not* Debian's.

me and Luca have different ideas about how bootstrapping a Debian chroot should
look like and I don't want to make an argument *for* changing PT_INTERP here as
I think that keeping compatibility with others by following ABI is a good thing
and because I think (and hope -- but Helmut is doing that analysis right now)
that the debootstrap problem can be solved in a way I envision without changing
PT_INTERP. But what I do not understand about the argument against Luca's
proposal is:

Obviously, with Luca's proposal, binaries from packages built with a different
dynamic linker path in them would not work on distributions without merged-/usr
symlinks. But if the property of stuff from Debian being able to run on
non-Debian non-merged-/usr systems is an important one, then why was it okay to
have merged-/usr as the default? Because with merged-/usr we already changed
the interface contract for a lot of things because now binaries and libraries
can also be found at other locations than on non-merged-/usr systems. A script
with a /usr/bin/bash shebang built on and for Debian will not work on a system
without the symlinks.

So did we not years ago decide, that the result of the "cross- and
inter-project discussion" is, that everybody is going merged-/usr and that's
why we need it too and that's why it is okay to build a system where binaries
and scripts built for it just may not run on those other systems that do not do
it?  With merged-/usr we already *did* "change fundamental things around" for
reasons that are really not clear to me (but which i do not want to discuss
here) and as a result did not "care about interoperability" (with those who do
not also adopt it). In my own Debian work I so far only got extra work because
of merged-/usr and I do not see the benefits (yet) and I was hoping that
"changing fundamental things around Linux and (basically) not caring about
interoperability" was *not* Debian's attitude but alas here we are.

So have we not already burned the bridges to the non-merged-/usr world? Why was
it okay back then to say "we can make this change because all other important
players are doing merged-/usr so we can/have to as well". And now in the
PT_INTERP discussion somehow we care again about those systems? I thought we
already had the "cross- and inter-project discussion" about merged-/usr and
because the result was "yes, go for it" we did it too. But if that is the case,
why do we now care for a subset of the interoperability problems caused by
merged-/usr for systems that don't have it?

As I said, I don't care much about the PT_INTERP value but I don't understand
yet, why this argument about interoperability with non-merged-/usr systems is
working now but it didn't wasn't enough to stop another very fundamental change
in how we build a Linux distro.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > That's self-evidently not true, as there are other distributions where
> > that already happens, it's been already mentioned.
>
> You've mentioned this a couple of times but I don't think I've seen the
> message where the details were explained.  Maybe this was only in your
> message posted to debian-gcc, which wasn't part of this thread?  (It's
> also possible that I just missed it somewhere.)
>
> That message only mentions GUIX, which I don't know very much about, but
> my recollection (maybe wrong?) is that it's a NIX variant that is doing
> special tricks to support immutable package trees and
> roll-forward/roll-back upgrades.  I can see why that might be motivation
> to build incompatible binaries in order to preserve some other invariant
> they're trying for as the point of their distribution (in particular, I
> suspect they're pinning binaries to a specific version of the dynamic
> loader as part of the whole immutable tree strategy).  That's a perfectly
> fine decision in a distribution that's trying to do something very
> different and is a bit of a science experiment, but I don't think that
> describes Debian.
>
> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
> player in Linux distributions, and I'm not sure how much they care about
> compatibility with anyone else.)

This is a counter-example to confute the assertion that *everybody*
does the same thing, which has been made multiple times. I'm not sure
whether it's an experiment or not, I mean if you ask their
users/developers I think they'd tell you it's very much production
ready, but it is largely irrelevant: it exists, and that's the only
reason why it was mentioned, as it shows that it is _possible_ to do
that and be a working distribution. Doesn't imply it's automatically
desirable, but that was not the intention.

> > Besides, we are not talking about sacred religious texts - the point is
> > making things work. If they do, is it _really_
> > non-compliant/incompatible?
>
> I understand your point in making this argument, but please understand
> that this sort of willingness to change things if one didn't think they
> would cause problems didn't work very well, and was part of what led to
> the development of standardized ABIs in the first place.  Those of us who
> have been around longer than Linux have ABIs have a bit of a strong
> reaction here (I think this is also what you're seeing from Steve),
> because we remember the bad old days.  I still have compatibility code
> around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
> because gcc didn't correctly implement the IRIX ABI.
>
> People are very bad at judging whether their new idea would be *really*
> incompatible.  This is why these days everyone tries to follow the ABI
> pretty closely.
>
> And in any case, changing PT_INTERP is trivially and obviously
> incompatible; the binary will simply not run on a system that doesn't have
> that path.  So it's not like we have to carefully judge nuance here.  Your
> argument, so far as I can tell, is basically "but no one will ever want to
> run those binaries on a non-/usr-merged system anyway," which is basically
> conceding the incompatibility point since the ABI doesn't require merged
> /usr.

Not quite: my argument is that binaries from these packages are not
intended and not required to be ran on non-Debian systems, so there's
no incompatibility introduced in the first place - everything still
works where it is supposed to, exactly as it was before.

> There's also some other history here: Debian is not super-happy with the
> PT_INTERP because ideally we'd prefer it use a path compatible with our
> multiarch approach.  I believe we raised that and no one had any interest
> in trying to change anything, so we lived with the limitations that
> creates.  (And I think that was the right decision.)
>
> > Thanks, that's the first actual real example mentioned so far. And it's
> > an interesting one: taking a $random Debian package and using it on a
> > completely different, non-Debian system. Is that a supported use case?
> > If so, does that mean that I can go ahead and raise a Severity: serious
> > bug on any package that doesn't work in such a way?
>
> I feel like you're distorting my argument here to try to make some sort of
> slippery slope argument, and it's coming across as possibly more
> aggressive than you had intended.

No aggression intended whatsoever, sorry if it appeared that way. I am
trying to understand what the rules are.

> The world does not divide neatly into supported and unsupported use cases.
> There are a lot of things I do to computers 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:

> That's self-evidently not true, as there are other distributions where
> that already happens, it's been already mentioned.

You've mentioned this a couple of times but I don't think I've seen the
message where the details were explained.  Maybe this was only in your
message posted to debian-gcc, which wasn't part of this thread?  (It's
also possible that I just missed it somewhere.)

That message only mentions GUIX, which I don't know very much about, but
my recollection (maybe wrong?) is that it's a NIX variant that is doing
special tricks to support immutable package trees and
roll-forward/roll-back upgrades.  I can see why that might be motivation
to build incompatible binaries in order to preserve some other invariant
they're trying for as the point of their distribution (in particular, I
suspect they're pinning binaries to a specific version of the dynamic
loader as part of the whole immutable tree strategy).  That's a perfectly
fine decision in a distribution that's trying to do something very
different and is a bit of a science experiment, but I don't think that
describes Debian.

(Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
player in Linux distributions, and I'm not sure how much they care about
compatibility with anyone else.)

> Besides, we are not talking about sacred religious texts - the point is
> making things work. If they do, is it _really_
> non-compliant/incompatible?

I understand your point in making this argument, but please understand
that this sort of willingness to change things if one didn't think they
would cause problems didn't work very well, and was part of what led to
the development of standardized ABIs in the first place.  Those of us who
have been around longer than Linux have ABIs have a bit of a strong
reaction here (I think this is also what you're seeing from Steve),
because we remember the bad old days.  I still have compatibility code
around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
because gcc didn't correctly implement the IRIX ABI.

People are very bad at judging whether their new idea would be *really*
incompatible.  This is why these days everyone tries to follow the ABI
pretty closely.

And in any case, changing PT_INTERP is trivially and obviously
incompatible; the binary will simply not run on a system that doesn't have
that path.  So it's not like we have to carefully judge nuance here.  Your
argument, so far as I can tell, is basically "but no one will ever want to
run those binaries on a non-/usr-merged system anyway," which is basically
conceding the incompatibility point since the ABI doesn't require merged
/usr.

There's also some other history here: Debian is not super-happy with the
PT_INTERP because ideally we'd prefer it use a path compatible with our
multiarch approach.  I believe we raised that and no one had any interest
in trying to change anything, so we lived with the limitations that
creates.  (And I think that was the right decision.)

> Thanks, that's the first actual real example mentioned so far. And it's
> an interesting one: taking a $random Debian package and using it on a
> completely different, non-Debian system. Is that a supported use case?
> If so, does that mean that I can go ahead and raise a Severity: serious
> bug on any package that doesn't work in such a way?

I feel like you're distorting my argument here to try to make some sort of
slippery slope argument, and it's coming across as possibly more
aggressive than you had intended.

The world does not divide neatly into supported and unsupported use cases.
There are a lot of things I do to computers that I expect to work in some
situations but not in others.  That includes, say, having a Debian chroot
on some other OS and running binaries from that chroot without going into
the chroot.  Often that will absolutely not work.  Sometimes it will work,
and it's convenient that it will work for some obscure recovery situations
or other weird one-off use cases.  I've also copied files from working
systems to broken systems running a different distribution before, and
there's a list of caveats as long as my arm, but sometimes it's a quick
fix for something.

But mostly my reaction is because breaking the ABI is a Really Big Deal.
Constructing the Linux ABI and getting the details actually published was
a hard-fought, arduous endeavor.  I doubt anyone enjoyed it; it's the sort
of annoying compatibility work that provides tons of small, subtle
benefits and takes a great deal of truly thankless work, and people often
don't realize all the tiny ways that it has made the world a better place,
or the range of weird compatibility problems that can arise from messing
with it.  Diverging from it is not something to do lightly, precisely
*because* it's often extremely difficult to understand what the effects
could be or what might break.

While I appreciate how it would

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

I'm *trying* to assume good faith here, but I'm running out of energy
to do so.

On Mon, May 15, 2023 at 01:42:27AM +0100, Luca Boccassi wrote:
>On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
>> Incidentally, that remains true even if we only do that in distribution
>> packages.  I certainly have copied binaries from a Debian package to other
>> Linux systems before for various reasons and expected them to run.  Sure,
>> this might not work for other reasons outside of our control, but that's
>> no reason to be gratuitously incompatible by breaking the ABI,
>> particularly for what seem to be annoyances of our own creation with known
>> workarounds.
>
>Thanks, that's the first actual real example mentioned so far. And
>it's an interesting one: taking a $random Debian package and using it
>on a completely different, non-Debian system. Is that a supported use
>case? If so, does that mean that I can go ahead and raise a Severity:
>serious bug on any package that doesn't work in such a way?

Russ has described copying *binaries* out of packages and running them
elsewhere. I've done that too, from time to time. This is one of the
things made possible by the ABI contract being followed.

You are the one proposing to break that contract, thereby
*guaranteeing* this will fail on systems where otherwise it could
work. I think the onus is on *you* to justify why this is a valid and
useful thing to do. Your apparent lack of care for agreed standards
here is horrifying.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
>On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
>
>> The x86-64 ABI is set. Feel free to make the case to the next
>> architecture designer that their new ABI should have the dynamic linker
>> in `/usr/lib`. That would *not* have the same downsides, as long as
>> everyone agrees on a path.
>
>In practice it is not, though. There are other distributions that
>change PT_INTERP for their own purposes, they've already been listed
>in this thread. And I am still not hearing any concrete, factual use
>case that would be impaired by such a change. I'm beginning to
>seriously think there aren't any? Is that really the case?

The ABI has been agreed and set down in documentation that *just
about* everybody has been following since its inception. This includes
the most basic set of definitions of what an x86-64 program must look
like, including the interpreter path. If this path is changed, then
*at the most basic level* we'd be making programs that are not valid
by the ABI we've agreed to. This is an *external interface contract*,
not something we should ever consider changing without significant
cross- and inter-project discussion.

Pointing at gentoo or nixos as examples of projects that have decided
to break compatibility doesn't cut it, I'm afraid. They're well known
for changing fundamental things around Linux and (basically) not
caring about interoperability. That attitude is *not* Debian's.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"The problem with defending the purity of the English language is that
 English is about as pure as a cribhouse whore. We don't just borrow words; on
 occasion, English has pursued other languages down alleyways to beat them
 unconscious and rifle their pockets for new vocabulary."  -- James D. Nicoll

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> My understanding is that this specific thread is about a mention that we
> may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
> similar path.
>
> If PT_INTERP points to a file that doesn't exist, the program is obviously
> not going to run.  The Linux x86_64 ABI says it must point to
> /lib64/ld-linux-x86-64.so.2.  If we build binaries that use some other
> value, then we are not building ABI-compliant binaries and they may not
> run on other systems.  This is the whole point of an ABI.

This is not about locally compiled software or such, only packages
(and maybe even just a subset of them).

> An obvious specific example of such a system would be one that didn't
> merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> path, but that's just one obvious example.  There may be others; the whole
> point of an ABI is that you do not change things like this, not even if
> you can't personally imagine why your change wouldn't be harmful.  There's
> a whole process for changing an ABI that involves everyone else agreeing
> as well, and unless one goes through that process, the ABI is what it is.
> Debian not building ABI-compliant binaries would be highly surprising.

That's self-evidently not true, as there are other distributions where
that already happens, it's been already mentioned. Besides, we are not
talking about sacred religious texts - the point is making things
work. If they do, is it _really_ non-compliant/incompatible?

> Incidentally, that remains true even if we only do that in distribution
> packages.  I certainly have copied binaries from a Debian package to other
> Linux systems before for various reasons and expected them to run.  Sure,
> this might not work for other reasons outside of our control, but that's
> no reason to be gratuitously incompatible by breaking the ABI,
> particularly for what seem to be annoyances of our own creation with known
> workarounds.

Thanks, that's the first actual real example mentioned so far. And
it's an interesting one: taking a $random Debian package and using it
on a completely different, non-Debian system. Is that a supported use
case? If so, does that mean that I can go ahead and raise a Severity:
serious bug on any package that doesn't work in such a way?

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 01:07, Peter Pentchev  wrote:
>
> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> > On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> > >
> > > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > > The loader is still available via the old path, so external/third
> > > > party/local/other software works unchanged. This should negatively
> > > > only affect our 1st party packages, when running on a non-merged
> > > > distro.
> > > > And are _all_ our packages really 100% compatible with other distros
> > > > at all? Are they even supposed to be?
> > >
> > > People build things on Debian that are not Debian packages. People
> > > compile binaries on Debian, and expect them to work on any system that
> > > has sufficiently new libraries.
> > >
> > > This is *not* about Debian packages failing to work on other
> > > distributions; this is about *software compiled on Debian* faliing to
> > > work in other environments.
> >
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> If an ELF executable, compiled on Debian, records its interpreter as
> /usr/lib/ld-linux.so.2, what happens when one tries to run it on
> a non-usr-merged system? Even one with a recent enough glibc version?

This is not about locally built ELF executables, no difference in those.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:

> Why would "software compiled on Debian" fail to work in other
> environments? Well, there are many reasons actually, people invented
> containers/flatpaks/snaps exactly for that reason. But nothing do with
> anything discussed here though, as far as I can tell?

My understanding is that this specific thread is about a mention that we
may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
similar path.

If PT_INTERP points to a file that doesn't exist, the program is obviously
not going to run.  The Linux x86_64 ABI says it must point to
/lib64/ld-linux-x86-64.so.2.  If we build binaries that use some other
value, then we are not building ABI-compliant binaries and they may not
run on other systems.  This is the whole point of an ABI.

An obvious specific example of such a system would be one that didn't
merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
path, but that's just one obvious example.  There may be others; the whole
point of an ABI is that you do not change things like this, not even if
you can't personally imagine why your change wouldn't be harmful.  There's
a whole process for changing an ABI that involves everyone else agreeing
as well, and unless one goes through that process, the ABI is what it is.
Debian not building ABI-compliant binaries would be highly surprising.

Incidentally, that remains true even if we only do that in distribution
packages.  I certainly have copied binaries from a Debian package to other
Linux systems before for various reasons and expected them to run.  Sure,
this might not work for other reasons outside of our control, but that's
no reason to be gratuitously incompatible by breaking the ABI,
particularly for what seem to be annoyances of our own creation with known
workarounds.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Peter Pentchev]

On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> >
> > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > The loader is still available via the old path, so external/third
> > > party/local/other software works unchanged. This should negatively
> > > only affect our 1st party packages, when running on a non-merged
> > > distro.
> > > And are _all_ our packages really 100% compatible with other distros
> > > at all? Are they even supposed to be?
> >
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
> >
> > This is *not* about Debian packages failing to work on other
> > distributions; this is about *software compiled on Debian* faliing to
> > work in other environments.
> 
> Why would "software compiled on Debian" fail to work in other
> environments? Well, there are many reasons actually, people invented
> containers/flatpaks/snaps exactly for that reason. But nothing do with
> anything discussed here though, as far as I can tell?

If an ELF executable, compiled on Debian, records its interpreter as
/usr/lib/ld-linux.so.2, what happens when one tries to run it on
a non-usr-merged system? Even one with a recent enough glibc version?

G'luck,
Peter
 
-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > The loader is still available via the old path, so external/third
> > party/local/other software works unchanged. This should negatively
> > only affect our 1st party packages, when running on a non-merged
> > distro.
> > And are _all_ our packages really 100% compatible with other distros
> > at all? Are they even supposed to be?
>
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.
>
> This is *not* about Debian packages failing to work on other
> distributions; this is about *software compiled on Debian* faliing to
> work in other environments.

Why would "software compiled on Debian" fail to work in other
environments? Well, there are many reasons actually, people invented
containers/flatpaks/snaps exactly for that reason. But nothing do with
anything discussed here though, as far as I can tell?

> If you build a dynamically linked binary that only depends on glibc, you
> can expect it to be reasonably portable, to any system that uses glibc
> and has a sufficiently new version.
>
> Debian stable is, in fact, one of the common environments people use to
> compile binaries for distribution.

"sufficiently new version" is doing a lot of work there. We have
shlibs dependencies for a reason. In fact, the most common environment
used to distribute binaries is the EOL Ubuntu 16.04, slowly switching
to the soon-to-be-EOL 18.04. glibc is not forward-compatible, new
symbols are added all the time and are used all the time, and you
don't jump on a brand new distribution to do that kind of work, that
would be self-defeating.

> > So, what I am asking is, what actual, real difference does it make if,
> > by default (and with an override available for example), packages
> > built on Debian for Debian record the ld path to point to its (actual)
> > location on Debian, via say a compiler spec file that is injected in a
> > deb build?
>
> Making binaries built *on* Debian different than binaries built *for*
> Debian would introduce a needless additional source of complexity,
> compared to just compiling code the same way in both cases.

That's not how it works today already. There are several significant
differences between just running "gcc sources.c" and building a
package via debhelper on a buildd, they are not the same thing at all,
and they haven't been since forever, there are dozens of
compiler/linker options that the Debian package build environment
sets. Or will you now also ask the distribution to rollback multiarch,
hardening, SOURCE_DATE_EPOCH, -ffile-prefix-map and all the other
reproducibility options, and so on? These and many more are all
"needless additional sources of complexity, compared to just compiling
code the same way" too. Because guess what, there are people who
couldn't possibly care less about
multiarch/security/reproducibility/etc, and there will also be a
subset of users who considers a subset of those compiler options
"needless". So are you going to push to have all of that reverted? And
also are you going to propose a Policy change that forbids adding any
new compiler/linker option to the package build process?

> To frame this in different terms: consider that one of the major goals
> of systemd has been to harmonize across distributions and eliminate
> needless variations that don't serve much actual purpose (e.g.
> variations in config file paths for the same config file). Consider how
> much effort systemd went to work with distributions, understand and deal
> with the *important* variations, and try to convince them to abandon the
> *unimportant* variations. Now imagine if someone came along and said
> "let's patch systemd to put unit files in /purple/; it'll work with
> everything in our distribution".

Pretty sure the Nix folks are already doing pretty much that. And if
it works for their case, all the power to them.

> Or, imagine if someone said "let's inject an argument to gzip, only for
> building the .gz files sihpped in our packages of course, to modify the
> gzip header and remove a few of the extraneous additional fields; it'll
> be fine, because we've patched our gzip to parse it"

Not really related, archives are _intended_ to be opened anywhere for
any reason. Do you have any actual related use case that would no
longer work? Because that would be the easiest and most convincing
counter-factual that could be provided.

> The x86-64 ABI is set. Feel free to make the case to the next
> architecture designer that their new ABI should have the dynamic linker
> in `/usr/lib`. That would *not* have the same downsides, as long as
> everyone agrees on a path.

In practice it is not, though. There are other distributions that
change PT_INTERP for their own purposes, they've already been listed
in this thread. And I am still not 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Josh Triplett]

On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> The loader is still available via the old path, so external/third
> party/local/other software works unchanged. This should negatively
> only affect our 1st party packages, when running on a non-merged
> distro.
> And are _all_ our packages really 100% compatible with other distros
> at all? Are they even supposed to be?

People build things on Debian that are not Debian packages. People
compile binaries on Debian, and expect them to work on any system that
has sufficiently new libraries.

This is *not* about Debian packages failing to work on other
distributions; this is about *software compiled on Debian* faliing to
work in other environments.

If you build a dynamically linked binary that only depends on glibc, you
can expect it to be reasonably portable, to any system that uses glibc
and has a sufficiently new version.

Debian stable is, in fact, one of the common environments people use to
compile binaries for distribution.

> So, what I am asking is, what actual, real difference does it make if,
> by default (and with an override available for example), packages
> built on Debian for Debian record the ld path to point to its (actual)
> location on Debian, via say a compiler spec file that is injected in a
> deb build?

Making binaries built *on* Debian different than binaries built *for*
Debian would introduce a needless additional source of complexity,
compared to just compiling code the same way in both cases.

To frame this in different terms: consider that one of the major goals
of systemd has been to harmonize across distributions and eliminate
needless variations that don't serve much actual purpose (e.g.
variations in config file paths for the same config file). Consider how
much effort systemd went to work with distributions, understand and deal
with the *important* variations, and try to convince them to abandon the
*unimportant* variations. Now imagine if someone came along and said
"let's patch systemd to put unit files in /purple/; it'll work with
everything in our distribution".

Or, imagine if someone said "let's inject an argument to gzip, only for
building the .gz files sihpped in our packages of course, to modify the
gzip header and remove a few of the extraneous additional fields; it'll
be fine, because we've patched our gzip to parse it"

The x86-64 ABI is set. Feel free to make the case to the next
architecture designer that their new ABI should have the dynamic linker
in `/usr/lib`. That would *not* have the same downsides, as long as
everyone agrees on a path.

- Josh Triplett

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Andreas Metzler]

On 2023-05-12 Ansgar  wrote:
[...]
> The core issue as I see it is as follows:
[...]
> Do you think this summary of the issue is right?

I think Simon's reading of the situation as posted in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035904#30
makes a lot of sense.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 15:30, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
> >>
> >> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
> >> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >> >>>
> >> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >>> >
> >> >>> >The core issue as I see it is as follows:
> >> >>> >
> >> >>> >- Debian has decided to support only merged-/usr, including possibly
> >> >>> >  moving /bin/sh to /usr/bin/sh or using 
> >> >>> > /usr/lib*/ld-linux-x86-64.so.2
> >> >>> >  as the interpreter in binaries.
> >> >>>
> >> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> >>> seen. The interpreter must *not* be changed willy-nilly.
> >> >>
> >> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >> >>seemingly crazy options, as in, "what would _actually_ explode if we
> >> >>do this or do that?", on this very d-devel thread. I posted a longer
> >> >>version here some days ago:
> >> >>
> >> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
> >> >
> >> >Oh holy fuck.
> >> >
> >> >You're talking about changing ABI by doing this. That *is* utterly
> >> >crazy. No.
> >>
> >> People have asked me to expand on this further...
> >>
> >> I've been involved in defining ABI before, specifically for
> >> armhf. It's not a quick and easy process. It needs buy-in from all
> >> sides to make things work, and people *really* value the
> >> interoperability that it enables.
> >>
> >> The interpreter path is one of the most important parts of the ABI
> >> spec, the bit that makes binaries compatible between all the various
> >> stakeholders: compiler/tools people, distros, software vendors,
> >> etc. Lots of the rest of the details downstream of this can be
> >> changed, and people do this all the time - compare multilib to
> >> multi-arch for example. That all works fine *so long as* the runtime
> >> linker can be located and started OK.
> >
> >The loader is still available via the old path, so external/third
> >party/local/other software works unchanged. This should negatively
> >only affect our 1st party packages, when running on a non-merged
> >distro.
>
> So why the hell do you want to break this in the first place? Does a
> symlink in the "wrong" place offend you for some reason? For that you
> want to change a core assumption in *every single binary* in Debian?
> Believe me, I've been here in the past when we made changes in armhf
> to accommodate earlier mistakes. That was just for one
> architecture. What possible benefit do you see in this change?

As it was mentioned on the list, because it makes bootstrapping
self-contained, that's a real and concrete benefit that some
developers like Helmut care greatly about, and that's why we are
talking about it. To me, it sounds very attractive to have a
self-contained and canonicalized distro-wide configuration. If the
canonical location where certain files are stored in /usr/bin or
/usr/lib, it seems sensible to me to configure Debian software to look
for it where we actually put it, while maintaining compatibility for
external/local software so that it keeps working. And it is also
unclear so far what would outright break - the externally defined ABI
in terms of where the loader can be accessed at, would still be
respected. Hence why questions are being asked. Nobody's being forced
to do anything, this is just a discussion.

> >And are _all_ our packages really 100% compatible with other distros
> >at all? Are they even supposed to be?
> >
> >For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
> >machine, when I try to run it, it fails:
> >
> >root@focal:/tmp# wget
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >--2023-05-12 12:46:17--
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
> >Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... 
> >connected.
> >HTTP request sent, awaiting 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Holger Levsen]

On Fri, May 12, 2023 at 03:29:29PM +0100, Steve McIntyre wrote:
> >> >Oh holy fuck.
> So why the hell do you want to break this in the first place? 
> You're wilfully missing the point, and you know it.
> I have better things to do than argue about this. I refuse to engage
> with this right now. You're talking about breaking things for *no*
> discernible benefit that I've seen any discussion about.
 
language please. and also assume good faith.

thanks.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Das Leben ist schön. Von 'einfach' war nie die Rede. (@lernzyklus)


signature.asc
Description: PGP signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
>On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
>>
>> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
>> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>> >>>
>> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>> >>> >
>> >>> >The core issue as I see it is as follows:
>> >>> >
>> >>> >- Debian has decided to support only merged-/usr, including possibly
>> >>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>> >>> >  as the interpreter in binaries.
>> >>>
>> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>> >>> seen. The interpreter must *not* be changed willy-nilly.
>> >>
>> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>> >>seemingly crazy options, as in, "what would _actually_ explode if we
>> >>do this or do that?", on this very d-devel thread. I posted a longer
>> >>version here some days ago:
>> >>
>> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>> >
>> >Oh holy fuck.
>> >
>> >You're talking about changing ABI by doing this. That *is* utterly
>> >crazy. No.
>>
>> People have asked me to expand on this further...
>>
>> I've been involved in defining ABI before, specifically for
>> armhf. It's not a quick and easy process. It needs buy-in from all
>> sides to make things work, and people *really* value the
>> interoperability that it enables.
>>
>> The interpreter path is one of the most important parts of the ABI
>> spec, the bit that makes binaries compatible between all the various
>> stakeholders: compiler/tools people, distros, software vendors,
>> etc. Lots of the rest of the details downstream of this can be
>> changed, and people do this all the time - compare multilib to
>> multi-arch for example. That all works fine *so long as* the runtime
>> linker can be located and started OK.
>
>The loader is still available via the old path, so external/third
>party/local/other software works unchanged. This should negatively
>only affect our 1st party packages, when running on a non-merged
>distro.

So why the hell do you want to break this in the first place? Does a
symlink in the "wrong" place offend you for some reason? For that you
want to change a core assumption in *every single binary* in Debian?
Believe me, I've been here in the past when we made changes in armhf
to accommodate earlier mistakes. That was just for one
architecture. What possible benefit do you see in this change?

>And are _all_ our packages really 100% compatible with other distros
>at all? Are they even supposed to be?
>
>For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
>machine, when I try to run it, it fails:
>
>root@focal:/tmp# wget
>http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
>--2023-05-12 12:46:17--
>http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
>Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
>Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... 
>connected.
>HTTP request sent, awaiting response... 200 OK
>Length: 27572 (27K) [application/vnd.debian.binary-package]
>Saving to: 'efibootmgr_17-2_amd64.deb'
>
>efibootmgr_17-2_amd64.deb
>100%[===>]  26.93K
>--.-KB/sin 0.04s
>
>2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved 
>[27572/27572]
>
>root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
>root@focal:/tmp# ./ebm/bin/efibootmgr
>./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
>`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)
>
>Should I file a severity: serious bug against efibootmgr because it is
>not interoperable?

You're wilfully missing the point, and you know it.

>The answer is obviously not, because it would be absurd to expect a
>binary compiled against libraries from one distro to "just work" on an
>entirely different distro. Glibc itself is not forward compatible, and
>is allowed to add new symbols, that are not present in older versions,
>and packages are allowed to depend on them. Aren't those also ABI
>breakages? What about all the libraries that bump soname? What about
>binaries that rely on newer kernel interfaces, or IPC interfaces?
>
&

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >>>
> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >>> >
> >>> >The core issue as I see it is as follows:
> >>> >
> >>> >- Debian has decided to support only merged-/usr, including possibly
> >>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >>> >  as the interpreter in binaries.
> >>>
> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >>> seen. The interpreter must *not* be changed willy-nilly.
> >>
> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >>seemingly crazy options, as in, "what would _actually_ explode if we
> >>do this or do that?", on this very d-devel thread. I posted a longer
> >>version here some days ago:
> >>
> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
> >
> >Oh holy fuck.
> >
> >You're talking about changing ABI by doing this. That *is* utterly
> >crazy. No.
>
> People have asked me to expand on this further...
>
> I've been involved in defining ABI before, specifically for
> armhf. It's not a quick and easy process. It needs buy-in from all
> sides to make things work, and people *really* value the
> interoperability that it enables.
>
> The interpreter path is one of the most important parts of the ABI
> spec, the bit that makes binaries compatible between all the various
> stakeholders: compiler/tools people, distros, software vendors,
> etc. Lots of the rest of the details downstream of this can be
> changed, and people do this all the time - compare multilib to
> multi-arch for example. That all works fine *so long as* the runtime
> linker can be located and started OK.

The loader is still available via the old path, so external/third
party/local/other software works unchanged. This should negatively
only affect our 1st party packages, when running on a non-merged
distro.
And are _all_ our packages really 100% compatible with other distros
at all? Are they even supposed to be?

For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
machine, when I try to run it, it fails:

root@focal:/tmp# wget
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
--2023-05-12 12:46:17--
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27572 (27K) [application/vnd.debian.binary-package]
Saving to: 'efibootmgr_17-2_amd64.deb'

efibootmgr_17-2_amd64.deb
100%[===>]  26.93K
--.-KB/sin 0.04s

2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved [27572/27572]

root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
root@focal:/tmp# ./ebm/bin/efibootmgr
./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)

Should I file a severity: serious bug against efibootmgr because it is
not interoperable?

The answer is obviously not, because it would be absurd to expect a
binary compiled against libraries from one distro to "just work" on an
entirely different distro. Glibc itself is not forward compatible, and
is allowed to add new symbols, that are not present in older versions,
and packages are allowed to depend on them. Aren't those also ABI
breakages? What about all the libraries that bump soname? What about
binaries that rely on newer kernel interfaces, or IPC interfaces?

So, what I am asking is, what actual, real difference does it make if,
by default (and with an override available for example), packages
built on Debian for Debian record the ld path to point to its (actual)
location on Debian, via say a compiler spec file that is injected in a
deb build?
There very likely is some real difference and impact, and I am
genuinely and honestly asking what it could be. If nothing else, it's
an interesting topic, even if likely nothing comes out of it.

> Changing the interpreter path would mean moving to a Debian-specific
> ABI, breaking that compatibility. Hand-waving that away with (and I
> quote):
>
>   "The vast majority of distros today ship the loader in /usr/lib as
>   /lib is just a symlink, so it would be interoperable."
>
> is appalling arrogance. No. You do *not* get to break ABI with that
>

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
>On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>>>
>>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>>> >
>>> >The core issue as I see it is as follows:
>>> >
>>> >- Debian has decided to support only merged-/usr, including possibly
>>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>>> >  as the interpreter in binaries.
>>>
>>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>>> seen. The interpreter must *not* be changed willy-nilly.
>>
>>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>>seemingly crazy options, as in, "what would _actually_ explode if we
>>do this or do that?", on this very d-devel thread. I posted a longer
>>version here some days ago:
>>
>>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
>Oh holy fuck.
>
>You're talking about changing ABI by doing this. That *is* utterly
>crazy. No.

People have asked me to expand on this further...

I've been involved in defining ABI before, specifically for
armhf. It's not a quick and easy process. It needs buy-in from all
sides to make things work, and people *really* value the
interoperability that it enables.

The interpreter path is one of the most important parts of the ABI
spec, the bit that makes binaries compatible between all the various
stakeholders: compiler/tools people, distros, software vendors,
etc. Lots of the rest of the details downstream of this can be
changed, and people do this all the time - compare multilib to
multi-arch for example. That all works fine *so long as* the runtime
linker can be located and started OK.

Changing the interpreter path would mean moving to a Debian-specific
ABI, breaking that compatibility. Hand-waving that away with (and I
quote):

  "The vast majority of distros today ship the loader in /usr/lib as
  /lib is just a symlink, so it would be interoperable."

is appalling arrogance. No. You do *not* get to break ABI with that
argument. The point of the ABI spec is that *everybody* follows
it. You don't change it just because you think it'll make your life a
little easier when bootstrapping a system.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Welcome my son, welcome to the machine.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 11:40, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >>
> >> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >
> >> >The core issue as I see it is as follows:
> >> >
> >> >- Debian has decided to support only merged-/usr, including possibly
> >> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >> >  as the interpreter in binaries.
> >>
> >> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> seen. The interpreter must *not* be changed willy-nilly.
> >
> >Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >seemingly crazy options, as in, "what would _actually_ explode if we
> >do this or do that?", on this very d-devel thread. I posted a longer
> >version here some days ago:
> >
> >https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
> Oh holy fuck.
>
> You're talking about changing ABI by doing this. That *is* utterly
> crazy. No.

It's a thought experiment on a mailing list. If we can't even have
those anymore, something went very wrong somewhere.

You seem to be aware of things that wouldn't work anymore (I think?).
If you have a couple of minutes to spare, may I please ask you to
reply to that thread with such examples? I am genuinely interested in
understanding and talking about it. Thank you.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>>
>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>> >
>> >The core issue as I see it is as follows:
>> >
>> >- Debian has decided to support only merged-/usr, including possibly
>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>> >  as the interpreter in binaries.
>>
>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>> seen. The interpreter must *not* be changed willy-nilly.
>
>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>seemingly crazy options, as in, "what would _actually_ explode if we
>do this or do that?", on this very d-devel thread. I posted a longer
>version here some days ago:
>
>https://lists.debian.org/debian-gcc/2023/05/msg00030.html

Oh holy fuck.

You're talking about changing ABI by doing this. That *is* utterly
crazy. No.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"...In the UNIX world, people tend to interpret `non-technical user'
 as meaning someone who's only ever written one device driver." -- Daniel Pead

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >
> >The core issue as I see it is as follows:
> >
> >- Debian has decided to support only merged-/usr, including possibly
> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >  as the interpreter in binaries.
>
> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> seen. The interpreter must *not* be changed willy-nilly.

Nothing's happening 'willy-nilly'. We are discussing a bunch of
seemingly crazy options, as in, "what would _actually_ explode if we
do this or do that?", on this very d-devel thread. I posted a longer
version here some days ago:

https://lists.debian.org/debian-gcc/2023/05/msg00030.html

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>
>The core issue as I see it is as follows:
>
>- Debian has decided to support only merged-/usr, including possibly
>  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>  as the interpreter in binaries.

WTF? *Nobody* has been talking about breaking ABI like this, that I've
seen. The interpreter must *not* be changed willy-nilly.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"I've only once written 'SQL is my bitch' in a comment. But that code 
 is in use on a military site..." -- Simon Booth

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Ansgar]

On Wed, 2023-05-10 at 19:01 -0700, Sean Whitton wrote:
> On Wed 10 May 2023 at 11:47PM +02, Ansgar wrote:
> > Cool, then let's ask tech-ctte.
> > 
> > Dear ctte, please consider overruling the dpkg maintainer to
> > include
> > the patch from #994388[1].
> > 
> > Thanks,
> > Ansgar
> > 
> >   [1]: https://bugs.debian.org/994388#397
> 
> This would require a new, maintainer-overruling vote.
> Our existing decisions do not apply, so far as I can tell.

Yes, I agree.

> I have written a separate message to the bug and to debian-dpkg with
> a proposal to avoid having to have such a vote.

That seems to be about an implementation detail on how to apply the
patch. I don't think that is the core of the issue?

The core issue as I see it is as follows:

- Debian has decided to support only merged-/usr, including possibly
  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
  as the interpreter in binaries.

- This change breaks on non-merged-/usr systems, including derivatives
  that do not revert *all* relevant changes. (Do you know one that
  does this or plans to do so?)

- dpkg recommends derivative users to move to non-merged-/usr.

I think this contradiction is not good and the core conflict. For me a
distribution should have some coherence. It is not just a distribution
of unrelated parts (like linux, libc, dpkg, dash, ...), but also
integrates them to work together.

And this also means not one package telling users to do X which breaks
other packages. Or (if other packages would do similar things as dpkg)
one package asking users to do X and the other asking users to do the
opposite of X. Just imagine dpkg asking users to move to split-/usr and
then another package starting to warn users to move back to merged-
/usr. Would that be a good state? I think not which is why this bug
exists.

Do you think this summary of the issue is right?

Is there some consensus about how this issue should be solved or do we
need a longer discussion to explore the solution space?

Ansgar

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Sean Whitton]

Hello,

On Wed 10 May 2023 at 11:47PM +02, Ansgar wrote:

> Cool, then let's ask tech-ctte.
>
> Dear ctte, please consider overruling the dpkg maintainer to include
> the patch from #994388[1].
>
> Thanks,
> Ansgar
>
>   [1]: https://bugs.debian.org/994388#397

This would require a new, maintainer-overruling vote.
Our existing decisions do not apply, so far as I can tell.

I have written a separate message to the bug and to debian-dpkg with a
proposal to avoid having to have such a vote.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035904: dpkg currently warning about merged-usr systems (revisited) (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Ansgar]

Package: tech-ctte
X-Debbugs-Cc: Russ Allbery , Sean Whitton 
, Helmut Grohne , Luca Boccassi 
, debian-d...@lists.debian.org, debian-devel@lists.debian.org

On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> Ansgar  writes:
> > Debian going out of its way to tell derivative users to switch back from
> > merged-/usr to split-/usr is the *opposite* of trying to make things as
> > smooth for them as possible.
> 
> Yes, I agree with that part and I think I objected to that at the time.
> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> we don't care about derivatives or their users.  I think we made a mistake
> there which is not in alignment with our ideals or our goals.  We should
> try to reverse that mistake, not double down on it.

Cool, then let's ask tech-ctte.

Dear ctte, please consider overruling the dpkg maintainer to include
the patch from #994388[1].

Thanks,
Ansgar

  [1]: https://bugs.debian.org/994388#397

Re: What to do with merged /usr and dpkg-fsys-usrunmess

[John Darrah]

On Wed, 2022-04-06 at 03:04 -0700, Krzysztof Sobiecki wrote:
> Hi,
> I saw warning and it advised me to use dpkg-fsys-usrunmess:
> But now I see:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008486

>From the link above:

"
Hi,

the `dpkg-fsys-usrunmess` program should include a warning that it
will convert the system into a state that will no longer be supported
in the future by Debian.

Ansgar
"

Here's hoping I don't have to reinstall soon.

(Running up-to-date 'testing')

-- john

Re: What to do with merged /usr and dpkg-fsys-usrunmess

[Bjørn Mork]

Andrey Rahmatullin  writes:
> On Wed, Apr 06, 2022 at 04:02:23PM +0200, Bjørn Mork wrote:
>> > Sorry, blame the dpkg maintainer.
>> 
>> Is that how we discuss technical issues around here?
> This is not a technical issue.

Ah, sorry.  I mistook this for the "Discussion about technical
development topics" list.  My fault


Bjørn

Re: What to do with merged /usr and dpkg-fsys-usrunmess

[Andrey Rahmatullin]

On Wed, Apr 06, 2022 at 04:02:23PM +0200, Bjørn Mork wrote:
> > Sorry, blame the dpkg maintainer.
> 
> Is that how we discuss technical issues around here?
This is not a technical issue.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do with merged /usr and dpkg-fsys-usrunmess

[Bjørn Mork]

Marco d'Itri  writes:

> Sorry, blame the dpkg maintainer.

Is that how we discuss technical issues around here?


Bjørn

Re: What to do with merged /usr and dpkg-fsys-usrunmess

[Marco d'Itri]

On Apr 06, Krzysztof Sobiecki  wrote:

> Now what I'm supposed to do? Wait for dpkg-fsys-usrunmess-unmess?
Install the usrmerge package and hopefully everything will be fixed.

> Is my system broken because I have used dpkg-fsys-usrunmess(that dpkg
> advised me to do?).
Maybe.

> As a Debian user I'm confused and a bit angry.
And you have every right to be. Sorry, blame the dpkg maintainer.

> What is way forward?
Merged /usr.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: What to do with merged /usr and dpkg-fsys-usrunmess

[Adam Borowski]

On Wed, Apr 06, 2022 at 12:03:55PM +0200, Krzysztof Sobiecki wrote:
> Hi,
> I saw warning and it advised me to use dpkg-fsys-usrunmess:
> But now I see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008486
> And even without that there are problems with that tool:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991218
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991192
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008316
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008478
> 
> Now what I'm supposed to do? Wait for dpkg-fsys-usrunmess-unmess?
> Is my system broken because I have used dpkg-fsys-usrunmess(that dpkg
> advised me to do?).
> As a Debian user I'm confused and a bit angry.

For an user: these boats are currently unstable, it's best to sit tight --
they topple only if you move around, especially when jumping between them.

> What is way forward?

That's the subject of a series of flamewars second in length and intensity
only to the systemd debacle.  You don't want to go there.

The usrmerged state, while wished for by some, is currently pretty broken
unless you refrain from a number of actions that are natural for packagers
(inside and outside Debian) and users to do, this is considered unacceptable
by the dpkg maintainer and eg. me.

There is a proposal for a constructive patch set to dpkg that _may_ resolve
this without nasty hacks everywhere; the patches themselves haven't been
AFAIK implemented yet but there is much hope.

If that fails, the drama will resume, with the big question being "WHY?".


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Eight legs good, four legs bad! -- when your drider pwns a
⣾⠁⢠⠒⠀⣿⡁ smelly goodie centaur.
⢿⡄⠘⠷⠚⠋⠀ Rearkick OP -- my grandpa's brother-in-law got one-shotted
⠈⠳⣄ from full hp in RL, please nerf!

What to do with merged /usr and dpkg-fsys-usrunmess

[Krzysztof Sobiecki]

Hi,
I saw warning and it advised me to use dpkg-fsys-usrunmess:
But now I see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008486
And even without that there are problems with that tool:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991218
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991192
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008316
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008478

Now what I'm supposed to do? Wait for dpkg-fsys-usrunmess-unmess?
Is my system broken because I have used dpkg-fsys-usrunmess(that dpkg
advised me to do?).
As a Debian user I'm confused and a bit angry.

What is way forward?

Thanks
Krzysztof Sobiecki

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Zack" == Zack Weinberg  writes:
Zack> Maybe it was not obvious to anyone in 2016 that the package
Zack> database being inconsistent with the filesystem could cause
Zack> actual data loss.  However, I think it was the responsibility
Zack> of the developers of usrmerge to find out how bad that problem
Zack> actually was, rather than dismissing it.  And, as it has
Zack> proven to be a genuinely critical problem, it is also their
Zack> responsibility to fix it, per the 'no one can be forced to do
Zack> anything' rule.

I agree with the above, but very definitely not with the further
conclusions you draw.

I think that the usrmerge proponents need to fix the problem before
encouraging moving  files from / to /usr in packages.
They can do that either by:

* proposing fixes to dpkg

or

* As Michael has pointed out, proposing mechanisms to guarantee that no
  Debian release both includes file migrations and package migrations.

We disagree about the rollback.
If we're ever communicating in a more realtime context I'd love to chat
about the balances around making sure we do not reward bad behavior.

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Ansgar  writes:

> So I do not think there should be a command-line option for this; unless
> it would behave like `--add-architecture` and register the setting
> somewhere.

Agreed.

I also think it needs to be a two-phase thing, because dpkg has to convert
its internal database to respect the new aliases.  So there should be a
desired configuration and then an applied configuration, and the latter is
only updated once the database conversion is complete, with dpkg refusing
to perform package operations if the desired configuration doesn't match
the applied configuration.

Even if the command that changes the configuration also does the database
conversion, I think we need this, since I don't believe it's possible to
do the database conversion atomically (or at least it would involve extra
complexity that I don't think we want), so we have to handle the database
conversion being interrupted.

> I do not think this should be a configuration file (unless you suggest
> supporting other values or removing individual lines; after all a
> configuration file exists to be changed).

Are you thinking of perhaps shipping this configuratino somewhere in
/usr/lib instead so that it's not marked as a configuration file?

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Ansgar]

On Mon, 2021-11-22 at 12:24 +, Simon McVittie wrote:
> Options to make path A an alias for path B would be a lot like --path-
> exclude in that they really only make sense in a configuration file in
> /etc/dpkg/dpkg.cfg.d, although for symmetry they would presumably also
> have to exist as command-line options just like --path-exclude does.

I would expect using --path-exclude when installing individual packages
to work: you just don't get the files from that package. (I haven't
tried.)

Merged-/usr (or --path-alias as you propose) does not work like this:
Unlike --path-exclude, you would almost always need to specify a
superset of --path-alias options for all future operations.

So I do not think there should be a command-line option for this;
unless it would behave like `--add-architecture` and register the
setting somewhere.

>     # in /etc/dpkg/dpkg.cfg.d/merged-usr
>     path-alias=/bin=usr/bin
>     path-alias=/lib32=usr/lib32
>     path-alias=/lib64=usr/lib64
>     path-alias=/lib=usr/lib
>     path-alias=/libo32=usr/libo32
>     path-alias=/libx32=usr/libx32
>     path-alias=/sbin=usr/sbin

I do not think this should be a configuration file (unless you suggest
supporting other values or removing individual lines; after all a
configuration file exists to be changed).

(Though we have various other settings that don't make sense as
configuration files in /etc, such as Debian's bugtracker and other
settings in /etc/dpkg/origins/debian, the version in
/etc/debian_version and so on.)

Ansgar

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Johannes Schauer Marin Rodrigues  writes:

> Please don't. Since 2014 it is possible to create a Debian chroot
> without taking care of the unpack order and without running any special
> setup beforehand that cannot be put into apt or dpkg itself (like
> creating /var/lib/dpkg/info which is not needed anymore since dpkg
> 1.20.0). One of my biggest motivations in writing mmdebstrap was to get
> rid of all the quirks that currently live in debootstrap, move them into
> apt and dpkg so that we end up with a declarative bootstrap process that
> is driven by the package contents, their metadata and maintainer scripts
> but does not need magic from the outside other than standardized
> interfaces. Requiring the merged-/usr setup to be done by the bootstrap
> script would be a step backwards.

I understand why you want this, but unfortunately it doesn't answer the
question of how to safely perform the upgrade and reconfigure dpkg.  Do
you have any thoughts on the drawbacks of the approach of putting the
symlinks in base-files that led me to lean away from doing that?

I'd love to get all of the machinery into packages as well, but I think
safely upgrading existing systems is even more important than that, and
I'm worried about our ability to do so if we put the machinery in
base-files.

-- 
Russ Allbery (r...@debian.org)  <https://www.eyrie.org/~eagle/>

Re: merged-/usr transition: debconf or not?

[Johannes Schauer Marin Rodrigues]

Quoting Russ Allbery (2021-11-20 18:22:27)
> * Create a new essential package that contains these symlinks and that
>   needs to be unpacked before any binaries are executed in the target file
>   system.  This has many of the advantages and drawbacks of the approach
>   of putting the symlinks in base-files, but may make it easier to handle
>   the upgrade problem.  Ideally an upgrade would then involve installing
>   usrmerge, letting it run, and then installing this new essential package
>   so that it takes over ownership of those symlinks.
> 
>   This still feels kind of complex and messy to me, but maybe less so.
> 
> * Create the symlinks directly in the bootstrapping script.  In other
>   words, after unpacking base-files, the bootstrapping script would
>   directly create the required symlinks in the target file system, before
>   unpacking any other package.
> 
>   This has the obvious drawback of moving things outside the packaging
>   system and creating a new special case that every bootstrapping script
>   needs to be aware of (and I know we have at least four or five that
>   would need modifications).  It has the advantage that the usrmerge
>   symlinks are now not in the dpkg database and thus not subject to
>   rewriting, and therefore won't need to be special-cased.  However, that
>   comes with the obvious disadvantage that they're not in the dpkg
>   database, so for instance dpkg -S /lib wouldn't find that symlink unless
>   it was added as some sort of dpkg-query special case (which doesn't seem
>   like a great idea).
> 
>   The advantage of this approach is that it closely mimicks what's already
>   happening now with the usrmerge package, and for which we therefore have
>   a lot of existing experience.

Please don't. Since 2014 it is possible to create a Debian chroot without
taking care of the unpack order and without running any special setup
beforehand that cannot be put into apt or dpkg itself (like creating
/var/lib/dpkg/info which is not needed anymore since dpkg 1.20.0). One of my
biggest motivations in writing mmdebstrap was to get rid of all the quirks that
currently live in debootstrap, move them into apt and dpkg so that we end up
with a declarative bootstrap process that is driven by the package contents,
their metadata and maintainer scripts but does not need magic from the outside
other than standardized interfaces. Requiring the merged-/usr setup to be done
by the bootstrap script would be a step backwards. Think about the following
points:

 * you create a Debian unstable chroot using snapshot.d.o -- now you have to
   somewhere put code that identifies that this is a chroot from the past and
   that has to decide whether that chroot should be merged-/usr or not. If
   merged-/usr setup was living somewhere in the Essential:yes package set
   the chroots would be created the right way automatically
 * same for derivatives -- now every bootstrap script has to learn which
   derivative defaults to merged-/usr, which ones do not and when they switched
 * we somehow need to convert installations of users that only run testing
   or unstable but never do full stable installation to merged-/usr -- one way
   to do that is to let one package in the Essential:yes set drive the
   conversion at which point the bootstrap script doesn't need to know about
   merged-/usr because the Essential:yes set knows how to set it up
 * think about it from a software engineering point of view: Debian as a
   component based software distribution should move specific functionality
   into its packages and global functionality into declarative metadata.

Just as we are currently trying to reduce maintainer scripts and replace them
by declarative approaches, we should also try to reduce the magic in our
bootstrap scripts and move it either to our package managers or replace them by
declarative solutions. We have been moving into the right direction during the
last few years as apt and dpkg have picked up functionality I had earlier put
into mmdebstrap (thanks a lot to guillem, DonKult and juliank for picking up my
patches or writing the functionality themselves). It would mean a step
backwards if all bootstrap scripts would have to carry the setup_merged_usr()
function from debootstrap and then decide by some $magic when this function
should be executed or not.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: merged-/usr transition: debconf or not?

[Simon McVittie]

On Sat, 20 Nov 2021 at 14:54:30 -0800, Russ Allbery wrote:
> That would make it akin to dpkg
> --add-architecture, which feels like a good model (although as you mention
> I don't think we want --remove-architecture).

Instead of dpkg --add-architecture, which is a top-level "verb" like
dpkg --remove or dpkg --install [1], probably a better thing to resemble
would be the options that already exist to change dpkg's behaviour,
and in particular --path-exclude and --path-include?

Options to make path A an alias for path B would be a lot like --path-exclude
in that they really only make sense in a configuration file in
/etc/dpkg/dpkg.cfg.d, although for symmetry they would presumably also
have to exist as command-line options just like --path-exclude does.

Here's a possible design (not fully thought out, and maybe wrong or
unimplementable):

--path-alias ALIAS=TARGET
When unpacking or querying packages, paths below ALIAS will be
transformed into the corresponding path below CANONICALIZED, where
CANONICALIZED is the result of resolving TARGET relative to the
directory containing ALIAS. This transformation is done before
checking for file conflicts.

ALIAS must be an absolute path.

If the file list of any unpacked package contains a directory at
ALIAS and/or CANONICALIZED, then ALIAS will be created as a symbolic
link to TARGET, with its target directory created at CANONICALIZED.

It is an error to have multiple --path-alias directives for
the same ALIAS with different TARGET paths.

This option should usually be set via /etc/dpkg/dpkg.cfg.d rather
than as a command-line option.

For example, a series of directives like path-alias=/bin=usr/bin
can be used to implement the "/usr merge", causing /bin/bash to be
unpacked as /usr/bin/bash while creating a symbolic link
/bin -> usr/bin.

With that, I believe a full implementation of merged-usr for all known
architectures would be to install something like this, perhaps via
base-files:

# in /etc/dpkg/dpkg.cfg.d/merged-usr
path-alias=/bin=usr/bin
path-alias=/lib32=usr/lib32
path-alias=/lib64=usr/lib64
path-alias=/lib=usr/lib
path-alias=/libo32=usr/libo32
path-alias=/libx32=usr/libx32
path-alias=/sbin=usr/sbin

Rationale for TARGET being allowed to be relative: I believe the compat
symlinks in merged-/usr are preferred in the form /bin -> usr/bin, to
make sysroots (containers, chroots, btrfs root filesystem snapshots,
etc.) easier to manipulate from outside.

Rationale for ALIAS only being created if ALIAS and/or CANONICALIZED
really exist in a package: we would probably prefer the multilib
directories to be created "lazily", rather than having a /libo32 on
systems that will never install a mips* package and a /libx32 on systems
that will never install an x32 package. This is slightly nicer than the
current implementation in convert-usrmerge and debootstrap, which creates
those directories "eagerly" whenever the appropriate architectures are
configured as either primary or foreign architectures.

smcv

[1] if dpkg had been designed with more recent CLI conventions in mind,
I expect these "verbs" might have been subcommands like "dpkg remove",
etc. as seen in apt/git/flatpak, which makes it a bit more obvious
that only one is allowed

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Gabor Gombas  writes:

> If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to
> /usr/lib64/*", then this can easily be avoided.

True, although I think you still want to ensure that dpkg never messes
with those links.  I don't think you want base-files to be able to drop
one of those links from the package and cause it to be removed by dpkg,
for instance.

Maybe I'm borrowing trouble for a scenario that won't happen (and I think
dpkg may have some special-casing of symlinks to directories anyway).

> Talking about "special casing" in dpkg is bothering me for a while. And
> there is a relatively simple way to avoid any kind of special casing:
> move the information out to a configuration file (which would _not_ be a
> conffile) - and now the code has no special casing, just
> configuration-driven logic.

All that I mean by special-case is that I think it would be a mistake to
try to shoehorn support for merged-/usr into some pre-existing dpkg
concept.  The merged-/usr links are special relative to other files that
dpkg is managing and need to be treated with special care.  We should add
a new concept to dpkg for these sorts of links.

I agree with making the exact set of links configurable.  This also allows
handling the architecture-specific links in a clean way.

> This new configuration file could be shipped by base-files itself, to
> ensure it does not get out of sync with the filesystem structure shipped
> in base-files. Then base-files' postinst would more or less need to
> include the current usrmerge package, plus whatever is needed to convert
> dpkg's database.

I don't think we want base-files to take over the logic of usrmerge.

usrmerge is both complex (in part because it relies on Perl) and not 100%
hardened against weird issues, being interrupted in the middle of running,
etc.  That's all fine for what it's doing as a one-time migration tool
that's run with care during an upgrade when the administrator already
knows Big Things are happening to the system.  Putting the logic into
base-files makes it more of a routine thing and I think it should be a lot
more hardened if we do that (and I don't think that's a good use of
anyone's time).  We ideally want usrmerge to be a tool each system runs
once, under controlled circumstances, and then never again (and isn't
needed at all for bootstrapping).

We have a migration strategy that's working.  Let's stick with that
migration strategy without messing with it and instead just add a dpkg
database conversion step to the end of that migration.

I do agree that something needs to handle installing the configuration
file and laid out some options for that in another message.

> This new configuration file would not be consumed by dpkg directly when
> installing packages, but only when the database conversion is called,
> and dpkg would keep an internal list of the rewriting rules which are
> active. Doing so would allow enforcing rewriting rules can only be added
> but never removed, which would avoid potential issues if base-files gets
> downgraded.

This is a very good idea.  I was starting to think along those lines, but
hadn't taken it quite far enough.  That would make it akin to dpkg
--add-architecture, which feels like a good model (although as you mention
I don't think we want --remove-architecture).

-- 
Russ Allbery (r...@debian.org)  <https://www.eyrie.org/~eagle/>

Re: merged-/usr transition: debconf or not?

[Gabor Gombas]

Hi,

On Sat, Nov 20, 2021 at 09:22:27AM -0800, Russ Allbery wrote:

>   The drawback here is that dpkg is going to rewrite all paths like /lib64
>   to /usr/lib64, which would naively *also* apply to the base-files
>   package when it looks at that package, but that can't be allowed because
>   now we're back to the situation where dpkg's state database is
>   inconsistent with the file system and dpkg thinks that base-files
>   contains some nonsensical /usr/lib64 to /usr/lib64 symlink.

If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to
/usr/lib64/*", then this can easily be avoided.

>   I think in this approach there would need to be some special-case code
>   directly in dpkg that recognizes the usrmerge symlinks [...]

Talking about "special casing" in dpkg is bothering me for a while. And
there is a relatively simple way to avoid any kind of special casing:
move the information out to a configuration file (which would _not_ be a
conffile) - and now the code has no special casing, just
configuration-driven logic.

This new configuration file could be shipped by base-files itself, to
ensure it does not get out of sync with the filesystem structure shipped
in base-files. Then base-files' postinst would more or less need to
include the current usrmerge package, plus whatever is needed to convert
dpkg's database.

This new configuration file would not be consumed by dpkg directly when
installing packages, but only when the database conversion is called,
and dpkg would keep an internal list of the rewriting rules which are
active. Doing so would allow enforcing rewriting rules can only be added
but never removed, which would avoid potential issues if base-files gets
downgraded.

Gabor

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Russ Allbery  writes:

> Well, bootstrapping a new Debian system involves running a tool that
> bootstraps a new Debian system.  I think you're constraining the problem
> too much.

> It's a nice property that everything on the system comes straight from a
> Debian package, but it's not a strict requirement, nor is it currently
> generally the case (maintainer scripts do things, for instance).  Those
> symlinks are very special from a dpkg perspective; dpkg needs to refuse
> to mess with them even when upgrading a package that provides them,
> which would mean some irritating special-casing I suspect.  It's not
> clear to me that shipping them as tar members of a package is the right
> way to go, as opposed to creating them separately as part of early
> system configuration.

Having slept on this, let me make this problem concrete.

I'm going to make the following assumptions:

* We have some mechanism to put dpkg into what I've been calling
  merged-/usr mode.  In this mode, it pre-filters all input paths from
  whatever source (including arguments to dpkg-divert,
  update-alternatives, etc.)  and canonicalizes them in a very specific
  way: the directories that become symlinks in merged /usr are replaced in
  each path with their canonical paths.  So /bin/ls becomes /usr/bin/ls,
  /lib64/ld-linux-x86-64.so.2 becomes /usr/lib64/ld-linux-x86-64.so.2, and
  so forth.

* When bootstrapping a new Debian system, we want to put dpkg into
  merged-/usr mode as early as possible.

* If dpkg is in merged-/usr mode, the first thing it does is checks the
  file system on which its operating and ensures that the expected
  symlinks already exist.  If they do not, it aborts because operating on
  that file system is unsafe.  (Leaving aside for the moment whether there
  should be some -force option, etc.)

This produces a bootstrapping problem: all dynamic binaries on, say, an
amd64 system refer to /lib64/ld-linux-x86-64.so.2 [1].  Therefore, before
the first binary is run from within the context of the newly-installed
system, either that path must exist as-is (which we don't want because we
want to create a merged-/usr system where it belongs in /usr/lib64) or the
symlink from /lib64 to /usr/lib64 must already exist.

I think it's fairly obvious that we don't want a long-term design in which
the libc6 package has to continue to ship /lib64/ld-linux-x86-64.so.2, we
originally unpack that file in that path during bootstrap, and then
something has to come along later and move it to /usr/lib64 and create a
symlink.  This has numerous drawbacks: weird special cases people have to
remember, not being able to reconcile the contents of packages with their
canonical paths in the long run, a window where we have to do file system
surgery atomically, etc.  Instead, we want to live in a world in which
libc6 can ship /usr/lib64/ld-linux-x86-64.so.2, dpkg sees it as shipping
that path, but everything works fine during bootstrap because the /lib64
to /usr/lib64 symlink is already established before we have to execute
binaries in the new environment.  (Obviously it may be some time before we
actually change the contents of the libc6 package; that's fine, the point
of adding a path filter to dpkg is that we can take our time on that.)

So, assuming we have a libc6 package that contains either
/usr/lib64/ld-linux-x86-64.so.2 or /lib64/ld-linux-x86-64.so.2, how do we
bootstrap this system?

I can see a few approaches:

* Put the required symlinks into base-files.  This preserves the nice
  property that every file on the system comes from and is registered as
  belonging to a Debian package (which we don't fully meet but which we
  always aspire to).  I haven't studied all the various tools for
  bootstrapping a Debian system, but I presume that unpacking base-files
  is the first thing that any of them do.  This will therefore lay down
  the symlinks from the start and it won't matter what path under which
  the libc6 package ships the files if it's extracted over the resulting
  file system with ar+tar.

  The drawback here is that dpkg is going to rewrite all paths like /lib64
  to /usr/lib64, which would naively *also* apply to the base-files
  package when it looks at that package, but that can't be allowed because
  now we're back to the situation where dpkg's state database is
  inconsistent with the file system and dpkg thinks that base-files
  contains some nonsensical /usr/lib64 to /usr/lib64 symlink.

  I think in this approach there would need to be some special-case code
  directly in dpkg that recognizes the usrmerge symlinks (and only
  specifically those symlinks) as special and preserves them as-is in the
  installed package database.  (I think it's probably better to
  special-case the specific symlink than to special-case base-files the
  package.)  We will then need rules that base-files must always contain
  those symlinks and they can't move between packages, etc., (or the

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Simon Richter  writes:

> Bootstrapping a new Debian system works by unpacking Essential packages
> with ar+tar, i.e. not using dpkg. These packages will always be unpacked
> to the file names listed inside the packages.

Well, bootstrapping a new Debian system involves running a tool that
bootstraps a new Debian system.  I think you're constraining the problem
too much.

It's a nice property that everything on the system comes straight from a
Debian package, but it's not a strict requirement, nor is it currently
generally the case (maintainer scripts do things, for instance).  Those
symlinks are very special from a dpkg perspective; dpkg needs to refuse to
mess with them even when upgrading a package that provides them, which
would mean some irritating special-casing I suspect.  It's not clear to me
that shipping them as tar members of a package is the right way to go, as
opposed to creating them separately as part of early system configuration.

> Dpkg has an elaborate per-file tracking to recover from errors, and this
> change introduces several new states and transitions,

Why?  I am not convinced this is true, apart from the obviously-needed
one-time conversion of the state database.

It involves a new filter, applied whenever a package is installed, that
changes all the paths in the package to their merged-/usr paths, something
that seems like it should be done before any other package processing.
Once that's done, why would any new states or transitions be required?

You could convince me that writing the filter is complicated because there
may not be one place in dpkg where you can currently put such path
rewriting since dpkg probably wasn't designed with that operation in mind.
But it's going to be harder to convince me that there are new states or
transitions; that smells like an over-complicated design.

> That is the goal, yes, but this is a massive undertaking.

I'm still not convinced of this.

> We already have a quick hack for usrmerge support in dpkg, which is the
> check that allows moving a file from /bin to /usr/bin within the same
> package without the file being deleted in the end (because deletions are
> processed last for obvious reasons), and the obscure rule is precisely
> the limitation of this hack.

This already sounds conceptually more complicated than just solving the
problem properly, with the caveat as mentioned above that writing the
necessary filter may be difficult.

-- 
Russ Allbery (r...@debian.org)  <https://www.eyrie.org/~eagle/>

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

> > (I've used /bin -> /usr/bin as an example, but the canonicalization must
> > deal with the other merged paths too.)

/lib and /lib64 are a bit special, because they are part of the ELF ABI,
and any manipulation of these paths needs to be atomic.

Bootstrapping a new Debian system works by unpacking Essential packages
with ar+tar, i.e. not using dpkg. These packages will always be unpacked
to the file names listed inside the packages.

So, new systems will either need a package that ships the /lib ->
usr/lib symlink in the Essential set, or we need to ship the dynamic
linker in /lib, which would be a conflict during unpacking.

> > The second bit ensures that all new operations write canonicalized paths
> > (e.g. /usr/bin/foo) into the database regardless of whether the .deb 
> > shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in
> > sync with the filesystem moving forward.

This loses the information about the original path, so this would be a
trapdoor operation that can not be undone without affected packages
going into reinstallation-required state (and we probably couldn't even
identify those).

Ideally, I'd rather not hardcode the list of top-level symlinks into
dpkg, because they might be architecture-specific, or change over time,
so there should be a mechanism to add and remove transformations.

> > The one-time canonicalization can be removed once non-usrmerged systems
> > are no longer supported. The second bit needs to live (probably a lot) 
> > longer, until it's no longer reasonable to install a .deb containing
> > non-usrmerged paths (which might be old or from a third party).

My expectation would be that this code will be here to stay, because it
will be required for bootstrap.

> Are we missing something?  If not, what is blocking this solution?  Is it
> simply a matter of someone digging into dpkg's code sufficiently to put
> these changes in the correct location?  Or is there some other issue?

If it were that simple, someone would have done that already. Dpkg has
an elaborate per-file tracking to recover from errors, and this change
introduces several new states and transitions, last but not least
because we also have filters and diversions as transformations on the
database -- but the usrmerge transformation is supposed to be orthogonal
to that.

> It seems like a huge waste of resources to me to do archive-wide package
> inspection to try to find patterns that might cause problems, and to ask
> all maintainers to remember this rather obscure rule, when we could just
> fix dpkg to make the problem go away.

That is the goal, yes, but this is a massive undertaking. We already
have a quick hack for usrmerge support in dpkg, which is the check that
allows moving a file from /bin to /usr/bin within the same package
without the file being deleted in the end (because deletions are
processed last for obvious reasons), and the obscure rule is precisely
the limitation of this hack.

I've stumbled across this hack while investigating if it was possible to
"just" fix dpkg. :P

   Simon

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Simon Richter  writes:

> I think the main blocker at the moment is that the dpkg change *also*
> has the potential to break a lot of things if it isn't carefully
> designed.

I think you can simplify that problem space considerably because we know
exactly what symlinks we care about and we don't intend to support going
backwards (in other words, once you have merged /usr, it's not supported
to turn /bin, etc., back into regular directories again).

Therefore, dpkg can be configured with the list of the symlinks that
*must* exist for it to be willing to operate on the system.  Once it has
been put into merged-/usr mode (which includes updating its state database
to reflect the new canonical paths of all installed files), if those
symlinks do not exist or do not have the correct targets, the system state
is not safe and dpkg can (and probably should) abort and refuse to operate
on that system.

That modifies the problem of systems in various weird configurations from
"weird things may happen" to "dpkg will (safely) refuse to run" and we can
provide a tool to fix such a system by creating the correct symlinks.
(This tool may well just be the existing usrmerge tool.)  Getting into
that state will be hopefully rare if we structure the upgrade process so
that first one converts the system to merged-/usr if this has not already
been done, then upgrades dpkg, then tells dpkg to switch into merged-/usr
mode, and then safely upgrades the remaining packages.

> From dpkg's perspective, we now (kind of) change the policy for
> directory-vs-symlink conflicts, which dpkg currently resolves in favour
> of the directory.

I don't think we should be doing anything of the sort.  I think we should
introduce a new concept specifically for this case and have the remappings
of files for merged-/usr, once dpkg is in merged-/usr mode, take
precedence over any rules like that.  We do not want dpkg to be applying
some policy or heuristic about directory vs. symlink conflicts for
symlinks related to merged-/usr.  We know exactly what we want the rule
and behavior to be.  That sort of remapping should happen way before any
other dpkg analysis of the package or system state.

In other words, I argue that once dpkg is in merged-/usr mode, it should
treat a package with files in /bin exactly as if the files were in
/usr/bin, at an earlier stage than when it starts looking at the file
system.

We're not trying to create a generalized feature here.  Merged-/usr has a
very specific definition, with very specific and clear behavior that we
need from the package manager.  Trying to generalize this is going to make
a mess and make it much harder to reason about the behavior or test it.

-- 
Russ Allbery (r...@debian.org)  <https://www.eyrie.org/~eagle/>

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

On Fri, Nov 19, 2021 at 12:28:56PM -0700, Sam Hartman wrote:

> But we could you know fix dpkg:-)
> I'm sure that will be painful at the political level, but personally I
> think it needs doing.

I think the main blocker at the moment is that the dpkg change *also*
has the potential to break a lot of things if it isn't carefully
designed.

The problem space right now is huge:

 - for upgraded systems, the system could be pre-usrmerge, or the
   conversion could have been performed by any usrmerge version that
   ever existed (and the usrmerge package could have been deinstalled
   and reinstalled since, so the patches to the conffiles it performs
   could be inconsistently applied).
 - for upgraded systems, any version of usrmerge since the last stable
   release could be installed
 - for upgraded systems, unless the usrmerge package is discontinued,
   it could be upgraded at any point during the update.
 - for freshly installed systems, dpkg is first invoked on a
   non-usrmerged tree, and should convert the installation as soon as it
   becomes aware that conversion is desired (which might not be
   something we want to hardcode in dpkg, but possibly configure from a
   separate, Debian-specific package like base-files).
 - for freshly installed systems, initial unpack might be controlled by
   debootstrap from stable, which is usrmerge aware, so some of its
   workarounds may be active, and we need to properly capture this
   state, too.
 - freshly installed systems might be created with cdebootstrap or
   multistrap, and could be generated in --foreign mode
 - the dynamic loader is specified to be on the root filesystem, and that
   is part of the ABI standard and not under our control, so we cannot get
   a conflict-free dpkg database in bookworm.

>From dpkg's perspective, we now (kind of) change the policy for
directory-vs-symlink conflicts, which dpkg currently resolves in favour
of the directory. That was done in ancient times because it was somewhat
easy to do this in a package by accident. Not sure if that still
applies, but if it does, then we need to formulate the new policy so
that we don't catch a regression here (which is why my original idea to
just ship the symlinks in base-files is a bad idea).

>From the way dpkg and the Debian Policy are initially designed, it is
obvious to me that they started out as specification documents, and only
when these were hammered out, they were poured into code and rules, and
we've been operating from that stable base since, with two exceptions
where technical facts were created without updating Policy accordingly,
and which has both times been controversial.

My interpretation of the "political" situation is that we need this to be a
group effort, because no single person has the necessary time to do a
thorough enough job that they would feel comfortable signing off on it. I'd
be reluctant to do so even if I had three months of uninterrupted time --
that's the level of complexity I see here, and I suspect Guillem feels the
same.

So I believe the way forward will be writing a specification and giving it
enough eyeballs to identify problem cases. Finding the adversarial example
for the status quo was easy, since I had to find just one -- the goal now
is to get to a state where we don't find such an example easily anymore.

The specification should explain how the transition can be finished with
the bookworm release for all the different ways packages can be installed,
and it should describe the necessary code changes and new test cases to get
full coverage, and the document should be signed off by multiple people who
jointly take responsibility, to avoid placing all of that on a single
person -- because that's what's impeding progress right now, that no one
wants to be that person or even feels able to.

The description of the problem space I put above is likely incomplete and
overzealous at the same time (for example, I haven't checked how different
the previous usrmerge packages are, and which of them ever made it into a
stable release), so this is also just a starting point. I'll happily spend
time refining this or any other proposal, but I'm too time-constrained to
be the main driving force here -- this is not my day job, after all.

   Simon

Re: merged-/usr transition: debconf or not?

[Timo Röhling]

* Russ Allbery  [2021-11-19 11:46]:

I also don't understand why this isn't the correct solution.  It seems
obvious to me that we should simply teach dpkg about path aliases and have
it update both its internal state and its processing of new packages to
canonicalize paths so that it has an accurate representation of the world,
and then ensure dpkg is upgraded first.

+1


Are we missing something?  If not, what is blocking this solution?  Is it
simply a matter of someone digging into dpkg's code sufficiently to put
these changes in the correct location?  Or is there some other issue?

AFAIR, Guillem has not commented on the solution when it was brought
up on d-devel [1], so I'm not sure if he would accept patches, let
alone support anyone working on this. I'd be willing to assist him
and contribute code, but I'm not going to spend a good chunk of my
spare time just to see me work summarily rejected.

Cheers
Timo

[1] https://lists.debian.org/debian-devel/2021/08/msg00438.html


--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Fri, Nov 19, 2021, at 3:23 PM, Zack Weinberg wrote:
> On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
>>> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
>>> Luca Bocassi wrote:
>>> ...
>>>> nobody has actually seen [the file disappearance bug]
>>>> happen, to the best of my knowledge.
>>>
>>> I already explained why that doesn't prove the bug is a non-issue.
>>> To the contrary; it means there is an enormous installed base of
>>> systems where the bug is latent, waiting to cause problems under
>>> conditions which we can reasonably expect to occur shortly after
>>> the release of bookworm.
>>
>> Why would the release of bookworm make any difference?
>
> Up until the release of bookworm, all Debian packages must be
> constructed on the assumption that they _might_ be unpacked on a
> system that has not yet been converted to merged /usr.  Particularly
> for priority-required packages, this means that no one will be moving
> files from /bin, /lib, etc to /usr in the bookworm cycle.
>
> Post-bookworm, if nothing changes, that assumption will no longer be
> in force, and people who maintain packages that install files into /
> will want to simplify their packaging by installing everything in /usr
> instead.  If they also want to change the binary packages that ships
> some of those files at any point in the same release cycle -- kaboom.

After having caught up on the thread, I see that the conditions required
to trigger the bug are somewhat more complicated, but the point remains:
particularly for the packages where a lost file could render the system
unbootable, changes that would trigger the bug have been deferred until
post-bookworm, *and* we can reasonably expect the maintainers of those
packages to *want* to make changes with a high risk of triggering the bug.
I imagine the coreutils maintainers are looking forward to getting rid
of their list of programs that go in /bin, and the extra debian/rules
logic to go with it, for instance
(https://sources.debian.org/src/coreutils/8.32-4.1/debian/rules/#L16).

zw

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
>> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
>> Luca Bocassi wrote:
>> ...
>>> nobody has actually seen [the file disappearance bug]
>>> happen, to the best of my knowledge.
>>
>> I already explained why that doesn't prove the bug is a non-issue.
>> To the contrary; it means there is an enormous installed base of
>> systems where the bug is latent, waiting to cause problems under
>> conditions which we can reasonably expect to occur shortly after
>> the release of bookworm.
>
> Why would the release of bookworm make any difference?

Up until the release of bookworm, all Debian packages must be
constructed on the assumption that they _might_ be unpacked on a
system that has not yet been converted to merged /usr.  Particularly
for priority-required packages, this means that no one will be moving
files from /bin, /lib, etc to /usr in the bookworm cycle.

Post-bookworm, if nothing changes, that assumption will no longer be
in force, and people who maintain packages that install files into /
will want to simplify their packaging by installing everything in /usr
instead.  If they also want to change the binary packages that ships
some of those files at any point in the same release cycle -- kaboom.

zw

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Richard Laager  writes:

> Is this particularly hard to fix, though?

> Can't we just do something like the following pseudo-code:

[...]

> (I've used /bin -> /usr/bin as an example, but the canonicalization must
> deal with the other merged paths too.)

> The second bit ensures that all new operations write canonicalized paths
> (e.g. /usr/bin/foo) into the database regardless of whether the .deb 
> shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in
> sync with the filesystem moving forward.

> The first bit performs a one-time canonicalization of paths in the
> existing database. An on-disk flag ensures that this is run only once 
> (for performance reasons only, because it's already idempotent). This
> corrects the existing database-filesystem mismatch.

> The one-time canonicalization can be removed once non-usrmerged systems
> are no longer supported. The second bit needs to live (probably a lot) 
> longer, until it's no longer reasonable to install a .deb containing
> non-usrmerged paths (which might be old or from a third party).

> Am I missing something here?

I also don't understand why this isn't the correct solution.  It seems
obvious to me that we should simply teach dpkg about path aliases and have
it update both its internal state and its processing of new packages to
canonicalize paths so that it has an accurate representation of the world,
and then ensure dpkg is upgraded first.

Are we missing something?  If not, what is blocking this solution?  Is it
simply a matter of someone digging into dpkg's code sufficiently to put
these changes in the correct location?  Or is there some other issue?

It seems like a huge waste of resources to me to do archive-wide package
inspection to try to find patterns that might cause problems, and to ask
all maintainers to remember this rather obscure rule, when we could just
fix dpkg to make the problem go away.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Paul Gevers]

Hi

On 18-11-2021 22:44, Marco d'Itri wrote:

On Nov 18, Zack Weinberg  wrote:


Are you seriously claiming that that phenomenon is not a severity:critical bug?

I am seriously claming that whatever you are referring to, if true, is
such a contrived example that does not actually happen in real life
(or at least, it does not happen frequently enough).


I'm thinking of bug #972936 and bug #953562. Did I remember those 
wrongly as being an example of this?


Paul



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Michael" == Michael Biebl  writes:

Michael> Am 17.11.2021 um 19:57 schrieb Sam Hartman:
Michael> AIUI simply moving files from / to /usr within the same
Michael> package is not problematic.

Sorry, I was being overly simplistic.
I think your understanding is mostly correct.
You need to make sure that in the same release you do not both
move files from / to /usr

and then later move files between packages.

It's not just that you can't do it at the same time.
It's that it cannot happen within a given release cycle.

Your packaging team may be organized enough to make sure you don't do
that.
And I guess if you've got some plan to make sure you never trip the
problematic conditions it would be okay.

I don't think the project as a whole is taht organized.
So, for example, I think it would be problematic for general tools like
debhelper to move files, because we cannot guarantee that maintainers
will not later (or earlier) move those files between packages.

But we could you know fix dpkg:-)
I'm sure that will be painful at the political level, but personally I
think it needs doing.

Anyway, thanks for calling me out on being imprecise.  There's enough
FUD going around in this discussion I do not want to add to it.

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:

Marco> On Nov 18, Zack Weinberg  wrote:
>> Are you seriously claiming that that phenomenon is not a
>> severity:critical bug?
Marco> I am seriously claming that whatever you are referring to, if
Marco> true, is such a contrived example that does not actually
Marco> happen in real life (or at least, it does not happen
Marco> frequently enough).

It was convincing enough to convince me, the TC and several other
participants in the discussion.

--Sam

Re: merged-/usr transition: debconf or not?

[Sven Joachim]

On 2021-11-19 15:37 +0100, Michael Biebl wrote:

> On 19.11.21 11:58, Philip Hands wrote:
>> Ansgar  writes:
>>
 * doing this will, in a non-negligible number of cases, trigger the
 bug to manifest on systems where that package is upgraded from a
 version where the move had not taken place to one where it has.
>>>
>>> Why do you claim that?
>>>
>>> Given packages already did such moves in the last years and you claim
>>> this happens in a non-negligible number of cases, could you please
>>> point to some examples where this already happens in practice?
>> My understanding is that in order to trigger this bug you need at
>> least
>> to both move a file from one place to the other, and also to rename the
>> package that contains that file or move ownership to another package.
>> I suspect that you might also need to be unlucky with the order that
>> apt/dpkg decides to do the installation and, depending upon how far
>> apart the move and the rename happens, also unlucky with your choice of
>> from and to versions of the packages in question.
>
> Right. I think it would be immensely useful to have an actual
> reproducer so one could study the issue more closely or at least a bug
> report, which describes the issue in more detail, like the exact
> circumstances when this can happen.
> So far this is merely theoretical, right?
> Or do we have a documented instance of this happening?

#953562 seems to be such an instance.

Cheers,
   Sven

Re: merged-/usr transition: debconf or not?

[Marc Haber]

On Fri, 19 Nov 2021 11:58:39 +0100, Philip Hands 
wrote:
>Given that these bugs are going to be utter bastards to reproduce, and
>you can be sure that we'll have enough diversity in installed systems
>that some people are going to manage to be sufficiently unlucky, it
>would be nice to know the sort of damage we might expect.
>
>It strikes me that we ought to be able to screen our own repos for
>packages that could be able to tickle this bug. That would give us the
>chance to look at what sorts of files we might realistically expect to
>be clobbered, it should give some indication of how many packages we
>should expect to be able to trigger this, and knowing this might suggest
>plausible work-arounds.

This is one of the cases where I wish that Debian would be a more
centrally organized project. Red Hat or SuSE would just fix their
package management and go on with their business.

It's a pity that we have actually THINK about alternatives to that
trivial and obvious approach because we leave our core package
maintainers too much freedom to stall.

Grüße
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: merged-/usr transition: debconf or not?

[Michael Biebl]

On 19.11.21 11:58, Philip Hands wrote:

Ansgar  writes:


* doing this will, in a non-negligible number of cases, trigger the
bug to manifest on systems where that package is upgraded from a
version where the move had not taken place to one where it has.


Why do you claim that?

Given packages already did such moves in the last years and you claim
this happens in a non-negligible number of cases, could you please
point to some examples where this already happens in practice?


My understanding is that in order to trigger this bug you need at least
to both move a file from one place to the other, and also to rename the
package that contains that file or move ownership to another package.

I suspect that you might also need to be unlucky with the order that
apt/dpkg decides to do the installation and, depending upon how far
apart the move and the rename happens, also unlucky with your choice of
from and to versions of the packages in question.


Right. I think it would be immensely useful to have an actual reproducer 
so one could study the issue more closely or at least a bug report, 
which describes the issue in more detail, like the exact circumstances 
when this can happen.

So far this is merely theoretical, right?
Or do we have a documented instance of this happening?



Given that these bugs are going to be utter bastards to reproduce, and
you can be sure that we'll have enough diversity in installed systems
that some people are going to manage to be sufficiently unlucky, it
would be nice to know the sort of damage we might expect.

It strikes me that we ought to be able to screen our own repos for
packages that could be able to tickle this bug. That would give us the
chance to look at what sorts of files we might realistically expect to
be clobbered, it should give some indication of how many packages we
should expect to be able to trigger this, and knowing this might suggest
plausible work-arounds.

Of course, that doesn't help with packages from third-party repos,
including our downstreams, but at present we seem to be discussing this
with very little hard data.

It occurs to me that one could lose quite a few files on the average
Debian install (if they were selected at random) without even noticing,


Shouldn't debsums be able to detect such missing files (at least for the 
vast majority of packages which ship a md5sums file).


I run that semi-regularly on all of my systems, most of them are 
/usr-merged and I haven't noticed any missing files yet which I could trace



whereas a very few files would render systems unbootable, so knowing a
bit more about which files are realistically at risk would be very
helpful in understanding the severity of the problem.

If anyone's got good ideas about how to gather this information, I'm
very happy to help with the effort to do so.


I'd be more then happy to help here as well.

Regards,
Michael




OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Fri, Nov 19, 2021 at 10:06:13AM +0100, Ansgar wrote:
> Given packages already did such moves in the last years and you claim
> this happens in a non-negligible number of cases, could you please
> point to some examples where this already happens in practice?

You need a / → /usr¹ and a pkg-a → pkg-b move in the same release.
Also, you need to have (the new) pkg-b be unpacked before pkg-a.

An example³ would be coreutils/util-linux/… moving everything from /bin
to /usr/bin and in the same Debian release splitting out one (or more)
of their tools into their own package (as they usually do).
As those are essential they will Pre-Depend² on the split out package
which guarantees that pkg-b will be unpacked before pkg-a.

The result is that the split out tool will be gone from /usr-merged
systems – which at that point should be all systems.


Another example would be systemd files debhelper moved for some time
already while the package does a foo and foo-data split in the same
Debian release. Just need to "solve" the unpack order now, but I will
leave that as an exercise for the reader.



The move and reorganisation is both forbidden by the CTTE for Debian 12
in "Moratorium on moving files' logical locations into /usr" which even
describes this problem as one of the reasons for it, but hopes to have
it resolved by 13 (without mentioning how).

Are you suggesting that Debian will use 13 to move each and every
file in / to its /usr counter-path while forbidding that packages
including moves are reorganised before 14 ?

Good thing 'which' isn't in /bin I guess. (SCNR)

Disclaimer: I am as usual not arguing for switching into full speed
reverse mode. I would "just" prefer if we acknowledge that problems
exist we have to deal with. Its gonna be hard enough to actually resolve
them given all bridges have been burned down years ago by pretending its
not a problem that dpkg has no idea what is done behind its back to the
files its supposed to manage.

(The problem itself isn't unique⁴ to /usr-merge, so ideally it would be
 resolved regardless, but /usr-merge undoubtedly makes heavy use of it
 so in an ideal world those interested in it would not only acknowledge
 the problems but actually work together to resolve them.
 Sadly, that isn't the world we seem to be stuck in at all.)


Best regards

David Kalnischkies

¹ You could of course also move the other way around.
² You can achieve the same with other dependency types, it is just
  harder to trigger in simple testcases as apt & dpkg try to avoid
  the auto-deconfiguration of pkg-a if there is an easy way out.
³ For fun I wrote an apt testcase with coreutils splitting out ln⁴,
  that might be a bit unrealistic, but you get the idea (attached).
⁴ as, you know, /usr-merge being the last symlink we will ever need
#!/bin/sh
set -e

TESTDIR="$(readlink -f "$(dirname "$0")")"
. "$TESTDIR/framework"

setupenvironment
configarchitecture 'native'

#mkdir -p rootdir/bin
ln -s usr/bin rootdir/bin

touch ln

mkdir -p tree/bin
cp -a ln tree/bin
buildsimplenativepackage 'coreutils' 'native' '1' 'stable' '' '' '' '' 
'tree/bin'
rm -rf tree

buildsimplenativepackage 'coreutils' 'native' '2' 'unstable' 'Pre-Depends: 
unneeded-ln'

mkdir -p tree/usr/bin
cp -a ln tree/usr/bin
buildsimplenativepackage 'unneeded-ln' 'native' '2' 'unstable' 'Breaks: 
coreutils (<< 2)
Replaces: coreutils (<< 2)' '' '' '' 'tree/usr'
rm -rf tree

setupaptarchive

testfailure test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln
testsuccess apt install coreutils=1 -y
testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln

testsuccess apt full-upgrade -y
testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Philip Hands]

Ansgar  writes:

>> * doing this will, in a non-negligible number of cases, trigger the
>> bug to manifest on systems where that package is upgraded from a
>> version where the move had not taken place to one where it has.
>
> Why do you claim that?
>
> Given packages already did such moves in the last years and you claim
> this happens in a non-negligible number of cases, could you please
> point to some examples where this already happens in practice?

My understanding is that in order to trigger this bug you need at least
to both move a file from one place to the other, and also to rename the
package that contains that file or move ownership to another package.

I suspect that you might also need to be unlucky with the order that
apt/dpkg decides to do the installation and, depending upon how far
apart the move and the rename happens, also unlucky with your choice of
from and to versions of the packages in question.

Given that these bugs are going to be utter bastards to reproduce, and
you can be sure that we'll have enough diversity in installed systems
that some people are going to manage to be sufficiently unlucky, it
would be nice to know the sort of damage we might expect.

It strikes me that we ought to be able to screen our own repos for
packages that could be able to tickle this bug. That would give us the
chance to look at what sorts of files we might realistically expect to
be clobbered, it should give some indication of how many packages we
should expect to be able to trigger this, and knowing this might suggest
plausible work-arounds.

Of course, that doesn't help with packages from third-party repos,
including our downstreams, but at present we seem to be discussing this
with very little hard data.

It occurs to me that one could lose quite a few files on the average
Debian install (if they were selected at random) without even noticing,
whereas a very few files would render systems unbootable, so knowing a
bit more about which files are realistically at risk would be very
helpful in understanding the severity of the problem.

If anyone's got good ideas about how to gather this information, I'm
very happy to help with the effort to do so.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Ansgar]

On Thu, 2021-11-18 at 23:01 -0500, The Wanderer wrote:
> * to date, package maintainers have not yet begun moving
> already-packaged files from / to /usr/ (specifically because doing so
> would break systems that have not yet been migrated to merged-/usr,
> and Debian has not yet declared that such systems are unsupported),

That claim is incorrect:

- Some packages have moved files from /(s)bin to /usr/(s)bin. (This
  sometimes requires a compat symlink.)

- More packages have moved files from /lib to /usr/lib. (This often
  doesn't require any extra care.)

> * after bookworm, package maintainers will start moving already-
> packaged files from / to /usr/, and

s/start/continue/

> * doing this will, in a non-negligible number of cases, trigger the
> bug to manifest on systems where that package is upgraded from a
> version where the move had not taken place to one where it has.

Why do you claim that?

Given packages already did such moves in the last years and you claim
this happens in a non-negligible number of cases, could you please
point to some examples where this already happens in practice?

Ansgar

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 18, Zack Weinberg  wrote:

> Are you seriously claiming that that phenomenon is not a severity:critical 
> bug?
I am seriously claming that whatever you are referring to, if true, is 
such a contrived example that does not actually happen in real life
(or at least, it does not happen frequently enough).

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[The Wanderer]

On 2021-11-18 at 20:06, Luca Boccassi wrote:

> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
> 
>> Luca Bocassi wrote:

>>> [merged /usr] is the default. It has been the default for
>>> multiple releases of multiple distributions. All Ubuntu
>>> installations that were not using this default are now forcibly
>>> converted upon upgrade to 21.10.
>>> 
>>> And yet nobody has actually seen [the file disappearance bug]
>>> happen, to the best of my knowledge.
>> 
>> I already explained why that doesn't prove the bug is a non-issue.
>> To the contrary; it means there is an enormous installed base of
>> systems where the bug is latent, waiting to cause problems under
>> conditions which we can reasonably expect to occur shortly after
>> the release of bookworm.  Please do not make me repeat myself.
>> 
>> zw
> 
> I'm afraid you have not. Why would the release of bookworm make any
> difference? There will be nothing new that hasn't already been
> happening for years.

I interpret Zack's comment as referring to this, which he said in
https://lists.debian.org/debian-devel/2021/11/msg00205.html:

> [P]eople aren't doing the package changes that trigger the bug, yet.
> They can't, because that would break systems where /usr hasn't been
> merged. If the bug is not fixed I expect it will start causing
> problems in unstable *after* bookworm, since (as I understand the
> current transition plan) bookworm+1 development is the earliest that
> package maintainers may assume /bin is a symlink.

IOW, I interpret him as disagreeing with you that "there will be nothing
new that hasn't already been happening for years". Specifically, I parse
him as arguing that:

* to date, package maintainers have not yet begun moving
already-packaged files from / to /usr/ (specifically because doing so
would break systems that have not yet been migrated to merged-/usr, and
Debian has not yet declared that such systems are unsupported),

* after bookworm, package maintainers will start moving already-packaged
files from / to /usr/, and

* doing this will, in a non-negligible number of cases, trigger the bug
to manifest on systems where that package is upgraded from a version
where the move had not taken place to one where it has.

(Zack, if I've gotten any of those wrong, please don't hesitate to
correct me; I'll either apologize, or drop right back out of the
discussion to go hide in a metaphorical hole, if not both.)

Do you dispute any of those three points? If so, I'd be interested to
know which one(s), and why.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Luca Boccassi]

On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
Luca Bocassi wrote:
...
> [merged /usr] is the default. It has been the
> default for multiple releases of multiple distributions. All Ubuntu
> installations that were not using this default are now forcibly
> converted upon upgrade to 21.10.
> 
> And yet nobody has actually seen [the file disappearance bug]
> happen, to the best of my knowledge.

I already explained why that doesn't prove the bug is a non-issue.  To the 
contrary; it means there is an enormous installed base of systems where the bug 
is latent, waiting to cause problems under conditions which we can reasonably 
expect to occur shortly after the release of bookworm.  Please do not make me 
repeat myself.

zw

I'm afraid you have not. Why would the release of bookworm make any
difference? There will be nothing new that hasn't already been
happening for years.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

Luca Bocassi wrote:
...
> [merged /usr] is the default. It has been the
> default for multiple releases of multiple distributions. All Ubuntu
> installations that were not using this default are now forcibly
> converted upon upgrade to 21.10.
>
> And yet nobody has actually seen [the file disappearance bug]
> happen, to the best of my knowledge.

I already explained why that doesn't prove the bug is a non-issue.  To the 
contrary; it means there is an enormous installed base of systems where the bug 
is latent, waiting to cause problems under conditions which we can reasonably 
expect to occur shortly after the release of bookworm.  Please do not make me 
repeat myself.

zw

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

Marco d'Itri wrote:
> On Nov 18, Zack Weinberg  wrote:
>> as it has proven to be a genuinely critical problem
> No, it has not.

In the previous long thread on debian-devel on this subject, someone posted a 
step-by-step recipe to reproduce a phenomenon where a file that has been moved 
(in its package) from an installation path of /bin (for example) to /usr/bin, 
possibly in conjunction with a change to which package owns it, while /bin is a 
symlink to /usr/bin, will disappear from the file system when the updated set 
of packages is installed.  (I regret I do not have time today to dig up the 
exact email in question.)

Are you seriously claiming that that phenomenon is not a severity:critical bug?

zw

Re: merged-/usr transition: debconf or not?

[Luca Boccassi]

On Thu, 2021-11-18 at 13:09 +0100, Marco d'Itri wrote:
> On Nov 18, Zack Weinberg  wrote:
> 
> > And, as it has proven to be a genuinely critical problem, it is also their
> No, it has not.

Indeed - it seems to me there's a convenient tendency to forget that
this is not something new that has never been seen before, to be
introduced without any testing. This is the default. It has been the
default for multiple releases of multiple distributions. All Ubuntu
installations that were not using this default are now forcibly
converted upon upgrade to 21.10.

And yet nobody has actually seen this happen, to the best of my
knowledge. So either this combination that would allegedly not work
hasn't been used once in 4+ years across multiple distros, or the
required alignment of coincidences to make the issue happen is so rare
that it just never happens in practice.

All we have is a proof of concept. By all means, if anybody cares
enough, go and fix it, no problem with that. Propose an actual, working
alternative that gives the same result too - will happily spend my time
to test it. And put in place QA checks and whatnot to be sure - that
seems a great idea.

But talks of blocking everything and reverting things with a hacky
script that has never actually been used before, in the face of
multiple decisions by the tech committee and no evidence whatsoever of
real-world impact, and despite tens of thousands of actual, real bugs
affecting Debian that don't get even a fraction of the same treatment
(even the Replaces: feature has been affected by unrelated, actual,
reported bugs, and might very well still be, haven't checked) seems to
me a tiny bit hyperbolic and exagerated.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 18, Zack Weinberg  wrote:

> And, as it has proven to be a genuinely critical problem, it is also their
No, it has not.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Bjørn Mork]

Michael Biebl  writes:
> Am 17.11.2021 um 19:57 schrieb Sam Hartman:
>> The question is whether we ever get to a place where people can update
>> files in a package currently installed to /bin/foo and instead install
>> them to /usr/bin/foo.
>> We have a consensus that dpkg bugs make that a bad idea.
>
> Is that really so? If I'm not mistaken, the problematic part was
> moving files from / to /usr and at the same time moving files between
> packages. So doing that at the same time can be problematic. If that
> would affect many packages is another question.
>
> AIUI simply moving files from / to /usr within the same package is not
> problematic.

Won't you then end up with a package which cannot be split for the next
two releases?  Maybe not a big problem, but I think it will be so hard
to keep track of that it's better to not move the files.


Bjørn

Re: merged-/usr transition: debconf or not?

[Richard Laager]

On 11/17/21 8:18 PM, Zack Weinberg wrote:

However, I think it was the responsibility of the developers of usrmerge
to find out how bad that problem actually was, rather than dismissing it.
And, as it has proven to be a genuinely critical problem, it is also their
responsibility to fix it, per the 'no one can be forced to do anything' rule.

At least for this email, I'll stipulate to everything above.

Is this particularly hard to fix, though?

Can't we just do something like the following pseudo-code:



/* At dpkg start... */
usrmerged = FALSE;
if (the_flag_is_set) {
usrmerged = TRUE;
} elif (bin symlinked to /usr/bin) {
usrmerged = TRUE;
/* Update e.g. /bin/foo to /usr/bin/foo in the database. */
update_database();
set_the_flag();
}


/* As dpkg is deciding to write a file path into the database. */
/* e.g. the package is trying to install to /bin/foo, put
 * /usr/bin/foo into the database. */
if (usrmerged) {
path = canonicalize(path);
}



(I've used /bin -> /usr/bin as an example, but the canonicalization must 
deal with the other merged paths too.)


The second bit ensures that all new operations write canonicalized paths 
(e.g. /usr/bin/foo) into the database regardless of whether the .deb 
shipped with /bin/foo or /usr/bin/foo. This ensures the database stays 
in sync with the filesystem moving forward.


The first bit performs a one-time canonicalization of paths in the 
existing database. An on-disk flag ensures that this is run only once 
(for performance reasons only, because it's already idempotent). This 
corrects the existing database-filesystem mismatch.


The one-time canonicalization can be removed once non-usrmerged systems 
are no longer supported. The second bit needs to live (probably a lot) 
longer, until it's no longer reasonable to install a .deb containing 
non-usrmerged paths (which might be old or from a third party).


Am I missing something here?

--
Richard



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

>>>>>> "Sam" == Sam Hartman  wrote:
>>>>>> "Zack" == Zack Weinberg  writes:

Sam> There's a third option.  We sit around in the state where /bin is
Sam> a symlink, but where you cannot move files to /usr paths in the
Sam> package system until the bug is fixed.

Zack> I guess that is a potential outcome.  In a sense we are
Zack> already in that state, given the installed base of systems
Zack> where /bin is already a symlink.
Zack> I don't *like* that outcome; I think it's asking for lots and
Zack> lots of accidental breakage in unstable post-bookworm.

Sam> I don't like that outcome either.
Sam> I think that we have reasonable mitigations for the specific problem you
Sam> describe.
Sam> In particular, we do as I understand it have QA plans to build both with
Sam> merged /usr and without prior to bookworm to catch problems like the
Sam> ones you describe.
Sam> After bookworm releases it's totally fine if programs reference
Sam> filesystem paths like /usr/bin/bash, so long as they don't move where
Sam> things get installed.

I don't think that will mitigate the problem.  I think that post-bookworm,
packages like coreutils and libc6 are going to notice, in their upstream
configure scripts, that /bin etc are symlinks into /usr, and start installing
stuff that used to be in / into /usr.  Maintainers are going to need to take
positive action to _prevent_ the move, and there will no longer be automation
to detect that files have moved.

Sam> I.E. I don't think the transition is going to get canceled; we're
Sam> too far down that path.

Zack> *Are* we?  It seems to me that we could still, at this point,
Zack> [roll back the transition]

Sam> I don't think we would find people to do enough testing to
Sam> validate that was a reasonable thing to do.  But also, the usrmerge
Sam> proponents get most of what they wanted even if we're stuck in the
Sam> middle.

That is exactly why I think a rollback should not be taken off the table:
it is very, very bad if the usrmerge proponents get most of what they want
while the rest of the project is left to clean up their mess.  The project
cannot afford to reward such misbehavior.  I honestly think the social
fallout of allowing that to happen would be *worse* than the social and
technical fallout of forcing a rollback through.

...
Sam> I want to reiterate that the initial process surrounding usrmerge was
Sam> horrible. The dpkg maintainer had what turned out to be legitimate
Sam> concerns that were not reasonably addressed. I think that's in part
Sam> because they were presented poorly and in part because people didn't
Sam> want to hear them. That's a real process failure.
...
Sam> I think people on both sides were unwilling to listen to each other.

I'm an outside observer and I have not followed this argument closely from
the beginning, but my perception of it is much more one-sided than you are
making it out to be.  This is what _I_ saw:

 - usrmerge was deployed to unstable in late 2014.  It's not clear to me
   how much testing it got in 2015.
 - Reports of it causing problems for dpkg started to appear in early 2016
   (e.g. #810499).  These seem to have been addressed civilly, but even then
   a "well it works for _me_ so :shrug:" attitude from the merged-/usr
   proponents is evident in the bug logs.
 - The dpkg maintainer filed a severity:serious bug on usrmerge in late
   2016 (#848622), saying that it breaks dpkg -S.  This is the earliest
   instance I can find of Marco in particular overtly blowing off a bug
   report that he didn't think was significant ("Over one year of
   experience with merged-/usr systems has not exposed any failures",
   which is an invalid argument).  This bug is also significant in
   hindsight because, although not described as such, it appears to me
   to have the same root cause as the file lossage on upgrade that you
   and I, at least, agree is severity:critical.
 - Over the next, um, *five years*, Marco continued to ignore or reject
   Guillem's attempts to get him to take problems seriously that were
   caused by usrmerge rendering the dpkg database inconsistent with the
   file system.
 - Marco also reacted hostilely to concerns raised by others, e.g.
   Adrian Bunk (#863269).

 - Guillem, for his part, has displayed far more patience than I would
   have in his situation, repeatedly attempting to explain why there is
   a serious problem, offering concrete solutions - that they may not be
   practical is not the point; he's done more work toward that end than
   anyone else - and never, that I have seen, losing his temper.
   At the same time, he has said in so many words that this has caused
   him to become burnt out on Debian in general and dpkg maintenance in
   particular.

There's no "both sides were unwilling to listen" when all one side is
bringi

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

>>>>> "Zack" == Zack Weinberg  writes:

Zack> On Wed, Nov 17, 2021, at 1:37 PM, Sam Hartman wrote:
>>>>>>> "Zack" == Zack Weinberg  writes:
Zack> Therefore: either someone fixes the bug, or the transition
Zack> will have to be canceled.  It appears to me that the tech ctte
Zack> agrees with all of this.
>> 
>> There's a third option.  We sit around in the state where /bin is
>> a symlink, but where you cannot move files to /usr paths in the
>> package system until the bug is fixed.

Zack> I guess that is a potential outcome.  In a sense we are
Zack> already in that state, given the installed base of systems
Zack> where /bin is already a symlink.

Zack> I don't *like* that outcome; I think it's asking for lots and
Zack> lots of accidental breakage in unstable post-bookworm, as
Zack> packages are rebuilt on systems that now have /bin a symlink.

I don't like that outcome either.
I think that we have reasonable migigations for the specific problem you
describe.
In particular, we do as I understand it have QA plans to build both with
merged /usr and without prior to bookworm to catch problems like the
ones you describe.
After bookworm releases it's totally fine if programs reference
filesystem paths like /usr/bin/bash, so long as they don't move where
things get installed.

So, while I agree with you getting stuck in the middle is not a good
place, 
I don't think it happens to suffer from the problems you describe.

>> I.E. I don't think the transition is going to get canceled; we're
>> too far down that path.

Zack> *Are* we?  It seems to me that we could still, at this point,
Zack> pull usrmerge from testing and stable, push point updates to
Zack> the installers for all supported releases to flip the default
Zack> back to non-merged /usr, and advise the installed base to run
Zack> dpkg-fsys-usrunmess before their next apt upgrade.

That seems crazy to me: dpkg-fsys-usrunmess has almost certainly
involved less testing than usrmerge.  Talk about not having people to do
the work: I don't think we would find people to do enough testing to
validate that was a reasonable thing to do.  But also, the usrmerge
proponents get most of what they wanted even if we're stuck in the
middle.  If you want a minimal root because you care about containers,
ostrees, and the like, you don't care that much where the package
database thinks files are being installed.

You've been working on this for years and you've gotten most of what you
hoped for.
You're going to fight really hard if someone tries to take that away.
You can easily fight hard enough to make that cost much higher than the
cost of fixing dpkg.
And you can present a reasonably looking political front: hey, we've
been working on this for years, the TC made a decision, the TC later
gave advice, we're basically just done except for this one dpkg bug.
Backing out is insane; just fix dpkg.

I want to reiterate that the initial process surrounding usrmerge was
horrible.
The dpkg maintainer had what turned out to be legitimate concerns that
were not reasonably addressed.
I think that's in part because they were presented poorly and in part
because people didn't want to hear them.
That's a real process failure.
We should learn from that mistake and do better in the future.

We don't have the political will to unwind years of work because we made
that process error, and I don't think it would be a good idea to do so
even if we did.
I do think we should be careful to be better about listening to each
other in the future.
We don't want a community where you can get your way by ignoring
legitimate objections.
I think people on both sides were unwilling to listen to each other.

And yes, if the dpkg maintainer had asked for usrmerge to block on dpkg
gaining support for aliasing, and if that had been done around the time
it was proposed to change the default for debootstrap, we as a project
should have listened.
That's not what happened though.
The dpkg maintainer refused to consider the usrmerge approach that the
usrmerge proponents wanted.
He proposed a different scheme that didn't actually meet the needs of
the usrmerge proponents.
He was unwilling to take the time to understand why his proposal didn't
meet other people's needs.

And yes, the usrmerge proponents (and especially the TC) should have
worked harder to understand the actual underlying objections.


So, there's a huge chunk of ways we could improve to go around--enough
for everyone involved.


But reliving past battles as anythying other than a way to do better in
the future will drag us down.

--Sam


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Wed, Nov 17, 2021, at 1:37 PM, Sam Hartman wrote:
>>>>>> "Zack" == Zack Weinberg  writes:
> Zack> Therefore: either someone fixes the bug,
> Zack> or the transition will have to be canceled.  It appears to me
> Zack> that the tech ctte agrees with all of this.
>
> There's a third option.
> We sit around in the state where /bin is a symlink, but where you cannot
> move files to /usr paths in the package system until the bug is fixed.

I guess that is a potential outcome.  In a sense we are already in that state, 
given the installed base of systems where /bin is already a symlink.

I don't *like* that outcome; I think it's asking for lots and lots of 
accidental breakage in unstable post-bookworm, as packages are rebuilt on 
systems that now have /bin a symlink.  But I can't personally offer to fix 
dpkg, either (way oversubscribed on other projects, and this doesn't seem like 
a job to be tackled by someone new to dpkg, tbh).

> I.E. I don't think the transition is going to get canceled; we're too
> far down that path.

*Are* we?  It seems to me that we could still, at this point, pull usrmerge 
from testing and stable, push point updates to the installers for all supported 
releases to flip the default back to non-merged /usr, and advise the installed 
base to run dpkg-fsys-usrunmess before their next apt upgrade.  It'd be ugly 
but it might well be *less* ugly than being stuck in the state you describe.  I 
understood the tech-ctte to be explicitly holding that option open.

The proponents of merged /usr would be unhappy, but that does not bother me.

zw

Re: merged-/usr transition: debconf or not?

[Michael Biebl]

Am 17.11.2021 um 19:57 schrieb Sam Hartman:

The question is whether we ever get to a place where people can update
files in a package currently installed to /bin/foo and instead install
them to /usr/bin/foo.
We have a consensus that dpkg bugs make that a bad idea.


Is that really so? If I'm not mistaken, the problematic part was moving 
files from / to /usr and at the same time moving files between packages. 
So doing that at the same time can be problematic. If that would affect 
many packages is another question.


AIUI simply moving files from / to /usr within the same package is not 
problematic.


Sam, am I missing something here?

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:

Marco> On Nov 10, Sam Hartman  wrote:
>> I'm sorry, but I think the only way in which that horse is dead
>> is that no one has proposed patches to dpkg.
Marco> Indeed, because the sides of this argument are like three
Marco> people (one of them being the dpkg maintainer) versus
Marco> everybody else.  Since some significant work on dpkg is
Marco> reasonably not forthcoming then this is clearly not a viable
Marco> transition method.

Um, no one still involved in this thread is talking about transition
methods other than what the TC proposed.
We're all talking about transition methods where /bin becomes a symlink
to /usr/bin.

The question is whether we ever get to a place where people can update
files in a package currently installed to /bin/foo and instead install
them to /usr/bin/foo.
We have a consensus that dpkg bugs make that a bad idea.
That consensus is validated by the TC.
It's not clear to me that bug is likely to get fixed in the bookworm +1
time.

I don't think that means we halt the transition: the TC has made a
decision that we're going forward.
I do think it means probably even after the release of bookworm we
cannot rearrange where packages claim files get installed.
That's the horse I think is not dead.


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Zack" == Zack Weinberg  writes:
Zack> The existence of the bug is not in question.  A step-by-step
Zack> reproduction recipe was posted in the previous thread.  Losing
Zack> files from Essential packages when they are upgraded is
Zack> severity:critical

Agreed so far.

Zack> .  Therefore: either someone fixes the bug,
Zack> or the transition will have to be canceled.  It appears to me
Zack> that the tech ctte agrees with all of this.

There's a third option.
We sit around in the state where /bin is a symlink, but where you cannot
move files to /usr paths in the package system until the bug is fixed.

I.E. I don't think the transition is going to get canceled; we're too
far down that path.
I do think that it might be a critical bug to move packages from /bin to
/usr/bin in the packaging system in bookworm+1 if this bug doesn't get
fixed.

In other words, there are people who are motivated to do the work of
creating /bin->/usr/bin symlinks.
There may not be people motivated to make it possible for the later
packaging changes to happen.
That may be the work that doesn't get done until motivated people show
up to do it.


signature.asc
Description: PGP signature