Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
On Fri, 1 Sept 2023 at 09:23, Helmut Grohne wrote: > > Hi Luca, > > At least three DDs have asked you to stop. Why do you continue? > > In all of your mails to this bug, I've seen little attempt at trying to > understand other participants. In a project as large as Debian, it is to > be expected that disagreement arises. That's not the end of the world, > but it is important to disagree respectfully. I'm not seeing that > respect in your interactions. Extraordinary claims require extraordinary evidence, as the saying goes. Asking for such evidence when it's not provided, for example for the claim that Ansible, Puppet, Chef, Docker, Rsync and whatnot no longer work on Debian/Ubuntu/Fedora/Arch/SUSE/CentOS/RHEL/etc, is not disrespectful.
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
Hi Ian, On Wed, 2023-08-30 at 12:52 +0100, Ian Jackson wrote: > Debian has historically been simply much more reliable. Could you quantify this? This is not my experience. As far as I understand you often use non-standard configurations (split-/usr, non-standard init system, ...) which might explain your impression. But non-standard ocnfigurations have been less reliable since forever. > usrmerge is a facet of Debian's culture wars. I would appreciate keeping the discussion to technical parts; could I ask you to keep your concerns about cancel culture elsewhere? > > > This argument is basically drawing together the common Affected > > > tooling includes not just our .debs, but also remote > > > configuration management systems like Ansible, image construction > > > (Dockerfiles), and 3rd-party software installation progrmas > > > (including > > > both 3rd-party .debs and 3rd-party script-based installation > > > systems). > > > > I don't follow the reasoning. [...] > > Sam has posted a helpful set of examples of things that one might do, > whose consequences with aliased directories are unclear. > > These are of course only examples. Tools like Ansible, Dockerfiles, ... have probably already been changed to handle merged-/usr. If they are still horribly broken, please document major issues (there are probably always cases that need a change). I somehow doubt many tools will be broken on many distributions for many years and nobody caring. So please substantiate this claim; it seems wild speculation that tools no longer work on Debian, Ubuntu, Red Hat, SuSE, Arch, Gentoo, ... and that they haven't been changed to work (in so far as this has been neccessary). I see this more as a reason against moving to the proposed Jackson layout with symlink farms: this new layout seems more likely to introduce problems with tooling like Ansible, Docker, ... than an existing layout shared between most(?) larger distributions and already the default for many years. Is there any risk evaluation what happen when moving to symlink farms? > > * Become more precise about what your layout looks like precisely. > > Our > ... > > Let me give some examples to get you started: > > libreoffice -> /bin/python > > ghostscript, imagemagick, mesa -> /bin/env > > bind9, manpages, net-snmp, qtbase-opensource-src -> /bin/perl > > Of these I would hope (by, say, 2033) to only include env. I think "hope" is not a good decision mechanism to produce a high quality operating system. How do you plan to do this? Drop links by throwing a coin, releasing stable, then waiting for user systems to be broken? This has been asked several times, but you refuse to give any answer. > perl upstream doctrine used to be that /usr/bin/perl was the correct > canonical path. I haven't checked if that recommendation still > stands. > > I don't know what Python upstream doctrine is on /bin vs /usr/bin; if > it were that Python is supposed to be in /bin, then we might need to > follow that. (But, are they dropping support for the BSDs?) > > With another hat on in an upstream project I've have been receiving > patches to change #!/bin/bash in CI scripts to #!/usr/bin/env bash. How do you ensure all users follow this scheme required by your proposed symlink farm layout, including for local scripts or scripts taken from systems where they just work? > I think it's worth pointing out that any software which is trying to > be portable to Unix systems other than just Linux (which includes the > BSDs and MacOS) will need to avoid assuming directory aliasing. Which seems irrelevant for what we do in Debian. On portable system you can't assume `/bin/sh` to be there either... Ansgar
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
Hi Luca, At least three DDs have asked you to stop. Why do you continue? In all of your mails to this bug, I've seen little attempt at trying to understand other participants. In a project as large as Debian, it is to be expected that disagreement arises. That's not the end of the world, but it is important to disagree respectfully. I'm not seeing that respect in your interactions. Even if working from the assumption that Ian's request was totally impractical, listening to him can help figure out what issues we cause in moving forward and what use cases we break. That sort of critical feedback is valuable as it points to weaknesses in our currently favoured plan. Past discussions of the matter have clearly alienated other participants. Therefore it is now difficult to receive critical feedback. If you want to continue discussing this matter on this issue, please only do so if you are able to respect the views of others. Helmut
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
On Wed, 30 Aug 2023 at 12:57, Ian Jackson
wrote:
>
> On the burden of proof and the correctness of software:
>
> I'm afraid I see a pattern, where blanket statements are made which
> are only "mostly" or "roughly" or "generally" true. But the
> discrepancies and details matter. When we make computer systems, it's
> not good enough to if they're only "mostly" right.
>
> Every program is an unproven conjecture whose truth we end up relying
> on. We need simple axioms, and rigorously correct reasoning.
Yes, and fact-free and evidence-free threads such as #1050001 go in
the exact opposite direction.
> Aliased usrmerge, when deployed, was a conjecture that was disputed:
> "usrmerge via directory aliasing functions correctly".
And it demonstrably does: it works beautifully and flawlessly on an
uncountable number of Debian, Ubuntu, Mint, Fedora, RHEL, CentOS,
Arch, SUSE, Yocto, Gentoo, etc. installations, for a decade in some of
those cases.
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing [and 3 more
> messages]"):
> > In order to prefer Debian over something else, we want it to be similar
> > enough to make switching feasible while making it differ from others
> > enough to make switching worthwhile. Not having aliasing symlinks hardly
> > seems like an aspect that makes Debian more suitable to people. I guess
> > that you disagree with this and that is fine.
>
> Debian has historically been simply much more reliable. That
> reliability doesn't come from one particular decision. It comes from
> an aggregate of, on the whole, making better decisions. That
> reliability is very valuable to our users and downstreams. It's
> one of our our most compelling advantages.
It comes in the largest part from having reliable upstreams that can
use common, stable, reliable and proven interfaces, such as the
usr-merged filesystem layout which has been the de-facto standard
across the vast majority of Linux distros for the past decade.
> > My impression has always been that the disagreement on the end
> > state was involving a minority.
>
> Well, if you disagree with aliased usrmerge and say you don't want it,
> you get abuse. Even here, many abusive messages have been posted with
> impunity by persons with considerable status.
Sure, insofar as asking for evidence for outrageous and unproven
claims can be considered 'abuse'
> > > This argument is basically drawing together the common factor in the
> > > DEP-17 problem list.
> >
> > And that's precisely why I don't understand your argument. All of the
> > DEP-17 problems are supposed to go away. So it appears as if you cite a
> > list of problems that I expect to not be problems for much longer as a
> > reason for changing the end goal.
>
> DEP-17's list of *problems* forms a category: directory-aliasing-
> induced malfunctions. But the DEP-17 *solutions* are ad-hoc, and many
> are only applicable to to malfunctions that occur during migration.
No, it's a list of potential bugs caused by dpkg being hopelessly
broken, and effective and demonstrably correct temporary workarounds
for them.
> > > Affected tooling includes not just our .debs, but also remote
> > > configuration management systems like Ansible, image construction
> > > (Dockerfiles), and 3rd-party software installation progrmas (including
> > > both 3rd-party .debs and 3rd-party script-based installation systems).
> >
> > I don't follow the reasoning. [...]
>
> Sam has posted a helpful set of examples of things that one might do,
> whose consequences with aliased directories are unclear.
>
> These are of course only examples.
Yes, they are only examples, more specifically evidence-free examples.
It would be trivial to demonstrate that Ansible/Puppet/whatever are
broken and don't work on Debian/Ubuntu/Fedora/RHEL/CentOS/Arch/etc,
but somehow evidence remains scarce. I wonder why.
> > > I would be much more convinced that all of this isn't a problem, if
> > > there existed, and had been formally adopted (eg by existing in some
> > > manual somewhere) a short and clear set of rules about what is and
> > > isn't allowed - not just rules for us within Debian, but rules for
> > > everyone, everywhere, referring to and modifying files.
> >
> > It appears to me that the empty set of rules (outside packaging) is too
> > simple for you to consider.
>
> "Packaging" is doing a lot of work there. I find your comment rather
> tendentious, I'm afraid.
>
> I think we need a short and clear set of rules for what you can do
> with ansible, rsync, docker, etc. etc. etc., as well as just apt and
> dpkg.
Again
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
On the burden of proof and the correctness of software:
I'm afraid I see a pattern, where blanket statements are made which
are only "mostly" or "roughly" or "generally" true. But the
discrepancies and details matter. When we make computer systems, it's
not good enough to if they're only "mostly" right.
Every program is an unproven conjecture whose truth we end up relying
on. We need simple axioms, and rigorously correct reasoning.
Aliased usrmerge, when deployed, was a conjecture that was disputed:
"usrmerge via directory aliasing functions correctly".
In the time since the TC dismissed the objections and ruled in favour
of believing in the truth of this conjecture, has been disproven.
It is false in such important wsys that even in Debian, where we have
every possible advantage, we have been significantly inconvenienced.
Now we have a long list of counterexamples demonstrating the
invalidity of the original thesis.
Nevertheless, in mathematical terms, we are trying to patch up
the conjecture with many qualifications - the mitigations.
Also, we are apparently trying to impose on 3rd parties qualifications
about when it is OK to refer to files by the "wrong" name, which
cannot even be stated clearly, let along succinctly.
Despite all this, apparently when we argue for continued reliance on
this disproven assertion we go back to imprecise statements.
Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing [and 3 more
messages]"):
> In order to prefer Debian over something else, we want it to be similar
> enough to make switching feasible while making it differ from others
> enough to make switching worthwhile. Not having aliasing symlinks hardly
> seems like an aspect that makes Debian more suitable to people. I guess
> that you disagree with this and that is fine.
Debian has historically been simply much more reliable. That
reliability doesn't come from one particular decision. It comes from
an aggregate of, on the whole, making better decisions. That
reliability is very valuable to our users and downstreams. It's
one of our our most compelling advantages.
> My impression has always been that the disagreement on the end
> state was involving a minority.
Well, if you disagree with aliased usrmerge and say you don't want it,
you get abuse. Even here, many abusive messages have been posted with
impunity by persons with considerable status.
usrmerge is a facet of Debian's culture wars. (Debian's culture wars
are mostly orthogonal to the wider world's, but it would be foolish to
ignore the huge social factors at play.) If one is on the
currently-losing side one is not optimistic about the value of making
one's views widely known. Most of my friends agree with me on
substance but think it's a lost cause.
Of course, I could try to "solve" this invisibility by making a blog
post drawing attention and asking random people across the internet to
"contribute" to this discussion (or even just ask them to "me too").
That might even be effective, since currently I'm rather alone here.
I'm not doing that because Debian is quite toxic enough. (And also
because as an apparent minority, seen as a reactionary outgroup, we
would get blamed for the behaviour of extremists.)
> > This argument is basically drawing together the common factor in the
> > DEP-17 problem list.
>
> And that's precisely why I don't understand your argument. All of the
> DEP-17 problems are supposed to go away. So it appears as if you cite a
> list of problems that I expect to not be problems for much longer as a
> reason for changing the end goal.
DEP-17's list of *problems* forms a category: directory-aliasing-
induced malfunctions. But the DEP-17 *solutions* are ad-hoc, and many
are only applicable to to malfunctions that occur during migration.
There is nothing inherently migration-specific about aliasing-induced
malfunction; migration is in this context mostly just a lot of Things
happening, each of which has a chance to go wrong.
The result of the current plan is that all directory-aliasing-induced
malfunctions must be averted, individually, forever. By us, but also
by everyone who has to modify any Debian-derived system.
> > Affected tooling includes not just our .debs, but also remote
> > configuration management systems like Ansible, image construction
> > (Dockerfiles), and 3rd-party software installation progrmas (including
> > both 3rd-party .debs and 3rd-party script-based installation systems).
>
> I don't follow the reasoning. [...]
Sam has posted a helpful set of examples of things that one might do,
whose consequences with aliased directories are unclear.
These are of course only examples.
> > I would be much more convinced that all of this isn't a problem, if
> > there existed, and had been forma
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Helmut Grohne writes: ... > As such, my expectation is that moving from where we are to your idea > is not any easier than moving from a post-DEP-17 state. Therefore, I > do not see any need to delay DEP-17 work. I've been wondering about this possibility too. If a symlink flowerbed is where we decided we wanted to end up, I get the impression that starting from post-DEP-17 would be rather easier than starting from where we are now. DEP-17 provides us with a detailed roadmap for the first leg of that trip, whereas the route via reversion seems to be littered with unknowns at present. Cheers, Phil. -- Philip Hands -- https://hands.com/~phil signature.asc Description: PGP signature
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sun, 27 Aug 2023 at 11:30, Matthew Vernon wrote: > > Dear Luca, > > On 27/08/2023 03:16, Luca Boccassi wrote: > > [things] > > You've already been asked by a couple of people to moderate your tone in > this thread. I appreciate there is a lot of frustration around > /usr-merge, but your contributions are not helping with that at all. Nor > do they help us have a constructive discussion. Constructive discussions about past events need to be based on facts and evidence. Both are conspicuously absent from the bug submitter's messages. Without any of that, there was never any possibility that this bug could be in any way constructive since the moment it was opened, long before I first replied. > The DEP-17 work continues whilst this discussion is ongoing; this > discussion is not delaying that work. > > I think it is fair to say that when ctte decided on /usr-merge it was > expected that a) the process would be reasonably straightforward and b) > dpkg could be taught to understand directory aliasing, so we would not > have to be doing things "behind dpkg's back", as it were. Sure, but then it turned out that package maintainers are not only allowed to ignore the CTTE and its formal decisions, but can also actively block them in their packages. How could responsibility of that be assigned to anybody but those who chose to do that completely eludes me - and yet, that's exactly what the bug submitter does. > I think the need for a releases-long moratorium on moving files and the > volume and depth of technical work represented by DEP17 demonstrate that > those expectations turned out not to be met. > > Given that, it seems to me that a) warnings that "it's not that simple & > easy" were not FUD and describing them thus is unhelpful b) it's > reasonable to at least ask the question "given what we know now, are we > still sure this is the correct approach?" a) except that warnings weren't as mild and meek as "it's not that simple", we were clearly told everything would break left and right, and yet that didn't happen. Sure, dpkg has annoying bugs that need workarounds, but that's hardly surprising given its situation. But these are all packaging-only issues that affect some distribution maintainers for some package builds and maintenance tasks, and that's about it. No installation is affected. No user is affected. No external/third party developer is affected. So, yes, all of that was and still is FUD. b) no, in August 2023 it is very much not reasonable, and no fully informed and evidence-based good faith discussion would ever ask something like that. The reasonable questions to ask in August 2023 are along the lines of "what's the best way to work around those dpkg bugs and lift the moratorium?", which is what we are doing in actual good faith discussions elsewhere, with very productive results. > Any such consideration must be mindful of the fact that the majority of > Debian installs are now /usr-merged, which means that the complexity of > unwinding such installs has to be a heavy factor in thinking about > alternative approaches. > > I'm hopeful, but not certain, that the DEP17 work will get us to a > coherent state again by foxy (which still feels a long time off); I > don't think we should underestimate the work to be done in properly > completing /usr-merge. > > This discussion has, I think, been broadly useful in clarifying some > folks' thinking in this area. I would very much like if we could keep it > focused on the technical matters. I honestly cannot see how there can be any useful technical discussion about a completely evidence-free proposal to move to a symlinks-farm layout in 2023, with all the already well-known wider context that has been explained multiple times. The useful and productive and high-quality technical discussions are happening elsewhere.
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sun, 27 Aug 2023 at 18:03, Sam Hartman wrote: > > > TL;DR: I think I understand one of Ian's points. I explain, but do not > believe it is compelling as an argument to switch direction. > > > "Helmut" == Helmut Grohne writes: > >> I think "package management" is the wrong term here. It's not > >> just our tools and packages that are affected. All forms of > >> operating system management are affected: anything that changes > >> the software, and not just its configuration. > >> > >> Affected tooling includes not just our .debs, but also remote > >> configuration management systems like Ansible, image construction > >> (Dockerfiles), and 3rd-party software installation progrmas > >> (including both 3rd-party .debs and 3rd-party script-based > >> installation systems). > > Helmut> I don't follow the reasoning. Much of the tasks you'd carry > Helmut> out with (wlog) ansible - even when updating files - will > Helmut> continue to work in the aliasing layout. The reason that > Helmut> dpkg is more affected is that it has an inventory of files > Helmut> and reasons about their ownership in terms of > Helmut> packages. That's not how any kind of configuration > Helmut> management operates. If you just "make install" something, > Helmut> chances are that it'll just work with an aliasing layout > Helmut> even when installing with --prefix=/. I continue to argue > Helmut> that the problems we are seeing are quite specific to dpkg > Helmut> in large parts. > > I spent some time with Ian's paragraph you quote above trying to > understand it, and I think I got somewhere. > I considered replying yesterday but decided against, because I think we > are already seeing these effects, and have been seeing them long enough > that we would have a good feel for how serious the problems are. > > I do think that configuration management does have enough of an > inventory of files and reasoning about structure that some of these > problems could arise. I agree that configuration management does not > typically reason in terms of packages. > > But mechanisms like > > * ADD/COPY in Containerfile > > * The rsync module in ansible > * The file module in Ansible > > * various inventories related to modification detection in > configuration management do reason about the filesystem. > > I don't know what would happen if I did > > hosts: lots > remote_user: root > name: Does this shoot me in the foot > tasks: > - rsync: src=install_root dest=/ > > where install_root has a bin directory containing a couple of scripts. > I don't know if rsync will replace a symlink with a directory, or will > just write through the symlink in that situation. > If rsync happens to write through the symlink, there's probably some > other tool somewhere else thatreplaces the symlink. > And when an admin does that, they get an unbootable system, and fix > their playbook/whatever. > > Similarly, I bet someone somewhere has integrity management scripts that > want to look at what pam modules are installed under /lib/security, and > depending on how it worked, > they had to adjust the config when that moved to > /lib/security/x86_64-linux-gnu and then again some of them had to adjust > when /lib became a symlink. And they were probably frustrated during > the time when new installs had /lib as a symlink and upgrades did not. > > Similarly, I bet you can run into trouble with apparmor profiles if you > try to profile /bin/python rather than /usr/bin/python or similar. > > I bet people who had custom selinux policies had to adjust their > labeling rules and again some of them probably found the mixed state > where upgraded systems behaved differently than installed systems > frustrating. AppArmor, SELinux, Ansible, Chef, Puppet and whatnot all exist and are used on all distributions. Such distributions have install bases vastly higher in numbers than Debian, and have adopted the usrmerged layout a decade ago or so. Is there any evidence of AppArmor, SELinux, Ansible, Chef, Puppet and whatnot breaking catastrophically as a consequence? Speculating about what might or might not happen with future events is all good and well, but this is in the past. These things either all broke and suddenly became unusable on Arch, Fedora, CentOS, RHEL, Alma, Rocky, SUSE, Mandriva, Yocto, Ubuntu and so on, or they did not. If there's no evidence they did, then it's not even worth mentioning or discussing, it's all moot. > I seem to recall having to make some minor adjustments at my day job > related to usrmerge back in the buster/bullseye time frame. > I don't remember what they were. This is much more likely to be closer to reality. As with any change or upgrade, there's some minor adjustment here or there, that most won't even remember about after a while given how trivial it is. I certainly do not remember all the changes I've had to do to software I maintain when
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
TL;DR: I think I understand one of Ian's points. I explain, but do not believe it is compelling as an argument to switch direction. > "Helmut" == Helmut Grohne writes: >> I think "package management" is the wrong term here. It's not >> just our tools and packages that are affected. All forms of >> operating system management are affected: anything that changes >> the software, and not just its configuration. >> >> Affected tooling includes not just our .debs, but also remote >> configuration management systems like Ansible, image construction >> (Dockerfiles), and 3rd-party software installation progrmas >> (including both 3rd-party .debs and 3rd-party script-based >> installation systems). Helmut> I don't follow the reasoning. Much of the tasks you'd carry Helmut> out with (wlog) ansible - even when updating files - will Helmut> continue to work in the aliasing layout. The reason that Helmut> dpkg is more affected is that it has an inventory of files Helmut> and reasons about their ownership in terms of Helmut> packages. That's not how any kind of configuration Helmut> management operates. If you just "make install" something, Helmut> chances are that it'll just work with an aliasing layout Helmut> even when installing with --prefix=/. I continue to argue Helmut> that the problems we are seeing are quite specific to dpkg Helmut> in large parts. I spent some time with Ian's paragraph you quote above trying to understand it, and I think I got somewhere. I considered replying yesterday but decided against, because I think we are already seeing these effects, and have been seeing them long enough that we would have a good feel for how serious the problems are. I do think that configuration management does have enough of an inventory of files and reasoning about structure that some of these problems could arise. I agree that configuration management does not typically reason in terms of packages. But mechanisms like * ADD/COPY in Containerfile * The rsync module in ansible * The file module in Ansible * various inventories related to modification detection in configuration management do reason about the filesystem. I don't know what would happen if I did hosts: lots remote_user: root name: Does this shoot me in the foot tasks: - rsync: src=install_root dest=/ where install_root has a bin directory containing a couple of scripts. I don't know if rsync will replace a symlink with a directory, or will just write through the symlink in that situation. If rsync happens to write through the symlink, there's probably some other tool somewhere else thatreplaces the symlink. And when an admin does that, they get an unbootable system, and fix their playbook/whatever. Similarly, I bet someone somewhere has integrity management scripts that want to look at what pam modules are installed under /lib/security, and depending on how it worked, they had to adjust the config when that moved to /lib/security/x86_64-linux-gnu and then again some of them had to adjust when /lib became a symlink. And they were probably frustrated during the time when new installs had /lib as a symlink and upgrades did not. Similarly, I bet you can run into trouble with apparmor profiles if you try to profile /bin/python rather than /usr/bin/python or similar. I bet people who had custom selinux policies had to adjust their labeling rules and again some of them probably found the mixed state where upgraded systems behaved differently than installed systems frustrating. I seem to recall having to make some minor adjustments at my day job related to usrmerge back in the buster/bullseye time frame. I don't remember what they were. I think we've already committed to this pain, and I think we have enough of a window into that commitment that it doesn't seem to be very much pain. I mean spread across all our users, yes people have had to spend hundreds or thousands of hours dealing with this. But that's true with any upgrade. If we move back--if we unwind--things would get much much worse WRT this type of pain for a while. I think many more things would be sensitive to /bin/perl being a symlink than to /bin being a symlink. You could get into situations where /usr/bin/perl and /bin/perl were both files and differed. You could get into situations where /bin/perl vanished on one system. Etc. And if we actually try to have a symlink flower patch rather than a symlink farm, we are left with the pain of where things live differing between distributions and releases in a distribution. Mostly all we have left is the dpkg database. That pain will only affect people producing debs, which isn't just us. And yes, it will dis proportionally affect people who don't have our infrastructure and the ability to catch problems. Perhaps we should think about ways to mitigate that. There will be other smaller pains; if someone else had a system
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Hi Ian,
On Sat, Aug 26, 2023 at 11:24:33AM +0100, Ian Jackson wrote:
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> > On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > > And, the approach being taken very seriously privileges Debian itself,
> > > and those well-staffed derivatives able to do the necessary transition
> > > auditing (albeit, indeed, with tooling from Debian). I am
> > > firmly ideologically opposed to such a tradeoff.
> >
> > I have difficulties disagreeing with that. Getting this done is more
> > important to me though.
>
> I have hoisted this to the start of the mail. I think it is a hugely
> important point.
>
> Debian is not simply a technical project trying to thread its way in a
> complicated world. Debian is an ideological project. At its best,
> Debian is the infrastructrure that enables vast swathes of people to
> massively enhance their own technological autonomy. Many of our most
> controversial decisions served this goal, and stood the test of time.
>
> That's why *I'm* involved in Debian. Our technical choices should
> serve those goals, always.
>
> (To an extent, this divergence in goals may explain why at times our
> comments have been talking slightly past each other.)
I think I understand this and it resonates with me, but there are limits
to that. I don't think that Debian is that technological leader that you
perceive it to be. I hoped that other distributions would adopt the
multiarch directory layout to regain compatibility with Debian and none
did even though there is a clear, technical advantage to doing this.
Debian does not exist in isolation. It is dependent on a lot of
upstreams and in order for that relationship to be healthy, there needs
to be cooperation.
> If you want to think about it on more practical (or even, selfish)
> level, we want Debian to continue to be the preferred choice, when
> someone is choosing an upstream. We didn't get where we are today by
> following bad technical decisions made by others.
In the grand theme of things, the aliasing symlinks may be a suboptimal
technical approach. Please keep in mind though, that this change in
large parts is about people rather than something deeply technical.
After we stopped supporting booting without /usr being mounted in the
initramfs, the split between / and /usr was effectively random and stuff
was moving back and forth every so often and inconsistent between
different distributions or releases. I still remember iptables being an
annoying instance in this regard. So leaving technical aspects aside,
having large parts of the open source community agree on having those
aliasing symlinks already is a significant benefit even if it has
technical downsides.
In order to prefer Debian over something else, we want it to be similar
enough to make switching feasible while making it differ from others
enough to make switching worthwhile. Not having aliasing symlinks hardly
seems like an aspect that makes Debian more suitable to people. I guess
that you disagree with this and that is fine.
> This is indeed a plausible practical reason to do it the aliased way.
> >From my point of view, it amounts to saying "everyone else has made
> this mistake, so to be compatible, we must too".
I wouldn't say it that way, but it comes quite close.
> But I think that seriously underestimates our influence. Debian
> derivatives make up well over half of all Linux installations.
> They're the default basis for most CI images. If we decide this was a
> failed experiment, then indeed there will be some pain for a while,
> but fairly soon people will stop making this assumption.
Quite evidently, we judge this differently. The two of us value the
benefit of the end states differently and the cost to getting to each of
them. Therefore we arrive at different conclusions.
> I don't like the phrase "symlink farm" because it suggests that all,
> most, or even a substantial minority, of files have these symlinks.
> True, at the start, there will be at least a symlink allotment
> but I'm hoping that in the end it'll be a symlink flowerbed.
Let me suggest that this is wishful thinking. It's not only me saying
this, but you can read this from other responses as well. I encourage
you to use codesearch.d.n to see how your flowerbed really is a farm.
Let me give some examples to get you started:
libreoffice -> /bin/python
ghostscript, imagemagick, mesa -> /bin/env
bind9, manpages, net-snmp, qtbase-opensource-src -> /bin/perl
So I see that we either get a symlink farm or we get to include a lot of
Debian-specific patches or we get to argue with a lot of upstreams about
something that may seem entirely pointless to them. In any of these
cases, I consider that a significant cost.
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Hi Matthew, On Sun, 2023-08-27 at 11:30 +0100, Matthew Vernon wrote: > Any such consideration must be mindful of the fact that the majority of > Debian installs are now /usr-merged, which means that the complexity of > unwinding such installs has to be a heavy factor in thinking about > alternative approaches. Yes, I think there are many technical challenges required before Debian would be in a position to adopt the proposed Jackson filesystem layout. If Debian would choose to adopt it at all (an open question). Besides conversion, there is also the telemetry service that seems to be required to safely move to the proposed layout (AFAIK no alternative to this has been proposed yet). I'm not sure if there is already any work done on the path by the proposers? I'm also still missing an overview what the Jackson layout's advantages over the existing filesystem layout (merged-/usr) or the 2000's layout is (split-/usr). As far as I can tell it combines the disadvantages of both with much additional work required to get to it; I don't really see any advantage so far (besides "our tools can't handle anything else", but in practice it seems to work fine, and of course avoiding stuff associated with a certain company which I understand is a goal in itself for some people)... I would appreciate a list of technical and ideological reasons why switching to the Jackson layout is important for Debian. Ansgar
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Dear Luca, On 27/08/2023 03:16, Luca Boccassi wrote: [things] You've already been asked by a couple of people to moderate your tone in this thread. I appreciate there is a lot of frustration around /usr-merge, but your contributions are not helping with that at all. Nor do they help us have a constructive discussion. The DEP-17 work continues whilst this discussion is ongoing; this discussion is not delaying that work. I think it is fair to say that when ctte decided on /usr-merge it was expected that a) the process would be reasonably straightforward and b) dpkg could be taught to understand directory aliasing, so we would not have to be doing things "behind dpkg's back", as it were. I think the need for a releases-long moratorium on moving files and the volume and depth of technical work represented by DEP17 demonstrate that those expectations turned out not to be met. Given that, it seems to me that a) warnings that "it's not that simple & easy" were not FUD and describing them thus is unhelpful b) it's reasonable to at least ask the question "given what we know now, are we still sure this is the correct approach?" Any such consideration must be mindful of the fact that the majority of Debian installs are now /usr-merged, which means that the complexity of unwinding such installs has to be a heavy factor in thinking about alternative approaches. I'm hopeful, but not certain, that the DEP17 work will get us to a coherent state again by foxy (which still feels a long time off); I don't think we should underestimate the work to be done in properly completing /usr-merge. This discussion has, I think, been broadly useful in clarifying some folks' thinking in this area. I would very much like if we could keep it focused on the technical matters. Regards, Matthew
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sat, 26 Aug 2023 at 11:27, Ian Jackson
wrote:
>
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> > On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > > And, the approach being taken very seriously privileges Debian itself,
> > > and those well-staffed derivatives able to do the necessary transition
> > > auditing (albeit, indeed, with tooling from Debian). I am
> > > firmly ideologically opposed to such a tradeoff.
> >
> > I have difficulties disagreeing with that. Getting this done is more
> > important to me though.
>
> I have hoisted this to the start of the mail. I think it is a hugely
> important point.
>
> Debian is not simply a technical project trying to thread its way in a
> complicated world. Debian is an ideological project. At its best,
> Debian is the infrastructrure that enables vast swathes of people to
> massively enhance their own technological autonomy. Many of our most
> controversial decisions served this goal, and stood the test of time.
Exactly, and usr-merge provides exactly that infrastructure, which is
one of the many reasons why it is necessary, as it should be beyond
obvious by now.
> If you want to think about it on more practical (or even, selfish)
> level, we want Debian to continue to be the preferred choice, when
> someone is choosing an upstream. We didn't get where we are today by
> following bad technical decisions made by others.
Very true, and a perfect example of such a "bad technical decision" is
the symlinks-farm layout that is proposed by a couple of people: it
doesn't solve any real problem, and just causes issues for developers
and users, and is purely designed to carefully tip-toe around one
singular hopelessly broken debian-specific tool, dpkg, which suffers
from long-standing bad design decisions that have never been fixed to
this day. It was attempted once, it failed, it was rolled back and
actual usr-merge was successfully deployed.
> > * Basically every other distro uses aliasing now. I expect that
> >a few upstreams have stopped paying attention to systems in your end
> >state. Therefore, they may not only hard code paths in /usr/bin, but
> >also the other way round assume that everything is to be found in /.
>
> This is indeed a plausible practical reason to do it the aliased way.
> >From my point of view, it amounts to saying "everyone else has made
> this mistake, so to be compatible, we must too".
>
> But I think that seriously underestimates our influence. Debian
> derivatives make up well over half of all Linux installations.
> They're the default basis for most CI images. If we decide this was a
> failed experiment, then indeed there will be some pain for a while,
> but fairly soon people will stop making this assumption.
In case you haven't noticed, it's not 1998 anymore. It's 2023, and
Debian is sliding fast into irrelevance, largely because it takes a
decade of fighting off trolls and saboteurs to achieve the most
innocuous of changesl, while most other projects can just implement
obviously correct and needed features such as this one and many others
in a couple of months and then move on to the next. In other words,
other distros innovate while we stagnate. Nobody would truly care if
somehow madness descended on this project and such a grievous act of
self-harm was actually perpetrated, at most some raised eyebrows and
some "they are at it again, aren't they" snarky remarks. Certainly
nobody would move back. For example, I can provide a cast-iron
guarantee that systemd will keep supporting only an usr-merged layout
and not work anywhere else starting from the next version (currently
in main), so there goes ~99.5% of Debian installations.
> > * A key motivation cited for doing the merged-/usr work is called
> >"hermetic /usr". [...] Setting up the aliasing symlinks is easier and
> >less prone to change over time than setting up the symlink farm that
> >you are proposing.
>
> I don't like the phrase "symlink farm" because it suggests that all,
> most, or even a substantial minority, of files have these symlinks.
> True, at the start, there will be at least a symlink allotment
> but I'm hoping that in the end it'll be a symlink flowerbed.
"I'm hoping" doesn't cut it. There is only one example of such an
attempt, in OpenSUSE, and it never materialized. It's been 10 years,
and still the handful of proponents of symlinks-farm have absolutely
nothing to show other than "I'm hoping". What are we even doing here?
> > And then we have a large body of people who simply
> > want this to be over and not having to thing about /usr-merge
> > consequences anymore.
>
&g
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > And, the approach being taken very seriously privileges Debian itself,
> > and those well-staffed derivatives able to do the necessary transition
> > auditing (albeit, indeed, with tooling from Debian). I am
> > firmly ideologically opposed to such a tradeoff.
>
> I have difficulties disagreeing with that. Getting this done is more
> important to me though.
I have hoisted this to the start of the mail. I think it is a hugely
important point.
Debian is not simply a technical project trying to thread its way in a
complicated world. Debian is an ideological project. At its best,
Debian is the infrastructrure that enables vast swathes of people to
massively enhance their own technological autonomy. Many of our most
controversial decisions served this goal, and stood the test of time.
That's why *I'm* involved in Debian. Our technical choices should
serve those goals, always.
(To an extent, this divergence in goals may explain why at times our
comments have been talking slightly past each other.)
If you want to think about it on more practical (or even, selfish)
level, we want Debian to continue to be the preferred choice, when
someone is choosing an upstream. We didn't get where we are today by
following bad technical decisions made by others.
> * Basically every other distro uses aliasing now. I expect that
>a few upstreams have stopped paying attention to systems in your end
>state. Therefore, they may not only hard code paths in /usr/bin, but
>also the other way round assume that everything is to be found in /.
This is indeed a plausible practical reason to do it the aliased way.
>From my point of view, it amounts to saying "everyone else has made
this mistake, so to be compatible, we must too".
But I think that seriously underestimates our influence. Debian
derivatives make up well over half of all Linux installations.
They're the default basis for most CI images. If we decide this was a
failed experiment, then indeed there will be some pain for a while,
but fairly soon people will stop making this assumption.
> * A key motivation cited for doing the merged-/usr work is called
>"hermetic /usr". [...] Setting up the aliasing symlinks is easier and
>less prone to change over time than setting up the symlink farm that
>you are proposing.
I don't like the phrase "symlink farm" because it suggests that all,
most, or even a substantial minority, of files have these symlinks.
True, at the start, there will be at least a symlink allotment
but I'm hoping that in the end it'll be a symlink flowerbed.
> And then we have a large body of people who simply
> want this to be over and not having to thing about /usr-merge
> consequences anymore.
Well, of course it is very tempting to declare the matter as settled
and hope that it will go away. I too want an end state where we will
eventually be able to forget about all of this.
But pushing ahead won't lead to such a state. As I say, I think
people will keep introducing new references to files by their
non-physical names, and we'll keep getting lossage, essentially
forever. (Adopting Simon's terminology.)
Or to put it another way, the delays to completion of this project
have not been due to the political opposition,. They have been
because the project encountered technical problems. Problems whose
existence was predicted by subject matter experts but dismissed at the
time as FUD. Problems which were apparently not regarded as real by the
non-expert decisionmakers on the TC. Problems which still remain in
large part unresolved, albeit in some caes "mitigated".
> > Aliasing is EBW, and "Only use canonical names" is not good enough
> > ==
> >
> > There is basically one underlying technical reason for preferring the
> > un-aliased usrmerge approach: aliasing directories in this way leads
> > to great complication in file management, especially in package
> > management software and in individual packages.
>
> I'm not sure I follow this argument precisely.
This argument is basically drawing together the common factor in the
DEP-17 problem list.
> these complications mostly affect ourselves and
> our package management while end users are mostly unaffected.
I think "package management" is the wrong term here. It's not just
our tools and packages that are affected. All forms of operating
system management are affected: anything that changes the software,
and not just its configuration.
Affected tooling includes not just our .debs, but also remote
configuration management systems like Ansible, image construction
(Dockerfile
Bug#1050001: Unwinding directory aliasing
On Fri, Aug 25, 2023 at 06:50:54AM +0200, Ansgar wrote: > If someone wants to go this way, I suggest to just have a GR about it > instead of iterating this at tech-ctte yet again. It's not very > motivating to have some people endlessly argue against moving forward > and wanting to revisit/reverse/change some decisions endlessly. Ian is a sufficient subject matter expert that I think his concerns are worth analysing. I lean towards disagreeing with his conclusion (I agree that aspects of usrmerge may uncover or result in subtle bugs, but suspect that having Debian diverge from the approach that most other distributions have taken would probably result in a larger number), but I feel it's legitimate to bring this up with the committee. At the moment I think the general feeling seems to be that the concerns aren't sufficient to change the approach we're taking, but when presented with new information it makes sense to take that into account. Please don't interpret this as an attempt to prevent us from moving forward - please think of this as additional verification that we *are* moving forward.
Bug#1050001: Unwinding directory aliasing
Control: retitle -1 migrate from merged-/usr to new alternative filesystem layout On Thu, 2023-08-24 at 13:36 -0600, Sam Hartman wrote: > > > > > > For myself, I think the issues raised in DEP17 are significant enough > that I would at least read a proposal to explain a different way to get > to a merged /usr system. > I.E. yes, I believe that something has changed significantly enough that > I would be willing to read proposals for alternate ways to get to the > end goal we've agreed to. > > However: > > 1) Jackson is not making such a proposal. Yes, we are talking about migrating all installations to a new, alternative filesystem layout now. Which was mentioned years ago already and dismissed (for reasons). The proposal to move to this alternative layout and why one would want to do so hasn't seen any recent discussion before coming to the tech- ctte as far as I know. If someone wants to go this way, I suggest to just have a GR about it instead of iterating this at tech-ctte yet again. It's not very motivating to have some people endlessly argue against moving forward and wanting to revisit/reverse/change some decisions endlessly. Ansgar
Bug#1050001: Unwinding directory aliasing
> "Ansgar" == Ansgar writes: Ansgar> And the more important question: how often do we want to Ansgar> rehash the usrmerge discussion? At some point we should Ansgar> stick with a decision and not endlessly restart discussions Ansgar> (unless something really significant changes, but I don't Ansgar> think this is the case). *mumble* leadership something Ansgar> *mumble* For myself, I think the issues raised in DEP17 are significant enough that I would at least read a proposal to explain a different way to get to a merged /usr system. I.E. yes, I believe that something has changed significantly enough that I would be willing to read proposals for alternate ways to get to the end goal we've agreed to. However: 1) Jackson is not making such a proposal. As it turns out, he has proposed a different end goal, and has not (in my mind) met the extraordinary burden of explaining why we should change goals. He did not even address our current goal and the divergence from that goal. 2) While I would be willing to read such proposals, I am not interested in pausing current efforts unless something really unexpected comes up in such a proposal. Part of building a community is listening to people and engaging with them even when you do not expect to be convinced. I do think there are limits to that. Prior to the discussions that led to DEP-17, I think a message like Jackson's would have been out of bounds. I am disappointed that Jackson did not more clearly articulate that his end state was different than the one we agreed to: for me, that complicates things. If you argue that his message should be out of bounds because he wasn't even trying to get us to the place we had agreed we wanted to reach, I will not object. But I think coming forward and asking what level of proof would be required for an alternate approach that involved unwinding current progress but ultimately got us to merged /usr would be something in good faith at this point. Other than having an entirely different goal in mind, that is what Jackson did. signature.asc Description: PGP signature
Bug#1050001: Unwinding directory aliasing
Hi Simon, On Wed, 2023-08-23 at 20:12 +0100, Simon McVittie wrote: > I'm keen to avoid the anti-pattern where a valid technical point gets > disregarded or even opposed because the way it was expressed puts > other project members on the defensive. [...] > I alluded to that in a previous mail to this thread, but I don't have > first-hand knowledge of the specifics of how that went, and it sounds > as though you might. Do you have references that you can point us to? > (To the list or as private email for me to summarize on-list later, > whichever seems more appropriate. Does it matter? Is it worth investing time into investigating this? Or are we not really considering moving to the proposed Jackson filesystem layout? (In which case it's probably not worth investing time to look what SuSE did in detail.) And the more important question: how often do we want to rehash the usrmerge discussion? At some point we should stick with a decision and not endlessly restart discussions (unless something really significant changes, but I don't think this is the case). *mumble* leadership something *mumble* If we want to restart discussions, we can talk about init system support in Debian and costs/benefits of supporting some of them. :-) Ansgar
Bug#1050001: Unwinding directory aliasing
Hi,
On Wed, 2023-08-23 at 17:04 +0100, Ian Jackson wrote:
> Simon McVittie writes ("Bug#1050001: Unwinding directory aliasing"):
> > What do you consider to be the end goal of this proposal?
>
> Desired end state
> =
>
> This is a very good question. I had a very constructive conversation
> with Helmut via video chat. It seems that there's a misunderstanding
> about the desired end state.
>
> My idea of a desired end state is as follows:
>
> /bin and /lib etc. remain directories (so there is no aliasing). All
> actual files are shipped in /usr. / contains compatibility symlinks
> pointing into /usr, for those files/APIs/programs where this is
> needed
> (which is far from all of them). Eventualloy, over time, the set of
> compatibility links is reduced to a mere handful.
Cool, so we need to touch all packages shipping packages in /usr/bin to
also include a /bin symlink? Like /bin/python3 -> /usr/bin/python3?
(And yes, users use these, so they are necessary unless you want to
break working systems; if so please provide good reasons).
How do you decide when to remove a link? Is there a simple mechanism to
detect when users no longer use it?
> I think this is a more desirable situation than the current planned
> end state, which is that /bin and /lib are symlinks.
Why should we invent again another, incompatible filesystem layout?
Is there any good technical reason to do so? One that was not discussed
already?
Sorry, I feel like we are going back ignoring any discussion in the
last years and would invite you to read what was discussed about these
approaches in previous discussions first. So far it looks like NIH
syndrome, similar to the need to invent yet another daemon readiness
protocol when systemd was the topic.
> Looking towards the future
> --
>
> It seems to me that directory aliasing will continue to be a source
> of very annoying bugs indefinitely, well after the transition is
> fully complete. In another 20 years we'll still be debugging strange
> installation breakage that will turn out to be due to directory
> aliasing.
But introducing an incompatible filesystem where we only detect that
something references files in the wrong path (as presumably only a
random subset will be in /bin) will not give any bugs?
We already *had* these bugs for several releases and found them only
after release (even before usrmerge which makes them non-bugs).
Please provide a plan how to fix these ahead of time; please be aware
that they might only occur with some hardware / configurations.
Ansgar
Bug#1050001: Unwinding directory aliasing
Hi Ian, On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote: > Desired end state > = > > This is a very good question. I had a very constructive conversation > with Helmut via video chat. It seems that there's a misunderstanding > about the desired end state. I concur. Thank you for taking the time to write this down. It now makes a lot more sense to me. > My idea of a desired end state is as follows: > > /bin and /lib etc. remain directories (so there is no aliasing). All > actual files are shipped in /usr. / contains compatibility symlinks > pointing into /usr, for those files/APIs/programs where this is needed > (which is far from all of them). Eventualloy, over time, the set of > compatibility links is reduced to a mere handful. This end state vaguely makes sense to me in principle. I think it does have a two noteworthy limitations though. * Basically every other distro uses aliasing now. I expect that a few upstreams have stopped paying attention to systems in your end state. Therefore, they may not only hard code paths in /usr/bin, but also the other way round assume that everything is to be found in /. I see that we already use #!/bin/python3 e.g. in https://sources.debian.org/src/uchardet/0.0.7-1/script/langs/sv.py/?hl=1#L1 and expect more of that in future. * A key motivation cited for doing the merged-/usr work is called "hermetic /usr". I'm not a hermetic /usr expert, but roughly speaking the idea is that the entire OS actually lives below /usr and everything else is either the user's data or constructed in a straight forward way. Setting up the aliasing symlinks is easier and less prone to change over time than setting up the symlink farm that you are proposing. > I think this is a more desirable situation than the current planned > end state, which is that /bin and /lib are symlinks. What situation is desirable is dependent on whom you ask. We also have a number of people who'd say that the aliasing approach is more desirable than your approach. And then we have a large body of people who simply want this to be over and not having to thing about /usr-merge consequences anymore. > Aliasing is EBW, and "Only use canonical names" is not good enough > == > > There is basically one underlying technical reason for preferring the > un-aliased usrmerge approach: aliasing directories in this way leads > to great complication in file management, especially in package > management software and in individual packages. I'm not sure I follow this argument precisely. Ubuntu has shipped this aliasing approach since quite a while including LTS releases. Debian has defaulted to it in two stable releases. While there are complications, my impression is that these complications mostly affect ourselves and our package management while end users are mostly unaffected. Then once packages have moved their files to canonical locations (as is proposed in DEP-17), are there that much complications that remain? > The DEP-17 problem list is a survey of the aliasing-induced problems > which have been discovered so far. But we (still!) keep discovering > new ones. Indeed, my main worry about this approach is discovering new classes of problems. (And effects on downstreams and derivatives as you also point out.) > The current plan, as I understand it, is that we will fix these > problems by arranging to *always* name files by their canonical paths, > ie the ones in /usr. I confirm. > Naming files by their canonical names will have to be done everywhere. > This is because any time a file is named by a non-canonical path, a > program that tries to manipulate that file might malfunction. > (Whether it malfunctions in practice depends on the precise details > and gets very complicated.) This seems overgeneralized to me. If you look into the list of known problems from DEP-17, it's not like any program would be affected. It is more like most of the problems affect dpkg. Also keep in mind that while we want all files to be named canonically (according to DEP-17), the problems we may see tend to scale with the number of exceptions to the rule. So if we canonicalize most files, chances are that most users don't experience symptoms anymore. > Spotting and mitigating violations is hard > -- > > We do not currently have good tooling that will spot violations of > this rule. It's not clear precisley what the right behaviour of our > tools would be; we need to alert *the right set of users* to the > mistakes, and *with the right level of severity*. Many of our key > tools don't have a good way to produce "critical warnings". The > consequences of violations are unpredicatable and can depend on event > ordering. But they can be very severe. So we are creating a source > of bad heisenbugs. We already have the Debian Usr Merge Analysis Tool
Bug#1050001: Unwinding directory aliasing
On Wed, 23 Aug 2023 at 17:04:36 +0100, Ian Jackson wrote:
> Simon McVittie writes ("Bug#1050001: Unwinding directory aliasing"):
> > What do you consider to be the end goal of this proposal?
>
> My idea of a desired end state is as follows:
>
> /bin and /lib etc. remain directories (so there is no aliasing). All
> actual files are shipped in /usr. / contains compatibility symlinks
> pointing into /usr, for those files/APIs/programs where this is needed
> (which is far from all of them). Eventualloy, over time, the set of
> compatibility links is reduced to a mere handful.
This is not merged-/usr with the meaning used by the technical committee's
past resolutions, and by most (all?) non-Debian distributions (among
which Fedora and Arch were among prominent early adopters).
I recognise that you don't want merged-/usr, and instead you want
this non-merged-/usr layout, which shares some of the properties of
merged-/usr; but it isn't the same thing, and it makes discussion
and reasoning unnecessarily difficult if we use the same name for two
different things, so please could you avoid the term "merged /usr"
for this?
> I think this is a more desirable situation than the current planned
> end state, which is that /bin and /lib are symlinks.
The meaning ascribed to "merged /usr" or "the /usr merge" by previous
TC resolutions is exactly the layout where /bin and /lib (and so on)
are symlinks. Outside Debian, that's also the layout described in
documents like "The Case for the /usr Merge".
I acknowledge that, whatever we choose to call it, you would prefer not
to end at that state, and this is a point on which our opinions differ.
> The current plan, as I understand it, is that we will fix these
> problems by arranging to *always* name files by their canonical paths,
> ie the ones in /usr.
Using the word "canonical" is not necessarily helpful here, because
there are two reasonable-but-contradictory things you might mean by
it. Depending who you ask, the canonical path of /bin/sh on a system
with some sort of unified /usr might be:
* /usr/bin/sh, because that's the physical path as returned by realpath()
(I believe this is what you mean when you say "canonical", because
you're thinking in terms of the path canonicalization operation done
by realpath() and similar things);
* or /bin/sh, because that's the interoperable path that has worked on
all Linux distributions since time immemorial (even though this might
not have anything to do with the physical path)
May I suggest we avoid saying "canonical" as ambiguous, and use something
like "physical path" and "traditional path" for these two concepts?
If by "name files" you mean references to filenames from elsewhere, then
that is not the plan as I understand it (see below).
If by "name files" you mean "name files in dpkg's database", then yes,
I believe the current plan is that we end up with dpkg's idea of the list
of installed files referring to every file by its physical path, so that
for example the dpkg database contains the physical path /usr/bin/sh,
even though the traditional path is /bin/sh. Another way to express
this is that if you install a Debian chroot and pack it into an archive
in the obvious way, what you should get (in the absence of any uses of
dpkg-divert, etc.) is the union of the data.tar of all the installed
packages, plus additional non-dpkg-managed files like /etc/passwd.
> Naming files by their canonical names will have to be done everywhere.
I would dispute that. We routinely name system-critical files by
non-physical paths - for example /bin/sh is really /{usr/,}bin/dash,
and /lib64/ld-linux-x86-64.so.2 is really
/{usr/,}lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 - and have done so
for a long time.
> violations of the "use only [physical] names" rule are not only
> expected, they are *necessary*:
Right, and that's why that is not a rule we are following.
One of the reasons that merged-/usr appeals to me personally is
that it takes an entire equivalence class of bugs, and turns them
into non-bugs. If merged-/usr is ubiquitous, then we don't need to
expect third-party software developers to "just know" that /bin/sh and
/usr/bin/perl are the traditionally "correct" paths, because /usr/bin/sh
and /bin/perl become equally valid and interoperable things to put at
the beginning of a script.
In the state we had before bookworm, where merged-/usr was supported but
not mandatory, we required Debian maintainers to be careful to refer to
files by their traditional name, even though on a newly-installed Debian
system with merged-/usr, the "other" name would have worked equally well;
and we were also implicitly expecting upstream and third-party developers
to also know that they had to use t
Bug#1050001: Unwinding directory aliasing
Also, one other thing I noticed:
tl;dr: *no* version of usrmerge relieves us of the obligation of
naming files correctly, via the proper name in /usr rather than /.
Ian Jackson writes ("Bug#1050001: Unwinding directory aliasing"):
> The current plan, as I understand it, is that we will fix these
> problems by arranging to *always* name files by their canonical paths,
> ie the ones in /usr.
>
> Naming files by their canonical names will have to be done everywhere.
> This is because any time a file is named by a non-canonical path, a
> program that tries to manipulate that file might malfunction.
> (Whether it malfunctions in practice depends on the precise details
> and gets very complicated.)
...
But Simon writes:
> > This does some but not all of what merged-/usr does: calling /usr/bin/sh
> > would become a non-bug, but calling /bin/env would still be an error,
> > /bin would still represent non-trivial on-disk and/or in-dpkg-database
> > state,
This suggests that a goal of the project is to "make it not be a bug
to refer to a file in /usr/bin by its name in /bin".
However, in the aliased-usrmerge such an incorrect reference *remains*
a bug (unless it can be demonstrated that the reference is only
read-only and no-one will take that path and use it in a non-read-only
context).
It's just that the consequences of the bug are different.
Without usrmerge, or with un-aliased-usrmerge for a file where there
is no compat symlink, the reseult is a "file not found" error. No
references via that path can work. (Except maybe transitionally,
while the file is moving.) Such a bug is not likely to survive long.
With un-aliased usrmerge for files which still have compat symlinks
(eventually, a handful of files considered quite special), using the
reference for mutation might result in breaking the symbolic link; or
it might result in errors from filename lookups in package management
databasesa, where the filename would be not recorded in / or not
recorded as a file. Broken symlinks could be detected post-hoc on the
affected system, and it could be detected simply and automatically by
QA tooling. But, this is a good reason to try to reduce the number of
compat symlinks to a very small number.
With aliased usrmerge, for all files forever, using the path in /
might result in confusing behaviour by package management and system
administration tooling. Reasoning about the consequences is
difficult, but in the worst case it might render the affected
subsystem totally broken.
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1050001: Unwinding directory aliasing
Simon McVittie writes ("Bug#1050001: Unwinding directory aliasing"):
> What do you consider to be the end goal of this proposal?
Desired end state
=
This is a very good question. I had a very constructive conversation
with Helmut via video chat. It seems that there's a misunderstanding
about the desired end state.
My idea of a desired end state is as follows:
/bin and /lib etc. remain directories (so there is no aliasing). All
actual files are shipped in /usr. / contains compatibility symlinks
pointing into /usr, for those files/APIs/programs where this is needed
(which is far from all of them). Eventualloy, over time, the set of
compatibility links is reduced to a mere handful.
I think this is a more desirable situation than the current planned
end state, which is that /bin and /lib are symlinks.
Aliasing is EBW, and "Only use canonical names" is not good enough
==
There is basically one underlying technical reason for preferring the
un-aliased usrmerge approach: aliasing directories in this way leads
to great complication in file management, especially in package
management software and in individual packages.
The DEP-17 problem list is a survey of the aliasing-induced problems
which have been discovered so far. But we (still!) keep discovering
new ones.
The current plan, as I understand it, is that we will fix these
problems by arranging to *always* name files by their canonical paths,
ie the ones in /usr.
Naming files by their canonical names will have to be done everywhere.
This is because any time a file is named by a non-canonical path, a
program that tries to manipulate that file might malfunction.
(Whether it malfunctions in practice depends on the precise details
and gets very complicated.)
Spotting and mitigating violations is hard
--
We do not currently have good tooling that will spot violations of
this rule. It's not clear precisley what the right behaviour of our
tools would be; we need to alert *the right set of users* to the
mistakes, and *with the right level of severity*. Many of our key
tools don't have a good way to produce "critical warnings". The
consequences of violations are unpredicatable and can depend on event
ordering. But they can be very severe. So we are creating a source
of bad heisenbugs.
Also, we only have direct control over the behaviour of our own
packages, images, etc. that we (Debian) ship. Any time anyone in the
field (perhaps an invididual sysadmin or user; perhaps a 3rd party
software supplier; perhaps a downstream distro) violates this rule
(whether through ignorance, or choice), affected systems will
malfunction. (I think this means that relying on lintian, for
example, as a defence against these mistakes, is not good enough.)
The answer implied by the current plan seems to be that these people
are just doing the wrong thing and will have learn not to? But the
very existence of the directory symlinks implies a recognition that
confusion over whether to name files in / or in /usr is expected to
continue for a long time. If it weren't, then there would be no need
for these symlinks.
Violations of the "only use canonical names" rule are required
--
Worse, violations of the "use only canonical names" rule are not only
expected, they are *necessary*:
There are quite a few places where we will have to keep naming files
by their names in /, becaue those things appear in highly stsble
public APIs/ABIs. For example, we must ship binaries that refer to
the dynamic loader in /lib; shell scripts must start #!/bin/sh.
Now, those references are almost all in "immutable" contexts, where it
doesn't actually matter, since the file is in fact available by the
non-canonical name. However, this introduces a new implied rule:
it becomes a bug to take a filename you see in a place where the file
is being *read*, and apply it in a context where the file is going to
be *updated*.
This reuse of a filename is a very natural approach. It is something
that is frequently done by humans, but it is also sometimes doen by
automatic software of many kinds. It's not something we've even had
to consider before as a thing. But now it is (sometimes) wrong.
Usually it will work, but sometimes it will make a (perhaps latent or
unpredictable) bug.
Looking towards the future
--
It seems to me that directory aliasing will continue to be a source of
very annoying bugs indefinitely, well after the transition is fully
complete. In another 20 years we'll still be debugging strange
installation breakage that will turn out to be due to directory
aliasing.
I don't doubt that the bug rate will kept "tolerably low" by QA
efforts. However, we all know what a "tolerably low" bug rate loo
Bug#1050001: Unwinding directory aliasing
On Fri, 18 Aug 2023 at 07:57:14 +0100, Ian Jackson wrote:
> I think we can probably fix it by backing out this change, and then
> doing usrmerge the traditional Debian way by making changes to
> debhelper, so that we move the files package by package, in the .debs.
What do you consider to be the end goal of this proposal?
If I understand your proposal correctly, it is to return all Debian
systems to their traditional (let's say circa 2010) layout, and then
restart the transition differently. Imagine for a moment that the
usrmerge package, debootstrap --merged-usr, etc. had never existed,
and all Debian systems were in their 2010 state (or equivalently, that
your proposed revert has happened and we are now ready for the second
stage of your plan).
If we carry out this transition package-by-package without central
coordination ("the traditional Debian way"), it seems to me that the
best we can achieve is for /bin, /sbin, /lib* to be symlink farms,
consisting of symlinks that are either owned by the same package that
owns the symlink target, or are unowned from dpkg's perspective and are
created by maintainer scripts.
Here's a simplified view of the resulting system:
/bin/sh -> /usr/bin/sh
/bin/rm -> /usr/bin/rm
/usr/bin/env (regular file)
/usr/bin/sh (regular file)
/usr/bin/perl (regular file)
/usr/bin/rm (regular file)
(Obviously in real life, /bin and /usr/bin are a lot larger than that,
and /sbin and /lib* are also necessary for a bootable system, but
considering a small number of /{usr/,}bin files is enough to illustrate
my point.)
This does some but not all of what merged-/usr does: calling /usr/bin/sh
would become a non-bug, but calling /bin/env would still be an error,
/bin would still represent non-trivial on-disk and/or in-dpkg-database
state, and we would still potentially have other issues triggered by
the directories being distinct from one another (like the one discussed
by the tech committee in #911225, which was exactly a regression caused
by having moved a library in the traditional Debian way).
The merged-/usr version of that same simplified system is this:
/bin -> usr/bin
/usr/bin/env (regular file)
/usr/bin/sh (regular file)
/usr/bin/perl (regular file)
/usr/bin/rm (regular file)
(Again, in real life we also have to consider /sbin and /lib*, but
showing /bin is enough to illustrate the difference.)
As was discussed in 2019 and repeatedly since then, some of the reasons
to want merged-/usr are only available with that layout, and are not
provided by the symlink farm.
If I remember correctly, openSUSE tried to get from unmerged /usr to
merged /usr by essentially the route you propose, successfully reached
the symlink-farm state, but then got stuck without a way to get from the
symlink farm to the single symbolic link. Do you have a plan for how that
would be achieved without breaking upgrades or going behind dpkg's back?
If your plan was a longer, slower, arguably more robust way to achieve
the same end goal, that would be more appealing than a longer, slower,
arguably more robust way to get halfway to the same end goal and then
become unable to go further.
smcv
