Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
On Fri, 1 Sept 2023 at 09:23, Helmut Grohne wrote: > > Hi Luca, > > At least three DDs have asked you to stop. Why do you continue? > > In all of your mails to this bug, I've seen little attempt at trying to > understand other participants. In a project as large as Debian, it is to > be expected that disagreement arises. That's not the end of the world, > but it is important to disagree respectfully. I'm not seeing that > respect in your interactions. Extraordinary claims require extraordinary evidence, as the saying goes. Asking for such evidence when it's not provided, for example for the claim that Ansible, Puppet, Chef, Docker, Rsync and whatnot no longer work on Debian/Ubuntu/Fedora/Arch/SUSE/CentOS/RHEL/etc, is not disrespectful.
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
Hi Ian, On Wed, 2023-08-30 at 12:52 +0100, Ian Jackson wrote: > Debian has historically been simply much more reliable. Could you quantify this? This is not my experience. As far as I understand you often use non-standard configurations (split-/usr, non-standard init system, ...) which might explain your impression. But non-standard ocnfigurations have been less reliable since forever. > usrmerge is a facet of Debian's culture wars. I would appreciate keeping the discussion to technical parts; could I ask you to keep your concerns about cancel culture elsewhere? > > > This argument is basically drawing together the common Affected > > > tooling includes not just our .debs, but also remote > > > configuration management systems like Ansible, image construction > > > (Dockerfiles), and 3rd-party software installation progrmas > > > (including > > > both 3rd-party .debs and 3rd-party script-based installation > > > systems). > > > > I don't follow the reasoning. [...] > > Sam has posted a helpful set of examples of things that one might do, > whose consequences with aliased directories are unclear. > > These are of course only examples. Tools like Ansible, Dockerfiles, ... have probably already been changed to handle merged-/usr. If they are still horribly broken, please document major issues (there are probably always cases that need a change). I somehow doubt many tools will be broken on many distributions for many years and nobody caring. So please substantiate this claim; it seems wild speculation that tools no longer work on Debian, Ubuntu, Red Hat, SuSE, Arch, Gentoo, ... and that they haven't been changed to work (in so far as this has been neccessary). I see this more as a reason against moving to the proposed Jackson layout with symlink farms: this new layout seems more likely to introduce problems with tooling like Ansible, Docker, ... than an existing layout shared between most(?) larger distributions and already the default for many years. Is there any risk evaluation what happen when moving to symlink farms? > > * Become more precise about what your layout looks like precisely. > > Our > ... > > Let me give some examples to get you started: > > libreoffice -> /bin/python > > ghostscript, imagemagick, mesa -> /bin/env > > bind9, manpages, net-snmp, qtbase-opensource-src -> /bin/perl > > Of these I would hope (by, say, 2033) to only include env. I think "hope" is not a good decision mechanism to produce a high quality operating system. How do you plan to do this? Drop links by throwing a coin, releasing stable, then waiting for user systems to be broken? This has been asked several times, but you refuse to give any answer. > perl upstream doctrine used to be that /usr/bin/perl was the correct > canonical path. I haven't checked if that recommendation still > stands. > > I don't know what Python upstream doctrine is on /bin vs /usr/bin; if > it were that Python is supposed to be in /bin, then we might need to > follow that. (But, are they dropping support for the BSDs?) > > With another hat on in an upstream project I've have been receiving > patches to change #!/bin/bash in CI scripts to #!/usr/bin/env bash. How do you ensure all users follow this scheme required by your proposed symlink farm layout, including for local scripts or scripts taken from systems where they just work? > I think it's worth pointing out that any software which is trying to > be portable to Unix systems other than just Linux (which includes the > BSDs and MacOS) will need to avoid assuming directory aliasing. Which seems irrelevant for what we do in Debian. On portable system you can't assume `/bin/sh` to be there either... Ansgar
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
Hi Luca, At least three DDs have asked you to stop. Why do you continue? In all of your mails to this bug, I've seen little attempt at trying to understand other participants. In a project as large as Debian, it is to be expected that disagreement arises. That's not the end of the world, but it is important to disagree respectfully. I'm not seeing that respect in your interactions. Even if working from the assumption that Ian's request was totally impractical, listening to him can help figure out what issues we cause in moving forward and what use cases we break. That sort of critical feedback is valuable as it points to weaknesses in our currently favoured plan. Past discussions of the matter have clearly alienated other participants. Therefore it is now difficult to receive critical feedback. If you want to continue discussing this matter on this issue, please only do so if you are able to respect the views of others. Helmut
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
On Wed, 30 Aug 2023 at 12:57, Ian Jackson
wrote:
>
> On the burden of proof and the correctness of software:
>
> I'm afraid I see a pattern, where blanket statements are made which
> are only "mostly" or "roughly" or "generally" true. But the
> discrepancies and details matter. When we make computer systems, it's
> not good enough to if they're only "mostly" right.
>
> Every program is an unproven conjecture whose truth we end up relying
> on. We need simple axioms, and rigorously correct reasoning.
Yes, and fact-free and evidence-free threads such as #1050001 go in
the exact opposite direction.
> Aliased usrmerge, when deployed, was a conjecture that was disputed:
> "usrmerge via directory aliasing functions correctly".
And it demonstrably does: it works beautifully and flawlessly on an
uncountable number of Debian, Ubuntu, Mint, Fedora, RHEL, CentOS,
Arch, SUSE, Yocto, Gentoo, etc. installations, for a decade in some of
those cases.
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing [and 3 more
> messages]"):
> > In order to prefer Debian over something else, we want it to be similar
> > enough to make switching feasible while making it differ from others
> > enough to make switching worthwhile. Not having aliasing symlinks hardly
> > seems like an aspect that makes Debian more suitable to people. I guess
> > that you disagree with this and that is fine.
>
> Debian has historically been simply much more reliable. That
> reliability doesn't come from one particular decision. It comes from
> an aggregate of, on the whole, making better decisions. That
> reliability is very valuable to our users and downstreams. It's
> one of our our most compelling advantages.
It comes in the largest part from having reliable upstreams that can
use common, stable, reliable and proven interfaces, such as the
usr-merged filesystem layout which has been the de-facto standard
across the vast majority of Linux distros for the past decade.
> > My impression has always been that the disagreement on the end
> > state was involving a minority.
>
> Well, if you disagree with aliased usrmerge and say you don't want it,
> you get abuse. Even here, many abusive messages have been posted with
> impunity by persons with considerable status.
Sure, insofar as asking for evidence for outrageous and unproven
claims can be considered 'abuse'
> > > This argument is basically drawing together the common factor in the
> > > DEP-17 problem list.
> >
> > And that's precisely why I don't understand your argument. All of the
> > DEP-17 problems are supposed to go away. So it appears as if you cite a
> > list of problems that I expect to not be problems for much longer as a
> > reason for changing the end goal.
>
> DEP-17's list of *problems* forms a category: directory-aliasing-
> induced malfunctions. But the DEP-17 *solutions* are ad-hoc, and many
> are only applicable to to malfunctions that occur during migration.
No, it's a list of potential bugs caused by dpkg being hopelessly
broken, and effective and demonstrably correct temporary workarounds
for them.
> > > Affected tooling includes not just our .debs, but also remote
> > > configuration management systems like Ansible, image construction
> > > (Dockerfiles), and 3rd-party software installation progrmas (including
> > > both 3rd-party .debs and 3rd-party script-based installation systems).
> >
> > I don't follow the reasoning. [...]
>
> Sam has posted a helpful set of examples of things that one might do,
> whose consequences with aliased directories are unclear.
>
> These are of course only examples.
Yes, they are only examples, more specifically evidence-free examples.
It would be trivial to demonstrate that Ansible/Puppet/whatever are
broken and don't work on Debian/Ubuntu/Fedora/RHEL/CentOS/Arch/etc,
but somehow evidence remains scarce. I wonder why.
> > > I would be much more convinced that all of this isn't a problem, if
> > > there existed, and had been formally adopted (eg by existing in some
> > > manual somewhere) a short and clear set of rules about what is and
> > > isn't allowed - not just rules for us within Debian, but rules for
> > > everyone, everywhere, referring to and modifying files.
> >
> > It appears to me that the empty set of rules (outside packaging) is too
> > simple for you to consider.
>
> "Packaging" is doing a lot of work there. I find your comment rather
> tendentious, I'm afraid.
>
> I think we need a short and clear set of rules for what you can do
> with ansible, rsync, docker, etc. etc. etc., as well as just apt and
> dpkg.
Again
Bug#1050001: Unwinding directory aliasing [and 3 more messages] [and 1 more messages]
On the burden of proof and the correctness of software:
I'm afraid I see a pattern, where blanket statements are made which
are only "mostly" or "roughly" or "generally" true. But the
discrepancies and details matter. When we make computer systems, it's
not good enough to if they're only "mostly" right.
Every program is an unproven conjecture whose truth we end up relying
on. We need simple axioms, and rigorously correct reasoning.
Aliased usrmerge, when deployed, was a conjecture that was disputed:
"usrmerge via directory aliasing functions correctly".
In the time since the TC dismissed the objections and ruled in favour
of believing in the truth of this conjecture, has been disproven.
It is false in such important wsys that even in Debian, where we have
every possible advantage, we have been significantly inconvenienced.
Now we have a long list of counterexamples demonstrating the
invalidity of the original thesis.
Nevertheless, in mathematical terms, we are trying to patch up
the conjecture with many qualifications - the mitigations.
Also, we are apparently trying to impose on 3rd parties qualifications
about when it is OK to refer to files by the "wrong" name, which
cannot even be stated clearly, let along succinctly.
Despite all this, apparently when we argue for continued reliance on
this disproven assertion we go back to imprecise statements.
Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing [and 3 more
messages]"):
> In order to prefer Debian over something else, we want it to be similar
> enough to make switching feasible while making it differ from others
> enough to make switching worthwhile. Not having aliasing symlinks hardly
> seems like an aspect that makes Debian more suitable to people. I guess
> that you disagree with this and that is fine.
Debian has historically been simply much more reliable. That
reliability doesn't come from one particular decision. It comes from
an aggregate of, on the whole, making better decisions. That
reliability is very valuable to our users and downstreams. It's
one of our our most compelling advantages.
> My impression has always been that the disagreement on the end
> state was involving a minority.
Well, if you disagree with aliased usrmerge and say you don't want it,
you get abuse. Even here, many abusive messages have been posted with
impunity by persons with considerable status.
usrmerge is a facet of Debian's culture wars. (Debian's culture wars
are mostly orthogonal to the wider world's, but it would be foolish to
ignore the huge social factors at play.) If one is on the
currently-losing side one is not optimistic about the value of making
one's views widely known. Most of my friends agree with me on
substance but think it's a lost cause.
Of course, I could try to "solve" this invisibility by making a blog
post drawing attention and asking random people across the internet to
"contribute" to this discussion (or even just ask them to "me too").
That might even be effective, since currently I'm rather alone here.
I'm not doing that because Debian is quite toxic enough. (And also
because as an apparent minority, seen as a reactionary outgroup, we
would get blamed for the behaviour of extremists.)
> > This argument is basically drawing together the common factor in the
> > DEP-17 problem list.
>
> And that's precisely why I don't understand your argument. All of the
> DEP-17 problems are supposed to go away. So it appears as if you cite a
> list of problems that I expect to not be problems for much longer as a
> reason for changing the end goal.
DEP-17's list of *problems* forms a category: directory-aliasing-
induced malfunctions. But the DEP-17 *solutions* are ad-hoc, and many
are only applicable to to malfunctions that occur during migration.
There is nothing inherently migration-specific about aliasing-induced
malfunction; migration is in this context mostly just a lot of Things
happening, each of which has a chance to go wrong.
The result of the current plan is that all directory-aliasing-induced
malfunctions must be averted, individually, forever. By us, but also
by everyone who has to modify any Debian-derived system.
> > Affected tooling includes not just our .debs, but also remote
> > configuration management systems like Ansible, image construction
> > (Dockerfiles), and 3rd-party software installation progrmas (including
> > both 3rd-party .debs and 3rd-party script-based installation systems).
>
> I don't follow the reasoning. [...]
Sam has posted a helpful set of examples of things that one might do,
whose consequences with aliased directories are unclear.
These are of course only examples.
> > I would be much more convinced that all of this isn't a problem, if
> > there existed, and had been forma
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Helmut Grohne writes: ... > As such, my expectation is that moving from where we are to your idea > is not any easier than moving from a post-DEP-17 state. Therefore, I > do not see any need to delay DEP-17 work. I've been wondering about this possibility too. If a symlink flowerbed is where we decided we wanted to end up, I get the impression that starting from post-DEP-17 would be rather easier than starting from where we are now. DEP-17 provides us with a detailed roadmap for the first leg of that trip, whereas the route via reversion seems to be littered with unknowns at present. Cheers, Phil. -- Philip Hands -- https://hands.com/~phil signature.asc Description: PGP signature
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sun, 27 Aug 2023 at 11:30, Matthew Vernon wrote: > > Dear Luca, > > On 27/08/2023 03:16, Luca Boccassi wrote: > > [things] > > You've already been asked by a couple of people to moderate your tone in > this thread. I appreciate there is a lot of frustration around > /usr-merge, but your contributions are not helping with that at all. Nor > do they help us have a constructive discussion. Constructive discussions about past events need to be based on facts and evidence. Both are conspicuously absent from the bug submitter's messages. Without any of that, there was never any possibility that this bug could be in any way constructive since the moment it was opened, long before I first replied. > The DEP-17 work continues whilst this discussion is ongoing; this > discussion is not delaying that work. > > I think it is fair to say that when ctte decided on /usr-merge it was > expected that a) the process would be reasonably straightforward and b) > dpkg could be taught to understand directory aliasing, so we would not > have to be doing things "behind dpkg's back", as it were. Sure, but then it turned out that package maintainers are not only allowed to ignore the CTTE and its formal decisions, but can also actively block them in their packages. How could responsibility of that be assigned to anybody but those who chose to do that completely eludes me - and yet, that's exactly what the bug submitter does. > I think the need for a releases-long moratorium on moving files and the > volume and depth of technical work represented by DEP17 demonstrate that > those expectations turned out not to be met. > > Given that, it seems to me that a) warnings that "it's not that simple & > easy" were not FUD and describing them thus is unhelpful b) it's > reasonable to at least ask the question "given what we know now, are we > still sure this is the correct approach?" a) except that warnings weren't as mild and meek as "it's not that simple", we were clearly told everything would break left and right, and yet that didn't happen. Sure, dpkg has annoying bugs that need workarounds, but that's hardly surprising given its situation. But these are all packaging-only issues that affect some distribution maintainers for some package builds and maintenance tasks, and that's about it. No installation is affected. No user is affected. No external/third party developer is affected. So, yes, all of that was and still is FUD. b) no, in August 2023 it is very much not reasonable, and no fully informed and evidence-based good faith discussion would ever ask something like that. The reasonable questions to ask in August 2023 are along the lines of "what's the best way to work around those dpkg bugs and lift the moratorium?", which is what we are doing in actual good faith discussions elsewhere, with very productive results. > Any such consideration must be mindful of the fact that the majority of > Debian installs are now /usr-merged, which means that the complexity of > unwinding such installs has to be a heavy factor in thinking about > alternative approaches. > > I'm hopeful, but not certain, that the DEP17 work will get us to a > coherent state again by foxy (which still feels a long time off); I > don't think we should underestimate the work to be done in properly > completing /usr-merge. > > This discussion has, I think, been broadly useful in clarifying some > folks' thinking in this area. I would very much like if we could keep it > focused on the technical matters. I honestly cannot see how there can be any useful technical discussion about a completely evidence-free proposal to move to a symlinks-farm layout in 2023, with all the already well-known wider context that has been explained multiple times. The useful and productive and high-quality technical discussions are happening elsewhere.
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sun, 27 Aug 2023 at 18:03, Sam Hartman wrote: > > > TL;DR: I think I understand one of Ian's points. I explain, but do not > believe it is compelling as an argument to switch direction. > > > "Helmut" == Helmut Grohne writes: > >> I think "package management" is the wrong term here. It's not > >> just our tools and packages that are affected. All forms of > >> operating system management are affected: anything that changes > >> the software, and not just its configuration. > >> > >> Affected tooling includes not just our .debs, but also remote > >> configuration management systems like Ansible, image construction > >> (Dockerfiles), and 3rd-party software installation progrmas > >> (including both 3rd-party .debs and 3rd-party script-based > >> installation systems). > > Helmut> I don't follow the reasoning. Much of the tasks you'd carry > Helmut> out with (wlog) ansible - even when updating files - will > Helmut> continue to work in the aliasing layout. The reason that > Helmut> dpkg is more affected is that it has an inventory of files > Helmut> and reasons about their ownership in terms of > Helmut> packages. That's not how any kind of configuration > Helmut> management operates. If you just "make install" something, > Helmut> chances are that it'll just work with an aliasing layout > Helmut> even when installing with --prefix=/. I continue to argue > Helmut> that the problems we are seeing are quite specific to dpkg > Helmut> in large parts. > > I spent some time with Ian's paragraph you quote above trying to > understand it, and I think I got somewhere. > I considered replying yesterday but decided against, because I think we > are already seeing these effects, and have been seeing them long enough > that we would have a good feel for how serious the problems are. > > I do think that configuration management does have enough of an > inventory of files and reasoning about structure that some of these > problems could arise. I agree that configuration management does not > typically reason in terms of packages. > > But mechanisms like > > * ADD/COPY in Containerfile > > * The rsync module in ansible > * The file module in Ansible > > * various inventories related to modification detection in > configuration management do reason about the filesystem. > > I don't know what would happen if I did > > hosts: lots > remote_user: root > name: Does this shoot me in the foot > tasks: > - rsync: src=install_root dest=/ > > where install_root has a bin directory containing a couple of scripts. > I don't know if rsync will replace a symlink with a directory, or will > just write through the symlink in that situation. > If rsync happens to write through the symlink, there's probably some > other tool somewhere else thatreplaces the symlink. > And when an admin does that, they get an unbootable system, and fix > their playbook/whatever. > > Similarly, I bet someone somewhere has integrity management scripts that > want to look at what pam modules are installed under /lib/security, and > depending on how it worked, > they had to adjust the config when that moved to > /lib/security/x86_64-linux-gnu and then again some of them had to adjust > when /lib became a symlink. And they were probably frustrated during > the time when new installs had /lib as a symlink and upgrades did not. > > Similarly, I bet you can run into trouble with apparmor profiles if you > try to profile /bin/python rather than /usr/bin/python or similar. > > I bet people who had custom selinux policies had to adjust their > labeling rules and again some of them probably found the mixed state > where upgraded systems behaved differently than installed systems > frustrating. AppArmor, SELinux, Ansible, Chef, Puppet and whatnot all exist and are used on all distributions. Such distributions have install bases vastly higher in numbers than Debian, and have adopted the usrmerged layout a decade ago or so. Is there any evidence of AppArmor, SELinux, Ansible, Chef, Puppet and whatnot breaking catastrophically as a consequence? Speculating about what might or might not happen with future events is all good and well, but this is in the past. These things either all broke and suddenly became unusable on Arch, Fedora, CentOS, RHEL, Alma, Rocky, SUSE, Mandriva, Yocto, Ubuntu and so on, or they did not. If there's no evidence they did, then it's not even worth mentioning or discussing, it's all moot. > I seem to recall having to make some minor adjustments at my day job > related to usrmerge back in the buster/bullseye time frame. > I don't remember what they were. This is much more likely to be closer to reality. As with any change or upgrade, there's some minor adjustment here or there, that most won't even remember about after a while given how trivial it is. I certainly do not remember all the changes I've had to do to software I maintain when
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
TL;DR: I think I understand one of Ian's points. I explain, but do not believe it is compelling as an argument to switch direction. > "Helmut" == Helmut Grohne writes: >> I think "package management" is the wrong term here. It's not >> just our tools and packages that are affected. All forms of >> operating system management are affected: anything that changes >> the software, and not just its configuration. >> >> Affected tooling includes not just our .debs, but also remote >> configuration management systems like Ansible, image construction >> (Dockerfiles), and 3rd-party software installation progrmas >> (including both 3rd-party .debs and 3rd-party script-based >> installation systems). Helmut> I don't follow the reasoning. Much of the tasks you'd carry Helmut> out with (wlog) ansible - even when updating files - will Helmut> continue to work in the aliasing layout. The reason that Helmut> dpkg is more affected is that it has an inventory of files Helmut> and reasons about their ownership in terms of Helmut> packages. That's not how any kind of configuration Helmut> management operates. If you just "make install" something, Helmut> chances are that it'll just work with an aliasing layout Helmut> even when installing with --prefix=/. I continue to argue Helmut> that the problems we are seeing are quite specific to dpkg Helmut> in large parts. I spent some time with Ian's paragraph you quote above trying to understand it, and I think I got somewhere. I considered replying yesterday but decided against, because I think we are already seeing these effects, and have been seeing them long enough that we would have a good feel for how serious the problems are. I do think that configuration management does have enough of an inventory of files and reasoning about structure that some of these problems could arise. I agree that configuration management does not typically reason in terms of packages. But mechanisms like * ADD/COPY in Containerfile * The rsync module in ansible * The file module in Ansible * various inventories related to modification detection in configuration management do reason about the filesystem. I don't know what would happen if I did hosts: lots remote_user: root name: Does this shoot me in the foot tasks: - rsync: src=install_root dest=/ where install_root has a bin directory containing a couple of scripts. I don't know if rsync will replace a symlink with a directory, or will just write through the symlink in that situation. If rsync happens to write through the symlink, there's probably some other tool somewhere else thatreplaces the symlink. And when an admin does that, they get an unbootable system, and fix their playbook/whatever. Similarly, I bet someone somewhere has integrity management scripts that want to look at what pam modules are installed under /lib/security, and depending on how it worked, they had to adjust the config when that moved to /lib/security/x86_64-linux-gnu and then again some of them had to adjust when /lib became a symlink. And they were probably frustrated during the time when new installs had /lib as a symlink and upgrades did not. Similarly, I bet you can run into trouble with apparmor profiles if you try to profile /bin/python rather than /usr/bin/python or similar. I bet people who had custom selinux policies had to adjust their labeling rules and again some of them probably found the mixed state where upgraded systems behaved differently than installed systems frustrating. I seem to recall having to make some minor adjustments at my day job related to usrmerge back in the buster/bullseye time frame. I don't remember what they were. I think we've already committed to this pain, and I think we have enough of a window into that commitment that it doesn't seem to be very much pain. I mean spread across all our users, yes people have had to spend hundreds or thousands of hours dealing with this. But that's true with any upgrade. If we move back--if we unwind--things would get much much worse WRT this type of pain for a while. I think many more things would be sensitive to /bin/perl being a symlink than to /bin being a symlink. You could get into situations where /usr/bin/perl and /bin/perl were both files and differed. You could get into situations where /bin/perl vanished on one system. Etc. And if we actually try to have a symlink flower patch rather than a symlink farm, we are left with the pain of where things live differing between distributions and releases in a distribution. Mostly all we have left is the dpkg database. That pain will only affect people producing debs, which isn't just us. And yes, it will dis proportionally affect people who don't have our infrastructure and the ability to catch problems. Perhaps we should think about ways to mitigate that. There will be other smaller pains; if someone else had a system
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Hi Ian,
On Sat, Aug 26, 2023 at 11:24:33AM +0100, Ian Jackson wrote:
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> > On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > > And, the approach being taken very seriously privileges Debian itself,
> > > and those well-staffed derivatives able to do the necessary transition
> > > auditing (albeit, indeed, with tooling from Debian). I am
> > > firmly ideologically opposed to such a tradeoff.
> >
> > I have difficulties disagreeing with that. Getting this done is more
> > important to me though.
>
> I have hoisted this to the start of the mail. I think it is a hugely
> important point.
>
> Debian is not simply a technical project trying to thread its way in a
> complicated world. Debian is an ideological project. At its best,
> Debian is the infrastructrure that enables vast swathes of people to
> massively enhance their own technological autonomy. Many of our most
> controversial decisions served this goal, and stood the test of time.
>
> That's why *I'm* involved in Debian. Our technical choices should
> serve those goals, always.
>
> (To an extent, this divergence in goals may explain why at times our
> comments have been talking slightly past each other.)
I think I understand this and it resonates with me, but there are limits
to that. I don't think that Debian is that technological leader that you
perceive it to be. I hoped that other distributions would adopt the
multiarch directory layout to regain compatibility with Debian and none
did even though there is a clear, technical advantage to doing this.
Debian does not exist in isolation. It is dependent on a lot of
upstreams and in order for that relationship to be healthy, there needs
to be cooperation.
> If you want to think about it on more practical (or even, selfish)
> level, we want Debian to continue to be the preferred choice, when
> someone is choosing an upstream. We didn't get where we are today by
> following bad technical decisions made by others.
In the grand theme of things, the aliasing symlinks may be a suboptimal
technical approach. Please keep in mind though, that this change in
large parts is about people rather than something deeply technical.
After we stopped supporting booting without /usr being mounted in the
initramfs, the split between / and /usr was effectively random and stuff
was moving back and forth every so often and inconsistent between
different distributions or releases. I still remember iptables being an
annoying instance in this regard. So leaving technical aspects aside,
having large parts of the open source community agree on having those
aliasing symlinks already is a significant benefit even if it has
technical downsides.
In order to prefer Debian over something else, we want it to be similar
enough to make switching feasible while making it differ from others
enough to make switching worthwhile. Not having aliasing symlinks hardly
seems like an aspect that makes Debian more suitable to people. I guess
that you disagree with this and that is fine.
> This is indeed a plausible practical reason to do it the aliased way.
> >From my point of view, it amounts to saying "everyone else has made
> this mistake, so to be compatible, we must too".
I wouldn't say it that way, but it comes quite close.
> But I think that seriously underestimates our influence. Debian
> derivatives make up well over half of all Linux installations.
> They're the default basis for most CI images. If we decide this was a
> failed experiment, then indeed there will be some pain for a while,
> but fairly soon people will stop making this assumption.
Quite evidently, we judge this differently. The two of us value the
benefit of the end states differently and the cost to getting to each of
them. Therefore we arrive at different conclusions.
> I don't like the phrase "symlink farm" because it suggests that all,
> most, or even a substantial minority, of files have these symlinks.
> True, at the start, there will be at least a symlink allotment
> but I'm hoping that in the end it'll be a symlink flowerbed.
Let me suggest that this is wishful thinking. It's not only me saying
this, but you can read this from other responses as well. I encourage
you to use codesearch.d.n to see how your flowerbed really is a farm.
Let me give some examples to get you started:
libreoffice -> /bin/python
ghostscript, imagemagick, mesa -> /bin/env
bind9, manpages, net-snmp, qtbase-opensource-src -> /bin/perl
So I see that we either get a symlink farm or we get to include a lot of
Debian-specific patches or we get to argue with a lot of upstreams about
something that may seem entirely pointless to them. In any of these
cases, I consider that a significant cost.
> But pushing ahead won't lead to such a state. As I say, I think
> people will keep introducing new references to files by their
> non-physical names, and we'll keep getting lossage, essentially
>
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Hi Matthew, On Sun, 2023-08-27 at 11:30 +0100, Matthew Vernon wrote: > Any such consideration must be mindful of the fact that the majority of > Debian installs are now /usr-merged, which means that the complexity of > unwinding such installs has to be a heavy factor in thinking about > alternative approaches. Yes, I think there are many technical challenges required before Debian would be in a position to adopt the proposed Jackson filesystem layout. If Debian would choose to adopt it at all (an open question). Besides conversion, there is also the telemetry service that seems to be required to safely move to the proposed layout (AFAIK no alternative to this has been proposed yet). I'm not sure if there is already any work done on the path by the proposers? I'm also still missing an overview what the Jackson layout's advantages over the existing filesystem layout (merged-/usr) or the 2000's layout is (split-/usr). As far as I can tell it combines the disadvantages of both with much additional work required to get to it; I don't really see any advantage so far (besides "our tools can't handle anything else", but in practice it seems to work fine, and of course avoiding stuff associated with a certain company which I understand is a goal in itself for some people)... I would appreciate a list of technical and ideological reasons why switching to the Jackson layout is important for Debian. Ansgar
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Dear Luca, On 27/08/2023 03:16, Luca Boccassi wrote: [things] You've already been asked by a couple of people to moderate your tone in this thread. I appreciate there is a lot of frustration around /usr-merge, but your contributions are not helping with that at all. Nor do they help us have a constructive discussion. The DEP-17 work continues whilst this discussion is ongoing; this discussion is not delaying that work. I think it is fair to say that when ctte decided on /usr-merge it was expected that a) the process would be reasonably straightforward and b) dpkg could be taught to understand directory aliasing, so we would not have to be doing things "behind dpkg's back", as it were. I think the need for a releases-long moratorium on moving files and the volume and depth of technical work represented by DEP17 demonstrate that those expectations turned out not to be met. Given that, it seems to me that a) warnings that "it's not that simple & easy" were not FUD and describing them thus is unhelpful b) it's reasonable to at least ask the question "given what we know now, are we still sure this is the correct approach?" Any such consideration must be mindful of the fact that the majority of Debian installs are now /usr-merged, which means that the complexity of unwinding such installs has to be a heavy factor in thinking about alternative approaches. I'm hopeful, but not certain, that the DEP17 work will get us to a coherent state again by foxy (which still feels a long time off); I don't think we should underestimate the work to be done in properly completing /usr-merge. This discussion has, I think, been broadly useful in clarifying some folks' thinking in this area. I would very much like if we could keep it focused on the technical matters. Regards, Matthew
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sat, 26 Aug 2023 at 11:27, Ian Jackson
wrote:
>
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> > On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > > And, the approach being taken very seriously privileges Debian itself,
> > > and those well-staffed derivatives able to do the necessary transition
> > > auditing (albeit, indeed, with tooling from Debian). I am
> > > firmly ideologically opposed to such a tradeoff.
> >
> > I have difficulties disagreeing with that. Getting this done is more
> > important to me though.
>
> I have hoisted this to the start of the mail. I think it is a hugely
> important point.
>
> Debian is not simply a technical project trying to thread its way in a
> complicated world. Debian is an ideological project. At its best,
> Debian is the infrastructrure that enables vast swathes of people to
> massively enhance their own technological autonomy. Many of our most
> controversial decisions served this goal, and stood the test of time.
Exactly, and usr-merge provides exactly that infrastructure, which is
one of the many reasons why it is necessary, as it should be beyond
obvious by now.
> If you want to think about it on more practical (or even, selfish)
> level, we want Debian to continue to be the preferred choice, when
> someone is choosing an upstream. We didn't get where we are today by
> following bad technical decisions made by others.
Very true, and a perfect example of such a "bad technical decision" is
the symlinks-farm layout that is proposed by a couple of people: it
doesn't solve any real problem, and just causes issues for developers
and users, and is purely designed to carefully tip-toe around one
singular hopelessly broken debian-specific tool, dpkg, which suffers
from long-standing bad design decisions that have never been fixed to
this day. It was attempted once, it failed, it was rolled back and
actual usr-merge was successfully deployed.
> > * Basically every other distro uses aliasing now. I expect that
> >a few upstreams have stopped paying attention to systems in your end
> >state. Therefore, they may not only hard code paths in /usr/bin, but
> >also the other way round assume that everything is to be found in /.
>
> This is indeed a plausible practical reason to do it the aliased way.
> >From my point of view, it amounts to saying "everyone else has made
> this mistake, so to be compatible, we must too".
>
> But I think that seriously underestimates our influence. Debian
> derivatives make up well over half of all Linux installations.
> They're the default basis for most CI images. If we decide this was a
> failed experiment, then indeed there will be some pain for a while,
> but fairly soon people will stop making this assumption.
In case you haven't noticed, it's not 1998 anymore. It's 2023, and
Debian is sliding fast into irrelevance, largely because it takes a
decade of fighting off trolls and saboteurs to achieve the most
innocuous of changesl, while most other projects can just implement
obviously correct and needed features such as this one and many others
in a couple of months and then move on to the next. In other words,
other distros innovate while we stagnate. Nobody would truly care if
somehow madness descended on this project and such a grievous act of
self-harm was actually perpetrated, at most some raised eyebrows and
some "they are at it again, aren't they" snarky remarks. Certainly
nobody would move back. For example, I can provide a cast-iron
guarantee that systemd will keep supporting only an usr-merged layout
and not work anywhere else starting from the next version (currently
in main), so there goes ~99.5% of Debian installations.
> > * A key motivation cited for doing the merged-/usr work is called
> >"hermetic /usr". [...] Setting up the aliasing symlinks is easier and
> >less prone to change over time than setting up the symlink farm that
> >you are proposing.
>
> I don't like the phrase "symlink farm" because it suggests that all,
> most, or even a substantial minority, of files have these symlinks.
> True, at the start, there will be at least a symlink allotment
> but I'm hoping that in the end it'll be a symlink flowerbed.
"I'm hoping" doesn't cut it. There is only one example of such an
attempt, in OpenSUSE, and it never materialized. It's been 10 years,
and still the handful of proponents of symlinks-farm have absolutely
nothing to show other than "I'm hoping". What are we even doing here?
> > And then we have a large body of people who simply
> > want this to be over and not having to thing about /usr-merge
> > consequences anymore.
>
> Well, of course it is very tempting to declare the matter as settled
> and hope that it will go away. I too want an end state where we will
> eventually be able to forget about all of this.
The matter is settled, and was settled long ago to boot.
> Or to put it another way, the delays to completion of this
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > And, the approach being taken very seriously privileges Debian itself,
> > and those well-staffed derivatives able to do the necessary transition
> > auditing (albeit, indeed, with tooling from Debian). I am
> > firmly ideologically opposed to such a tradeoff.
>
> I have difficulties disagreeing with that. Getting this done is more
> important to me though.
I have hoisted this to the start of the mail. I think it is a hugely
important point.
Debian is not simply a technical project trying to thread its way in a
complicated world. Debian is an ideological project. At its best,
Debian is the infrastructrure that enables vast swathes of people to
massively enhance their own technological autonomy. Many of our most
controversial decisions served this goal, and stood the test of time.
That's why *I'm* involved in Debian. Our technical choices should
serve those goals, always.
(To an extent, this divergence in goals may explain why at times our
comments have been talking slightly past each other.)
If you want to think about it on more practical (or even, selfish)
level, we want Debian to continue to be the preferred choice, when
someone is choosing an upstream. We didn't get where we are today by
following bad technical decisions made by others.
> * Basically every other distro uses aliasing now. I expect that
>a few upstreams have stopped paying attention to systems in your end
>state. Therefore, they may not only hard code paths in /usr/bin, but
>also the other way round assume that everything is to be found in /.
This is indeed a plausible practical reason to do it the aliased way.
>From my point of view, it amounts to saying "everyone else has made
this mistake, so to be compatible, we must too".
But I think that seriously underestimates our influence. Debian
derivatives make up well over half of all Linux installations.
They're the default basis for most CI images. If we decide this was a
failed experiment, then indeed there will be some pain for a while,
but fairly soon people will stop making this assumption.
> * A key motivation cited for doing the merged-/usr work is called
>"hermetic /usr". [...] Setting up the aliasing symlinks is easier and
>less prone to change over time than setting up the symlink farm that
>you are proposing.
I don't like the phrase "symlink farm" because it suggests that all,
most, or even a substantial minority, of files have these symlinks.
True, at the start, there will be at least a symlink allotment
but I'm hoping that in the end it'll be a symlink flowerbed.
> And then we have a large body of people who simply
> want this to be over and not having to thing about /usr-merge
> consequences anymore.
Well, of course it is very tempting to declare the matter as settled
and hope that it will go away. I too want an end state where we will
eventually be able to forget about all of this.
But pushing ahead won't lead to such a state. As I say, I think
people will keep introducing new references to files by their
non-physical names, and we'll keep getting lossage, essentially
forever. (Adopting Simon's terminology.)
Or to put it another way, the delays to completion of this project
have not been due to the political opposition,. They have been
because the project encountered technical problems. Problems whose
existence was predicted by subject matter experts but dismissed at the
time as FUD. Problems which were apparently not regarded as real by the
non-expert decisionmakers on the TC. Problems which still remain in
large part unresolved, albeit in some caes "mitigated".
> > Aliasing is EBW, and "Only use canonical names" is not good enough
> > ==
> >
> > There is basically one underlying technical reason for preferring the
> > un-aliased usrmerge approach: aliasing directories in this way leads
> > to great complication in file management, especially in package
> > management software and in individual packages.
>
> I'm not sure I follow this argument precisely.
This argument is basically drawing together the common factor in the
DEP-17 problem list.
> these complications mostly affect ourselves and
> our package management while end users are mostly unaffected.
I think "package management" is the wrong term here. It's not just
our tools and packages that are affected. All forms of operating
system management are affected: anything that changes the software,
and not just its configuration.
Affected tooling includes not just our .debs, but also remote
configuration management systems like Ansible, image construction
(Dockerfiles), and 3rd-party software installation progrmas (including
both 3rd-party .debs and 3rd-party script-based installation systems).
And yes, actual *end users* (especially of something like Ubuntu)
