On the burden of proof and the correctness of software:
I'm afraid I see a pattern, where blanket statements are made which
are only "mostly" or "roughly" or "generally" true. But the
discrepancies and details matter. When we make computer systems, it's
not good enough to if they're only "mostly" right.
Every program is an unproven conjecture whose truth we end up relying
on. We need simple axioms, and rigorously correct reasoning.
Aliased usrmerge, when deployed, was a conjecture that was disputed:
"usrmerge via directory aliasing functions correctly".
In the time since the TC dismissed the objections and ruled in favour
of believing in the truth of this conjecture, has been disproven.
It is false in such important wsys that even in Debian, where we have
every possible advantage, we have been significantly inconvenienced.
Now we have a long list of counterexamples demonstrating the
invalidity of the original thesis.
Nevertheless, in mathematical terms, we are trying to patch up
the conjecture with many qualifications - the mitigations.
Also, we are apparently trying to impose on 3rd parties qualifications
about when it is OK to refer to files by the "wrong" name, which
cannot even be stated clearly, let along succinctly.
Despite all this, apparently when we argue for continued reliance on
this disproven assertion we go back to imprecise statements.
Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing [and 3 more
messages]"):
> In order to prefer Debian over something else, we want it to be similar
> enough to make switching feasible while making it differ from others
> enough to make switching worthwhile. Not having aliasing symlinks hardly
> seems like an aspect that makes Debian more suitable to people. I guess
> that you disagree with this and that is fine.
Debian has historically been simply much more reliable. That
reliability doesn't come from one particular decision. It comes from
an aggregate of, on the whole, making better decisions. That
reliability is very valuable to our users and downstreams. It's
one of our our most compelling advantages.
> My impression has always been that the disagreement on the end
> state was involving a minority.
Well, if you disagree with aliased usrmerge and say you don't want it,
you get abuse. Even here, many abusive messages have been posted with
impunity by persons with considerable status.
usrmerge is a facet of Debian's culture wars. (Debian's culture wars
are mostly orthogonal to the wider world's, but it would be foolish to
ignore the huge social factors at play.) If one is on the
currently-losing side one is not optimistic about the value of making
one's views widely known. Most of my friends agree with me on
substance but think it's a lost cause.
Of course, I could try to "solve" this invisibility by making a blog
post drawing attention and asking random people across the internet to
"contribute" to this discussion (or even just ask them to "me too").
That might even be effective, since currently I'm rather alone here.
I'm not doing that because Debian is quite toxic enough. (And also
because as an apparent minority, seen as a reactionary outgroup, we
would get blamed for the behaviour of extremists.)
> > This argument is basically drawing together the common factor in the
> > DEP-17 problem list.
>
> And that's precisely why I don't understand your argument. All of the
> DEP-17 problems are supposed to go away. So it appears as if you cite a
> list of problems that I expect to not be problems for much longer as a
> reason for changing the end goal.
DEP-17's list of *problems* forms a category: directory-aliasing-
induced malfunctions. But the DEP-17 *solutions* are ad-hoc, and many
are only applicable to to malfunctions that occur during migration.
There is nothing inherently migration-specific about aliasing-induced
malfunction; migration is in this context mostly just a lot of Things
happening, each of which has a chance to go wrong.
The result of the current plan is that all directory-aliasing-induced
malfunctions must be averted, individually, forever. By us, but also
by everyone who has to modify any Debian-derived system.
> > Affected tooling includes not just our .debs, but also remote
> > configuration management systems like Ansible, image construction
> > (Dockerfiles), and 3rd-party software installation progrmas (including
> > both 3rd-party .debs and 3rd-party script-based installation systems).
>
> I don't follow the reasoning. [...]
Sam has posted a helpful set of examples of things that one might do,
whose consequences with aliased directories are unclear.
These are of course only examples.
> > I would be much more convinced that all of this isn't a problem, if
> > there existed, and had been formally adopted (eg by existing in some
> > manual somewhere) a short and clear set of rules about what is and
> > isn't allowed - not just rules for us within Debian, but rules for
> > everyone, everywhere, referring to and modifying files.
>
> It appears to me that the empty set of rules (outside packaging) is too
> simple for you to consider.
"Packaging" is doing a lot of work there. I find your comment rather
tendentious, I'm afraid.
I think we need a short and clear set of rules for what you can do
with ansible, rsync, docker, etc. etc. etc., as well as just apt and
dpkg.
Even with apt and dpkg, within Debian, we don't have a clear rule for
when it is OK for a package's source code to name a file by its name
in /.
The selling point of aliasing was "you can freely pun / and /usr",
but it's false.
> > > We already have the Debian Usr Merge Analysis Tool available at
> > > https://salsa.debian.org/helmutg/dumat and its output at
> > > https://subdivi.de/~helmut/dumat.yaml. As explained on d-d@l.d.o, I want
> > > to turn those findings into automatic RC bugs. Does that alleviate your
> > > concerns to some extent?
> >
> > That's certainly helpful for the transition now. Are we going to
> > maintain this or something like it indefinitely ?
>
> I don't think so. Once packages have moved to /usr, we can simply have a
> lintian-checkable policy of not shipping stuff in aliased paths. What's
> the benefit of continuing this crazy approach then?
Not just not *shipping*. All management operations (eg that inspect
non-filesystem databases that mention files), even some read-only
ones, must use physical paths. So we need lintian *at least* to
search maintainer scripts etc. (And see also my ideological comments
about privileging Debian.)
> > But, a goal of the directory aliasing is to be compatible with other
> > systems, so that 3rd party software is *allowed* to refer to, and,
> > presumably, ship files in /bin (because "there's no difference now,
> > right?").
>
> If they install without dpkg, yes.
Many upstreams make a .deb by running their ad-hoc installation shell
script (which in this brave new world will talk about /bin, probably)
and stuffing the result in a .deb.
What engeineered safeguard are we planning to prevent the
installation of usrmerge-broken .debs on end user systems ?
I think the current answer is "we hope ISVs will run lintian" ?
Also I'm don't think the statement is true on its own terms.
Ad-hoc installation scripts frequently do all manner of things,
including querying or even modifying system package databases etc.
> > Our downstreams (of all kinds) are are more likely to use other
> > tooling (of all kinds).
>
> Tooling that does not have some kind of inventory of the filesystem is
> mostly unaffected. Given the wide adoption of the aliasing approach, I
> think it is fair to ask for evidence at this time.
Tooling that *uses* an inventory of the filesystem can be affected,
even if it doesn't modify that database. So "has" isn't right. It's
not tolls that *have* such a database. It's any program that
exchanges filenames with a tool that uses such a database. Also, note
"mostly".
> Given the state of discussion, I think both of us understand quite
> precisely what we disagree about. We don't seem to be moving beyond this
> state of understanding one another. How can we move forward?
>
> If you still see Debian reverting to a non-aliased layout (and you seem
> to), I see the following possible steps forward:
...
> Please don't jump into quick replies to these items. My understanding is
> that each of these steps requires significant effort. If you have a
> ready answer here, we likely have more misunderstandings.
Well of course I don't have funding to pursue this, and many of my
allies have been driven away. It's certainly not worth doing that
substantial amount of work if it's likely that the TC is going decide
to go with the aliased version anyway.
Hence my original question: is it worth trying to make a plan ?
It probably makes sense for me to give a quick answer to this bit:
> * Become more precise about what your layout looks like precisely. Our
...
> Let me give some examples to get you started:
> libreoffice -> /bin/python
> ghostscript, imagemagick, mesa -> /bin/env
> bind9, manpages, net-snmp, qtbase-opensource-src -> /bin/perl
Of these I would hope (by, say, 2033) to only include env.
perl upstream doctrine used to be that /usr/bin/perl was the correct
canonical path. I haven't checked if that recommendation still
stands.
I don't know what Python upstream doctrine is on /bin vs /usr/bin; if
it were that Python is supposed to be in /bin, then we might need to
follow that. (But, are they dropping support for the BSDs?)
With another hat on in an upstream project I've have been receiving
patches to change #!/bin/bash in CI scripts to #!/usr/bin/env bash.
Sam Hartman writes ("Bug#1050001: Unwinding directory aliasing [and 3 more
messages]"):
> TL;DR: I think I understand one of Ian's points. I explain, but do not
> believe it is compelling as an argument to switch direction.
Thanks for your contribution. Yes, you appear to have seen the kind
of thing I was getting at.
> I mean spread across all our users, yes people have had to spend
> hundreds or thousands of hours dealing with this.
> But that's true with any upgrade.
A big part of my point is that this isn't just an *upgrade* issue.
Your examples demonstrate things that are at the very least confusing
and complicated, but which will still be that way in 20 years time.
> And if we actually try to have a symlink flower patch rather than a
> symlink farm, we are left with the pain of where things live differing
> between distributions and releases in a distribution.
I think it's worth pointing out that any software which is trying to
be portable to Unix systems other than just Linux (which includes the
BSDs and MacOS) will need to avoid assuming directory aliasing.
> On the other hand, symlinks have been around throughout the lifetime of
> Linux, and I think sysadmins do have a feel for how they work.
Directory symlinks have always been troublesome. Sometimes they're
the best answer, but IMO not here.
Ian.
