Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > And, the approach being taken very seriously privileges Debian itself,
> > and those well-staffed derivatives able to do the necessary transition
> > auditing (albeit, indeed, with tooling from Debian). I am
> > firmly ideologically opposed to such a tradeoff.
>
> I have difficulties disagreeing with that. Getting this done is more
> important to me though.
I have hoisted this to the start of the mail. I think it is a hugely
important point.
Debian is not simply a technical project trying to thread its way in a
complicated world. Debian is an ideological project. At its best,
Debian is the infrastructrure that enables vast swathes of people to
massively enhance their own technological autonomy. Many of our most
controversial decisions served this goal, and stood the test of time.
That's why *I'm* involved in Debian. Our technical choices should
serve those goals, always.
(To an extent, this divergence in goals may explain why at times our
comments have been talking slightly past each other.)
If you want to think about it on more practical (or even, selfish)
level, we want Debian to continue to be the preferred choice, when
someone is choosing an upstream. We didn't get where we are today by
following bad technical decisions made by others.
> * Basically every other distro uses aliasing now. I expect that
> a few upstreams have stopped paying attention to systems in your end
> state. Therefore, they may not only hard code paths in /usr/bin, but
> also the other way round assume that everything is to be found in /.
This is indeed a plausible practical reason to do it the aliased way.
>From my point of view, it amounts to saying "everyone else has made
this mistake, so to be compatible, we must too".
But I think that seriously underestimates our influence. Debian
derivatives make up well over half of all Linux installations.
They're the default basis for most CI images. If we decide this was a
failed experiment, then indeed there will be some pain for a while,
but fairly soon people will stop making this assumption.
> * A key motivation cited for doing the merged-/usr work is called
> "hermetic /usr". [...] Setting up the aliasing symlinks is easier and
> less prone to change over time than setting up the symlink farm that
> you are proposing.
I don't like the phrase "symlink farm" because it suggests that all,
most, or even a substantial minority, of files have these symlinks.
True, at the start, there will be at least a symlink allotment
but I'm hoping that in the end it'll be a symlink flowerbed.
> And then we have a large body of people who simply
> want this to be over and not having to thing about /usr-merge
> consequences anymore.
Well, of course it is very tempting to declare the matter as settled
and hope that it will go away. I too want an end state where we will
eventually be able to forget about all of this.
But pushing ahead won't lead to such a state. As I say, I think
people will keep introducing new references to files by their
non-physical names, and we'll keep getting lossage, essentially
forever. (Adopting Simon's terminology.)
Or to put it another way, the delays to completion of this project
have not been due to the political opposition,. They have been
because the project encountered technical problems. Problems whose
existence was predicted by subject matter experts but dismissed at the
time as FUD. Problems which were apparently not regarded as real by the
non-expert decisionmakers on the TC. Problems which still remain in
large part unresolved, albeit in some caes "mitigated".
> > Aliasing is EBW, and "Only use canonical names" is not good enough
> > ==================================================================
> >
> > There is basically one underlying technical reason for preferring the
> > un-aliased usrmerge approach: aliasing directories in this way leads
> > to great complication in file management, especially in package
> > management software and in individual packages.
>
> I'm not sure I follow this argument precisely.
This argument is basically drawing together the common factor in the
DEP-17 problem list.
> these complications mostly affect ourselves and
> our package management while end users are mostly unaffected.
I think "package management" is the wrong term here. It's not just
our tools and packages that are affected. All forms of operating
system management are affected: anything that changes the software,
and not just its configuration.
Affected tooling includes not just our .debs, but also remote
configuration management systems like Ansible, image construction
(Dockerfiles), and 3rd-party software installation progrmas (including
both 3rd-party .debs and 3rd-party script-based installation systems).
And yes, actual *end users* (especially of something like Ubuntu) are
largely unaffected because they don't do much operation system
management. Regarding Ubuntu specifically: Ubuntu's approach to 3rd
party softare during upgrades is (for very plausible reasons) quite
sledgehammery.
> > Naming files by their canonical names will have to be done everywhere.
...
> This seems overgeneralized to me. [...]
Well, this is a key part of the problem. IMO we need to be able to
state simple and clear rules, which when followed, will result in
reliable construction of working software systems.
We build our systems by building on layers of abstractions. Layers of
abstraction allow us to narrow the scope of our consideration. In our
systems, the filesystem is a pervasive abstraction. A filesystem with
directory aliasing is a much more complicated and subtle abstraction.
So we can either write broad and restrictive rules ("overgeneralised"
as you put it), or subtle and complicated ones.
I would be much more convinced that all of this isn't a problem, if
there existed, and had been formally adopted (eg by existing in some
manual somewhere) a short and clear set of rules about what is and
isn't allowed - not just rules for us within Debian, but rules for
everyone, everywhere, referring to and modifying files.
I think one reason that hasn't been done is that it's hard. Another
is perhaps that some of the rules required for successful and reliable
operation contradict some of the ostensible goals of the aliasing.
> > Spotting and mitigating violations is hard
> > ------------------------------------------
> >
> > We do not currently have good tooling that will spot violations of
> > this rule. It's not clear precisley what the right behaviour of our
> > tools would be; we need to alert *the right set of users* to the
> > mistakes, and *with the right level of severity*. Many of our key
> > tools don't have a good way to produce "critical warnings". The
> > consequences of violations are unpredicatable and can depend on event
> > ordering. But they can be very severe. So we are creating a source
> > of bad heisenbugs.
>
> We already have the Debian Usr Merge Analysis Tool available at
> https://salsa.debian.org/helmutg/dumat and its output at
> https://subdivi.de/~helmut/dumat.yaml. As explained on d-d@l.d.o, I want
> to turn those findings into automatic RC bugs. Does that alleviate your
> concerns to some extent?
That's certainly helpful for the transition now. Are we going to
maintain this or something like it indefinitely ?
But I don't think it addresses the point I intended. We need to be
able to spot when a user installs a .deb, that they got from
"somewhere", when a directory-aliasing thing is going to go wrong.
> > Also, we only have direct control over the behaviour of our own
> > packages, images, etc. that we (Debian) ship. [...]
>
> This is very true. On the flip side, shipping files in /bin or /lib is
> usually done by essential packages or packages required for booting.
> Neither of these topics are very popular in 3rd party software.
But, a goal of the directory aliasing is to be compatible with other
systems, so that 3rd party software is *allowed* to refer to, and,
presumably, ship files in /bin (because "there's no difference now,
right?").
> > Violations of the "only use canonical names" rule are required
...
> > Now, those references are almost all in "immutable" contexts, where it
> > doesn't actually matter, since the file is in fact available by the
> > non-canonical name. However, this introduces a new implied rule:
> > it becomes a bug to take a filename you see in a place where the file
> > is being *read*, and apply it in a context where the file is going to
> > be *updated*.
>
> This is accurate and a nice way to describe it. Are there actually that
> many tools that need to update files of a Debian installation?
I listed some above. There are going to be a lot more.
We, here in Debian, are likely to have a much more restricted view of
what really goes on out there in the world. After all when *we* want
to modify a Debian system we usually do it by modifying .debs,
modifying .dsc's within Debian. That is easy for us but a lot harder
for our downstreams and users.
Our downstreams (of all kinds) are are more likely to use other
tooling (of all kinds).
> This feels vague to me. Given the general adoption of the aliasing
> approach in other distributions, I'd expect that we have at least
> anecdotal evidence. Are you aware of any?
No. But most of my closer friends, and most of the systems I maintain
myself (especially the more complicated ones) have so far managed to
put off having usrmerge, in the hope that the technical situation
would become less buggy in the future. And, frankly, the political
situation is so toxic that I expect many people who have trouble with
usrmerge will avoid sharing it with anyone but their close friends.
It took a significant externally funded research programme to document
the DEP-17 list. Let us not say, once more, that categories of bug
predicted by an expert are probably imaginary or rare.
> > Looking towards the future
> > --------------------------
> >
> > It seems to me that directory aliasing will continue to be a source of
> > very annoying bugs indefinitely, well after the transition is fully
> > complete. In another 20 years we'll still be debugging strange
> > installation breakage that will turn out to be due to directory
> > aliasing.
>
> I very much hope that this is not where we're heading. Quite contrary,
> over the past 20 years, we've had packages move between / and /usr and
> catching up with where stuff was located. I expect that we get rid of
> this bug class and trade it for that other bug class that you are
> describing. The big question then is the frequency of each class.
We can get rid of the former bug class simply by moving everything to
/usr, once. We'll *experience* those bugs now, but if we do it as
part of a coordinated programme we can have tooling to spot it. When
that transition is complete, those bugs won't arise any more.
> Speaking for myself without any Freexian or CTTE hat, I mainly want the
> transition to be finished such that I can spend brain cells on more
> useful (to me) matters.
As I say, I don't think the directory aliasing situation is ever going
to be finished. We can revert it, or have it forever be weird.
> > The non-aliased approach
> > ========================
>
> > >From my conversation with Helmut, it seems that we are envisaging, as
> > part of the aliased-usrmerge approach, that there will be tools to
> > detect violations of the "refer only by canonical path" rule.
> >
> > But detecting violations of "these directories only ought to contain
> > compat symlinks into /usr" rule is a *lot* simpler. It can be done,
> > quite reliably, on end-user systems.
>
> While that may be true, I don't think we do QA on end-user systems. What
> we do QA on is binary packages. In that sense, we can more reliably
> detect problems in packages, no?
For the non-aliased approach, we can certainly do a form of QA on
end-user systems. We can provide, or even install by default, tools
which will detect violations of the intended filesystem structure.
This is much harder for the aliased approach, because there the bugs
don't end up reliably reified on the filesystem. They remain latent
until they cause hard-to-diagnose lossage.
Simon McVittie writes ("Bug#1050001: Unwinding directory aliasing"):
> This is not merged-/usr with the meaning used by the technical committee's
> past resolutions,
(Assuming you're not just arguing terminology:)
I am indeed asking the TC to change its mind. We now have a great
deal more information than we did. We now know for sure that
Guillem's concerns were not just FUD. I think it unlikely that the TC
in 2019 would have decided as they did, if they had had the list from
DEP-17 in front of them.
> It seems to me that *not* aliasing /bin and /usr/bin will continue
> to be a source of very annoying bugs indefinitely, because each path
> you might want to refer to will have a "right" version and a "wrong"
> version.
I find this is a funny way of looking at it. Most files have only one
name. We don't consider this a source of bugs in this way.
My end state is a system where most files have one name.
Note that even if files *shipped by Debian* are all in /usr, this does
not obvioate the need for PATH-searching (and similar), because we
still want to be able to use /usr/local.
Ansgar writes ("Bug#1050001: Unwinding directory aliasing"):
> Why should we invent again another, incompatible filesystem layout?
I'm not inventing a new filesystem layout. What I am proposing is in
fact the simple composition of tried and tested layout principles:
(i) the traditional Unix filesystem layout + (ii) file motion
(from / to /usr), which is a thing we know how to do and can do
routinely.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
