Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?
Le jeudi, 21 février 2019, 15.28:23 h CET Ian Jackson a écrit : > Back in the wider world, of course many people build packages on > Debian systems for installation `somewhere else'. I have done it > myself and probably many of the people reading this message have too. > > What is implicitly going on here is that it is mostly-tacitly[2] being > assumed (or declared) that `I built this .deb on my laptop[1] and gave > it to my friend' is wrongheaded. I don't think it is wrongheaded, but I do think that assuming that it is meant to work consistently under any circumstances, is. There are _so many_ circumstances that make this exercise error-prone: "I built this amd64 .deb and gave it to my friend for use on their RasbperryPi arm64 host" "I built this .deb on my laptop on which I had installed a /usr/local/bin/ grep" "I built this .deb an gave it to my friend who runs CentOS" etc > I don't think doing that is wrongheaded. Certainly it is extremely > common; we don't have any public pronouncements saying it is somehow > wrong; and we have had little discussion where we as a project decided > that this was now a use case which we feel free to just break. Frankly, while it's not _broken_ per se, it is and has always really been fragile. True, merged-/usr arguably worsens _one_ aspect of building packages on one's host, but in ways that are really easy to detect, and warn for. > Personally I think that this is a very important thing to keep > working. It is the very core of users' collective software freedom. You seem to be conflating "building on one's host" with "outside of any chroot", and equating this with "users' collective software freedom" is really making your point a disservice. Being able to improve, build and share binary artifacts of free software is _vital_, and the whole point of building software distributions in the first place, but insisting that these rights are only "truly" exercised when builds are done outside of chroots is disingenuine. .deb packages already have to be built using certain tools, so we do set the limit (of what minimal set of tools are mandatory) somewhere, and my point is that this limit might not be at the right place. I'd be totally OK with saying "the only supported way to build .deb packages from buster on is using by pdebuild; however, you _can_ use dpkg-buildpackage alone, but be aware (and you'll get warnings) that this can lead to building .deb packages with undesireable properties." The second half of the sentence is already true, and has always been; we probably just failed at communicating it sufficiently clearly. > And that software freedom needs to be easy to exercise in practice. > Despite a lot of excellent tooling, chroots are still not entirely > trivial to set up and maintain. Then that's maybe what we should be fixing. > I would invite the TC to read this manpage, which is trying to explain > to a Debian user how to change the programs on their own computer > https://manpages.debian.org/stretch-backports/dgit/dgit-user.7.en.html > and then consider how much even worse it would be if we had to write > about chroots too. Perhaps dpkg-buildpackage should be grown to build in a chroot by default? Or get an option to do that? Really, I think we should stop considering that .debs built outside of "controlled environments" are more than temporary software transports. Our "standard" package building tools should build in such environments by default. > To TC members who are minded to uphold the current approach, I would > ask: can you please explicitly state your opinion on this ? That is: > > Is it OK for a Debian user to build .deb on their laptop and give it > to their friend ? Yes; provided that said .deb is built in a sane (sanitized / controlled) environment. As you're talking about non-chroot builds; it is OK, provided that the tools warn about that. In that area, I think the "Build-Tainted-By" are steps in the right direction. > If it is OK, how will we make sure that it does not pointlessly break ? As it is already broken in many ways, your question is biased. But I think the good way is to normalize "proper builds", by making sure our standard tools "do the right thing" by default. > I readily admit that there are many ways that this can produce a > result significantly different to the official Debian package, > particularly because the package build system may configure itself > differently in the the presence of unexpected. The resulting package > is probably not going to be suitable for wide distribution. > > But those kind of problems are (a) not serious in practice > (b) generally have obvious symptoms and (c) are easily worked around > by various means. Working around a usrmerge-generated failure is > difficult; it usually involves doing serious violence to the upstream > build system, or perhaps horrific rules file bodges. Given a Build-Tainted-By flag in the binary artifact
Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?
Le mardi, 19 février 2019, 01.59:18 h CET Steve McIntyre a écrit : > While I'm not necessarily disagreeing with the overall point(s) here, > some of the points in this list of arguments seem dubious at > best. Maybe you could expand? Sure. Sorry for publishing a new ballot before answering. > >* booting with `/` only is not systematically tested in Debian anymore; > > I'm assuming you mean "/ without /usr" here? Yes. Clarified this point by merging it with the "separate / and /usr" one above. > >* the packaging infrastructure to install files outside of `/usr` is not > > standard and represents technical debt: > > I have no idea what you're suggesting here. Same here (clarified in the ballot). What I'm trying to say here is that installing (e.g.) libs to /lib instead of /usr/lib is usually a deviation from standard software building (either in upstream packaging, or in debian/rules), that needs to be maintained on the long-term. Cheers, OdyX signature.asc Description: This is a digitally signed message part.
Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?
On Sun, 24 Feb 2019 14:16:03 +0100 Johannes Schauer wrote: > I found that some important arguments are still missing. A recent mail by > Guillem [1] nicely summarizes also many of my own thoughts. I'm going to paste > the relevant content into this mail for convenience of the reader: I think those have actually been addressed before, even if they're not explicitly listed or refuted in the draft. Guillem seems to accept moving away from current filesystem layout, but also opposes the resulting layout used by other distributions (which includes the /bin -> /usr/bin symlink). Do you share that view? The main issue with Guillem's view is that in practice it would make the transition more cumbersome and error-prone. If there is no directory-level symlink, then migrating things requires either creating individual symlinks for everything instead, or having a flag day to make sure nothing references the old location any more. In practice either of those is more problematic than a directory symlink. > > 1) The merged-/usr-via-symlinks deployment method used by both > >debootstrap and usrmerge, means that those systems will get > >permanent filesystem aliasing problems, forever. Even when all > >files might have been moved to their /usr counterparts in the > >packages! This is due to the fact that different pathnames can > >end up pointing (before any canonicalization!) to the same dentry. > > > >This does not only affect dpkg (dpkg, dpkg-divert, dpkg-trigger, > >etc), and update-alternatives (in a worse way), but any program > >that uses pathanmes in the filesystem as keys in databases f.ex. This has already been in use, both in Debian and in other distributions, for quite some time. No major problems have been reported. Arguing about "any program" is not useful when discussing only Debian choices, because the layout is used by other distributions in any case and programs have to cope. > > 2) It bypasses the packaging system understanding of what is on the > >filesystem and creates mismatches between what's on binary packages > >and what's on disk. This also has not caused major problems, but it can be solved straightforwardly in dpkg if needed. Dpkg can special-case any legacy /bin path contained in a package and internally change it to /usr/bin. > > 3) Switching packages to the merged-/usr layout could have been > >accomplished automatically via debhelper for a coverage of around > >99% (?) of the archive. With something along the lines of: This would just create a more cumbersome and error-prone migration. > > 4) Due to having to support the broken merged-/usr-via-symlinks > >deployment, when we want to move the contents of the binary > >packages to the merged-/usr layout, we require now to include tons > >of logic in (probably new) maintainer scripts, when we have been > >trying to remove them altogether. :( With even more files untracked > >by dpkg itself, bypassing the packaging system even more, when the > >compat symlinks could have been shipped in the binary packages. This seems to be based on questionable implicit assumptions. Namely that the move would want to create an individual per-file compatibility symlink in /bin for non-merged systems, and this is the part which would cause problems. There seem to be neither technical reasons to prefer non-merged systems for any use nor practical difficulties migrating existing systems to the merged layout, so long-term it seems OK to deal with this by just requiring merged-usr and not shipping any individual compatibility links for non-merged systems. This allows the simplest possible move - just install the files in /usr normally. > > 5) Most new installations will not even benefit from this hacky and > >rushed deployment, as long as they do not use a separate parition > >for /usr! Is there a point to this? Yes, migrating to merged /usr is mostly an internal implementation detail for most users. But this does not exactly give any argument against the migration, other than calling it "hacky and rushed" while it seems to be the most practical way to do things. > > 6) The merged-/usr-via-symlinks deployment hack is irreversible right > >now. This is not a reason to avoid using the merged-usr layout. > Just like Guillem, I'm personally not against merged-/usr but against how we > implement it. Not against, except for the directory symlinks being part of the merged-usr layout?
Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?
Johannes Schauer writes: > With how merged-/usr is getting deployed via debootstrap, we are placing more > of the instructions of how a Debian system is supposed to be setup *outside* > of > the packages themselves (and into debootstrap). It would make a cleaner design > if we could have all necessary definitions of how to create a Debian chroot > from scratch (on a Debian system) in the packages *themselves* instead of > requiring code around the packages that is doing some magic setup (usually > debootstrap). One could drop the magic setup if merged-/usr was mandatory: (1) have /bin -> /usr/bin symlinks shipped in base-files (2) have all packages install to /usr The additional complexity of having to setup symlinks outside the packaging system comes from supporting both merged and nonmerged setups. debootstrap or the usrmerge package implement some way to setup the symlink which would not be necessary if they were included in the packages; it would not be necessary if (1) was implemented (plus handling system upgrades from legacy installations). On merged-/usr systems, packages could just switch install location gradually; they would install to the same location anyway, no compat symlinks or maintainer script logic required. Ansgar
Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?
Hi,
On Sat, 02 Feb 2019 15:38:01 +0100 Didier 'OdyX' Raboud wrote:
> Gunnar and myself have started working on a draft, the latest version of
> which is available at
>
>
> https://salsa.debian.org/debian/tech-ctte/blob/master/914897_merged_usr/ballot.md
I found that some important arguments are still missing. A recent mail by
Guillem [1] nicely summarizes also many of my own thoughts. I'm going to paste
the relevant content into this mail for convenience of the reader:
[1] https://lists.debian.org/20190219044924.gb21...@gaara.hadrons.org
Quoting Guillem Jover (2019-02-19 05:49:24)
> The underlaying problem with the merged-/usr-via-symlinks is not
> really that it has generated bugs, many transitions we perform incur
> those, for the better. The problem is that it has generated those when
> it was really not necessary in the first place, has made things
> unnecessarily more complicated for everyone, and it's a terrible hack
> trying to reap "quick" benefits at the cost of everyhing else, with
> long lasting consequences even after a full migration to /usr would
> have been finished. And here's why:
>
> 1) The merged-/usr-via-symlinks deployment method used by both
>debootstrap and usrmerge, means that those systems will get
>permanent filesystem aliasing problems, forever. Even when all
>files might have been moved to their /usr counterparts in the
>packages! This is due to the fact that different pathnames can
>end up pointing (before any canonicalization!) to the same dentry.
>
>This does not only affect dpkg (dpkg, dpkg-divert, dpkg-trigger,
>etc), and update-alternatives (in a worse way), but any program
>that uses pathanmes in the filesystem as keys in databases f.ex.
>
> 2) It bypasses the packaging system understanding of what is on the
>filesystem and creates mismatches between what's on binary packages
>and what's on disk.
>
> 3) Switching packages to the merged-/usr layout could have been
>accomplished automatically via debhelper for a coverage of around
>99% (?) of the archive. With something along the lines of:
>
> ,---
> D=debian/tmp
> for d in /bin /sbin /lib; do
>for p in $(find $D/$d ! -type d); do
> b="${p##$D/}"
> m="$D/usr$b"
> if [ ! -e "$m" ]; then
>mkdir -p "$(dirname $m)"
>mv "$p" "$m"
>ln -s "${m##$D}" "$p"
> fi
>done
> done
> `---
>
>With the property that it would handle gracefully all cases were the
>maintainer has checked that no compat symlinks are required and has
>then progressively moved the pathname installation to their final
>destination under /usr.
>
> 4) Due to having to support the broken merged-/usr-via-symlinks
>deployment, when we want to move the contents of the binary
>packages to the merged-/usr layout, we require now to include tons
>of logic in (probably new) maintainer scripts, when we have been
>trying to remove them altogether. :( With even more files untracked
>by dpkg itself, bypassing the packaging system even more, when the
>compat symlinks could have been shipped in the binary packages.
>
> 5) Most new installations will not even benefit from this hacky and
>rushed deployment, as long as they do not use a separate parition
>for /usr!
>
> 6) The merged-/usr-via-symlinks deployment hack is irreversible right
>now.
Just like Guillem, I'm personally not against merged-/usr but against how we
implement it. In addition to Guillems arguments, I also want to make another.
With how merged-/usr is getting deployed via debootstrap, we are placing more
of the instructions of how a Debian system is supposed to be setup *outside* of
the packages themselves (and into debootstrap). It would make a cleaner design
if we could have all necessary definitions of how to create a Debian chroot
from scratch (on a Debian system) in the packages *themselves* instead of
requiring code around the packages that is doing some magic setup (usually
debootstrap). With mmdebstrap I am trying to write a POC to show that it is
possible to set up a Debian chroot with only very little magic and the help of
apt. My goal is to move the remaining magic setup that is currently required
into apt and dpkg (the respective maintainers are in favor of that plan but I'm
waiting for the buster release). If merged-/usr is getting implemented in the
way as it is done by debootstrap, then every debootstrap-like program has to
reproduce these mechanisms which are already architecture specific and thus not
trivial to maintain over time. It would be more elegant if the way merged-/usr
is being deployed is being encoded in the packages themselves so that we can
further *reduce* the amount of code that needs to live outside of packages
until we eventually need zero setup code before debootstrap-like programs can
install binary packages into a chroot. If we decide to deploy mer
