Bug#914897: debootstrap, buster: Please disabled merged /usr by default
Hello Didier, Thanks for providing your reasoning. On Sun, Dec 02, 2018 at 01:28:20AM +0100, Didier 'OdyX' Raboud wrote: [...] > In other words, if the debootstrap maintainers now agree with Ian about > toggling the "merged-/usr" default back to false, there's really no point for > the TC to overrule unnecessarily, or even continue discussing this. FWIW If I where to receive the previous mail you adressed to debootstrap maintainers I would very likely have interpretted what you where saying more like: "I'm telling you with my TC hat on to change the default, or we will proceed to voting on overruling your decision anyway." ... which if I'm reading you correctly was not at all your intention. > And if they don't agree, their _current_ reasoning will provide value > to the discussion, and ideally provide for a better TC decision. Exactly what I was thinking. Might have been better with just asking for their current view on things and if they are planning on changing anything soon based on the recent discussions (and maybe also tell Ian to talk directly to the maintainer instead of using TC as a proxy). Regards, Andreas Henriksson
Bug#914897: debootstrap, buster: Please disabled merged /usr by default
Hello, On Sat, Dec 01, 2018 at 11:36:45PM +0100, Didier 'OdyX' Raboud wrote: > Le samedi, 1 décembre 2018, 19.29:59 h CET Marc Haber a écrit : > > This might sound like a stupid question, what will happen when a package > > built on a usrmerged System will be installed on a non-usrmerged system? > > FTR, I try to always assume that no question is stupid. Only answers can be. > > My understanding is that there's no universal answer to this; for a lot of > packages, the answer is "nothing"; it will just work. For some packages > though, if they embed executable paths such as /usr/bin/grep where only > /bin/grep exists on the host system, this will break. Or it could end up as /usr/local/bin/grep or as /home/randomuser/bin/grep just as well. This is really a generic problem of packages not being reproducible when embedding full path found in PATH or the build system. Even if debootstrap and usrmerge both where nuked into orbit the problem wouldn't go away! (This is not a problem for packages built on official buildds already. It's a problem when built on users machines in a "dirty" environment and then shared.) > > > Will binaries move from /usr/bin to /bin? Or will binaries move from > > /bin to /usr/bin? > > A merged-/usr has a /bin → /usr/bin symlink; so a .deb package unpacking > /bin/grep will make that binary end up in /usr/bin/grep; but the > /bin → /usr/bin symlink ensures that you can either access /usr/bin/grep > directly or /bin/grep through the symlink. The opposite can also happen as some packages fiddle with PATH (eg. by using AC_PATH_PROGs fourth argument) and you can thus end up embedding /bin/awk (as on usrmerged systems /bin -> /usr/bin), when a non-merged system only has /usr/bin/awk. > > An open question I think is whether dpkg should/could at one point in the > future always unpack binaries to /usr/bin and never in /bin no matter what > the > package contains (+ setup a convenience symlink in /bin if /bin is a real dir > ?). I don't see how this is any better than just using usrmerge. The long term solution, once noone any longer carries the opinion that non-merged systems needs to be supported, would be to rebuild packages so that the binary archives doesn't ship things in /bin et.al anymore. Then dpkg doesn't need to do anything (and dpkg-query will be all happy again). If you think modifying packages individually for this is to be avoided, then someone has already mentioned the idea of doing a dh_usrmerge that would relocate a packages files before the binary package is compressed. Regards, Andreas Henriksson
Bug#914897: debootstrap, buster: Please disabled merged /usr by default
On Sat, Dec 01, 2018 at 07:20:57PM +0100, Didier 'OdyX' Raboud wrote:
> As a way to improve my understanding of the issue at hand, here's my current
> collection of data points regarding the "merged-/usr" question:
>
> * "merged-/usr" was enabled by default in debootstrap on June 13. 2018, some
> 5+ months ago. Any buster rootfs setup since has the /bin → /usr/bin
> symlink (and other caracteristics of merged-/usr).
>
>
> https://tracker.debian.org/news/965045/accepted-debootstrap-10102-source-all-into-unstable/
For completeness sake you might want to start the story at the
point in pre-stretch-freeze era where the default was changed:
https://tracker.debian.org/news/807889/accepted-debootstrap-1085-source-into-unstable/
And then disabled and post-postponed to post-stretch release:
https://tracker.debian.org/news/815610/accepted-debootstrap-1087-source-all-into-unstable/
>
> * With my TC hat on, I have formally asked the debootstrap maintainers
> (and specifically Hideki Yamane) if they would consider a toggle of the
> default.
>
> https://bugs.debian.org/914897#73
May I ask you to share why you did that? I'm interested to know the
reasoning.
(It's not my understanding that this is how TC normally operates.)
>
> * a discussion "usrmerge -- plan B?" was started on Nov. 20 on debian-
> devel@l.d.o
>
> https://lists.debian.org/20181120211617.gxnuwxpx2hy44...@angband.pl
>
> It apparently follows #913229, #914204 and others.
>
> * Currently, according to my `apt-file`, 259 binaries are shipped in /bin
> directly, accross 85 packages. (for /sbin, 597 binaries for 190 packages).
>
> * There exists a 'usrmerge' package since 2015, which transforms a rootfs
> with
> separate /{bin,sbin,lib} into a "merged-usr/" rootfs. It has had 28 bugs
> over its lifetime; of which 4 are currently open. After successful
> "installation" (when the postinst successfully ran the
> /usr/lib/convert-usrmerge program), /{bin,sbin,lib} are made symlinks and
> the package can be removed. The package doesn't include a way to revert the
> rootfs to its previous state.
>
> * Building source packages in a merged-/usr rootfs can make the resulting
> binary packages broken in non-merged-/usr rootfs'es, because they'd be
> embedding references to /usr/bin/grep (e.g.), which don't exist in non-
> merged-/usr rootfs'es.
--->
> To ensure this doesn't happen for packages built on
> Debian infrastructure, debootstrap has been updated to disable merged-/usr
> for the `buildd` variant (and buildd chroots have been rebuilt).
... while that change happened, I don't think that's what technically
disabled merged-usr on the debian buildd chroots. AIUI the buildd
chroots are created using debootstrap from stable-backports repository.
There was a new upload to stable-backports that enabled the merged-usr
by default which triggered a few problems. The admins thus explicitly
disabled merged-usr via the debootstrap commandline flag in the
continuous deployment system for buildd chroots and triggered chroot
recreation jobs.
>
> * According to investigation done thanks to reproducible builds (which now
> also varies the merged-/usr state of the build rootfs), and by others,
> packages affected by this problem have begun receiving either
> reproducibility issue tags or Debian bugs:
>
>
> https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
> (currently: 59)
>
>
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=m...@linux.it;tag=usrmerge
> (current total: 81; outstanding: 17)
FWIW I've analyzed all packages that are tagged on the reproducible
build page. Messing with the debian bts is just too time consuming as
if looking at this wasn't boring enough to begin with. If anyone thinks
there are some interesting data points to be found please ask.
>
> I'll post my thoughts separately; please enhance or correct the above data
> as needed!
>
> Cheers,
> OdyX
Regards,
Andreas Henriksson
Bug#914897: debootstrap, buster: Please disabled merged /usr by default
Le samedi, 1 décembre 2018, 19.29:59 h CET Marc Haber a écrit : > This might sound like a stupid question, what will happen when a package > built on a usrmerged System will be installed on a non-usrmerged system? FTR, I try to always assume that no question is stupid. Only answers can be. My understanding is that there's no universal answer to this; for a lot of packages, the answer is "nothing"; it will just work. For some packages though, if they embed executable paths such as /usr/bin/grep where only /bin/grep exists on the host system, this will break. > Will binaries move from /usr/bin to /bin? Or will binaries move from > /bin to /usr/bin? A merged-/usr has a /bin → /usr/bin symlink; so a .deb package unpacking /bin/grep will make that binary end up in /usr/bin/grep; but the /bin → /usr/bin symlink ensures that you can either access /usr/bin/grep directly or /bin/grep through the symlink. An open question I think is whether dpkg should/could at one point in the future always unpack binaries to /usr/bin and never in /bin no matter what the package contains (+ setup a convenience symlink in /bin if /bin is a real dir ?). Cheers, OdyX signature.asc Description: This is a digitally signed message part.
Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default
On Sat, 01 Dec 2018 at 17:18:35 +, Holger Levsen wrote: > https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html > lists these packages. > > what surprises me currently, are those 3 packages which are reproducible > in buster (even though we also vary usrmerge when testing buster). gzip, icecc and mailagent were most recently built for buster on 2018-11-08, which might be long enough ago that the buster chroot was not merged-/usr? smcv
Bug#914897: debootstrap, buster: Please disabled merged /usr by default
Marc Haber writes:
> On Sat, Dec 01, 2018 at 07:20:57PM +0100, Didier 'OdyX' Raboud wrote:
>> * Currently, according to my `apt-file`, 259 binaries are shipped in /bin
>> directly, accross 85 packages. (for /sbin, 597 binaries for 190 packages).
>
> This might sound like a stupid question, what will happen when a package
> built on a usrmerged System will be installed on a non-usrmerged system?
>
> Will binaries move from /usr/bin to /bin? Or will binaries move from
> /bin to /usr/bin?
They will be installed in the place the package installs them. Note
that there are no /bin -> /usr/bin symlink in the staging area where the
package is built (i.e. debian/${package} or debian/tmp).
Packages have to continue shipping binaries in /bin unless we decide to
make merged-/usr mandatory and convert existing systems.
Ansgar
Bug#914897: debootstrap, buster: Please disabled merged /usr by default
On Sat, Dec 01, 2018 at 07:20:57PM +0100, Didier 'OdyX' Raboud wrote:
> * Currently, according to my `apt-file`, 259 binaries are shipped in /bin
> directly, accross 85 packages. (for /sbin, 597 binaries for 190 packages).
This might sound like a stupid question, what will happen when a package
built on a usrmerged System will be installed on a non-usrmerged system?
Will binaries move from /usr/bin to /bin? Or will binaries move from
/bin to /usr/bin?
The former is likely to cause fun on upgrades, since root file systems
on systems that were installed with /usr on its own partition tend to be
rather smallish. Dumping the entire former content of /usr/{bin,sbin} to
/{bin,sbin} is likely to wreck those systems, most of them probably
being either embedded or in big datacenters where people usually don't
waste disk space for root file systems.
Greetings
Marc
--
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#914897: debootstrap, buster: Please disabled merged /usr by default
As a way to improve my understanding of the issue at hand, here's my current
collection of data points regarding the "merged-/usr" question:
* "merged-/usr" was enabled by default in debootstrap on June 13. 2018, some
5+ months ago. Any buster rootfs setup since has the /bin → /usr/bin
symlink (and other caracteristics of merged-/usr).
https://tracker.debian.org/news/965045/accepted-debootstrap-10102-source-all-into-unstable/
* With my TC hat on, I have formally asked the debootstrap maintainers
(and specifically Hideki Yamane) if they would consider a toggle of the
default.
https://bugs.debian.org/914897#73
* a discussion "usrmerge -- plan B?" was started on Nov. 20 on debian-
devel@l.d.o
https://lists.debian.org/20181120211617.gxnuwxpx2hy44...@angband.pl
It apparently follows #913229, #914204 and others.
* Currently, according to my `apt-file`, 259 binaries are shipped in /bin
directly, accross 85 packages. (for /sbin, 597 binaries for 190 packages).
* There exists a 'usrmerge' package since 2015, which transforms a rootfs with
separate /{bin,sbin,lib} into a "merged-usr/" rootfs. It has had 28 bugs
over its lifetime; of which 4 are currently open. After successful
"installation" (when the postinst successfully ran the
/usr/lib/convert-usrmerge program), /{bin,sbin,lib} are made symlinks and
the package can be removed. The package doesn't include a way to revert the
rootfs to its previous state.
* Building source packages in a merged-/usr rootfs can make the resulting
binary packages broken in non-merged-/usr rootfs'es, because they'd be
embedding references to /usr/bin/grep (e.g.), which don't exist in non-
merged-/usr rootfs'es. To ensure this doesn't happen for packages built on
Debian infrastructure, debootstrap has been updated to disable merged-/usr
for the `buildd` variant (and buildd chroots have been rebuilt).
* According to investigation done thanks to reproducible builds (which now
also varies the merged-/usr state of the build rootfs), and by others,
packages affected by this problem have begun receiving either
reproducibility issue tags or Debian bugs:
https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
(currently: 59)
https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=m...@linux.it;tag=usrmerge
(current total: 81; outstanding: 17)
I'll post my thoughts separately; please enhance or correct the above data
as needed!
Cheers,
OdyX
signature.asc
Description: This is a digitally signed message part.
Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default
Hi, Ansgar, thanks a lot for doing this! On Sat, Dec 01, 2018 at 06:06:28PM +0100, Ansgar Burchardt wrote: > So, I went through all reproducible build failures in unstable without > notes and added notes for differences caused by building in merged-/usr > vs non-merged-/usr packages. Together with what other people tagged, > about 60 problems were found. (The oldest rebuild result for unstable > is 17 days old, the merged-/usr variation was deployed before that[1].) https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html lists these packages. what surprises me currently, are those 3 packages which are reproducible in buster (even though we also vary usrmerge when testing buster). -- cheers, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default
Ansgar Burchardt writes: > There were discussions about enabling this by default years ago, I > don't think minor issues should be a reason to delay this change. > > Note that it has been delayed for after the stretch release as there > were major issues back then (it was enabled by default for a short time > in stretch). It took >5 months for someone to find a problem this time > which is pretty good for any change. So, I went through all reproducible build failures in unstable without notes and added notes for differences caused by building in merged-/usr vs non-merged-/usr packages. Together with what other people tagged, about 60 problems were found. (The oldest rebuild result for unstable is 17 days old, the merged-/usr variation was deployed before that[1].) [1] https://bugs.debian.org/901473#43 There might be some more that as diffoscope had problems or the diff was too large for some packages (but I'll assume that in this case merged-/usr is not the only problem the package had). This should cover all packages that had no other problems and were reproducible before, that is about 80% of the archive. Assuming the rate is similar for packages which have other reproducible build problems, I would expect 60 / 80*20 = 15 more problems. I haven't looked at build failures, but I would expect these to be usually not caused by merged-/usr. So an estimated ~80 packages which might need adjustments for building correctly in both merged-/usr and non-merged-/usr. I guess less than for the average new release of GCC ;-) The problems are usually easy to fix by passing an explicit path the the package's configure script at build time; of course there might be some package where it is more complicated. Ansgar
