Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Gunnar Wolf]

Ansgar Burchardt dijo [Wed, Dec 05, 2018 at 08:17:56AM +0100]:
> The Reproducible Builds project was so kind to help and now runs one
> build in a non-merged-/usr and a second build in a merged-/usr
> environment.  Packages that hardcode the path to utilities, but would
> pick up the wrong one in a merged-/usr environment will result in a
> difference between the two builds and can thus be found.
> 
> See [1] for an overview of issues found this way; as the entire archive
> was already rebuilt in this setup, there shouldn't be many more issues
> of this type that we don't know about[2].
> (...)

Thanks for this report, Ansgar. This clearly sums up the issue. I am
amazed the whole archive managed to be rebuilt on such a short time!

I guess the next step is to file the multiple bugs pending to be
filed.

> Bug reports were already submitted for over half the packages, often
> including a simple patch (usually something like adding BASH=/bin/bash
> to dh_auto_configure).
> 
> So we look to be on a good track to address the remaining issues.

...And I guess this points towards the Technical Committee not having
to intervene in the issue. Which is, IMO, the best possible outcome.

Thanks to everybody following through with this!

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Ansgar Burchardt]

Hi,

Hideki Yamane writes:
> On Sun, 2 Dec 2018 15:15:21 +
> Simon McVittie  wrote:
>> >   - What is the problem? (broken build for which packages? Just R?)
>> 
>> The problem we're aware of is:
>> 
>> Some packages auto-detect the absolute path to an executable (for example
>> bash or perl) and hard-code it into their output (for example the #! line
>> of the bash scripts in quilt).
>
>  Can we check and track this behavior in our packages?

The Reproducible Builds project was so kind to help and now runs one
build in a non-merged-/usr and a second build in a merged-/usr
environment.  Packages that hardcode the path to utilities, but would
pick up the wrong one in a merged-/usr environment will result in a
difference between the two builds and can thus be found.

See [1] for an overview of issues found this way; as the entire archive
was already rebuilt in this setup, there shouldn't be many more issues
of this type that we don't know about[2].

Not all of these differences even cause issues as in a few packages the
utility with the hardcoded path is not even used at all.

Bug reports were already submitted for over half the packages, often
including a simple patch (usually something like adding BASH=/bin/bash
to dh_auto_configure).

So we look to be on a good track to address the remaining issues.

I don't think that the debootstrap default has to be reverted
temporarily again to deal with this: there are only very few packages
causing problems and these should have a patch soon.

In addition one has to actually built one of the very few packages in a
merged-/usr environment and then install them in a non-merged-/usr
environment to actually trigger the problem and debootstrap already
defaults to non-merged-usr for buildd chroots for now.

Ansgar

  [1] 
https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
  [2] https://bugs.debian.org/914897#81

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Hideki Yamane]

Hi,

 Thanks Simon, it's perhaps clear for me now.

On Sun, 2 Dec 2018 15:15:21 +
Simon McVittie  wrote:
> >   - What is the problem? (broken build for which packages? Just R?)
> 
> The problem we're aware of is:
> 
> Some packages auto-detect the absolute path to an executable (for example
> bash or perl) and hard-code it into their output (for example the #! line
> of the bash scripts in quilt).

 Can we check and track this behavior in our packages?

 Once disable merged-usr is good to prevent confusion but we detect such
 as a bug for manage non-merged-usr and merged-usr Debian system in the end,
 right? (or do you want to stay change in debootstrap 1.0.111 forever?)


-- 
Hideki Yamane 

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Holger Levsen]

On Sat, Dec 01, 2018 at 10:21:50PM +, Simon McVittie wrote:
> gzip, icecc and mailagent were most recently built for buster on
> 2018-11-08, which might be long enough ago that the buster chroot was
> not merged-/usr?

right. I triggered their builds and now they are all shown as unreproducible.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Simon McVittie]

On Sun, 02 Dec 2018 at 21:21:40 +0900, Hideki Yamane wrote:
>   - What is the problem? (broken build for which packages? Just R?)

The problem we're aware of is:

Some packages auto-detect the absolute path to an executable (for example
bash or perl) and hard-code it into their output (for example the #! line
of the bash scripts in quilt). When built on a system with merged /usr,
they can detect and hard-code a path that exists on merged /usr systems
but does not exist on systems with unmerged /usr, such as #!/usr/bin/bash
or #!/bin/perl.  This results in the package working fine on merged-/usr
systems but not on unmerged-/usr systems.

The quilt bug https://bugs.debian.org/913226 is the one that prompted
me to look at this (it caused build failures in a Debian derivative
that I work on, which is in the process of rebasing from stretch to
buster), and is a fairly typical example. The problem case usually
involves an executable that is canonically in /bin, like bash or sed,
being detected in /usr/bin. I've also seen one occurrence of the reverse,
where an executable that is canonically in /usr/bin is found in /bin on
a merged /usr system, although I can't remember which package that was
(it would have to have been re-ordering PATH to look in /bin first).

The same packages would be similarly broken if they were built with
/usr/local/bin in the PATH, on a system where the executable they are
looking for is present in /usr/local/bin. For example, versions of quilt
where #913226 is unfixed could exhibit a similar problem if built on a
system with a /usr/local/bin/bash.

The same things can happen with /usr/sbin and /sbin.

>   - How many packages are affected?

In ,
Ansgar points to 60 packages found to have this failure mode in a rebuild
covering around 80% of the archive, and estimates that there should be
about 80 packages in total affected by this class of bug.

>   - Why it was caused? (just symlink to /bin or /sbin isn't enough
> to deal with it?)

See the quilt bug I linked above, which is a typical example.

On merged /usr systems there is no problem with the "broken" packages,
because the compatibility symlinks /bin -> /usr/bin and /sbin -> /usr/sbin
ensure that paths like /bin/sh and /usr/bin/sh both work equally well.
The problem only occurs when they are *built* on a system *with* merged
/usr, then *used* on a system *without* merged /usr.

>   - Does it cause any failure on users' machine?

Yes, for example see the quilt bug. The failure mode is:

* A developer, D, has a system with merged /usr
* A user, U, has a system without merged /usr
* D builds a package that has this type of bug
  (without using a non-merged-/usr chroot, for example --variant=buildd from
  debootstrap >= 1.0.111)
* The package works for D
* The package doesn't work for U

smcv

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Hideki Yamane]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

On Fri, 30 Nov 2018 19:40:45 +0100
"Didier 'OdyX' Raboud"  wrote:
> tl;dr: debootstrap maintainers; can you agree to disable "merged /usr" by 
> default now, or are you OK letting the TC decide on this subject?

 Hmm, I'm still considering what's the good way...

> Hideki, if I read the debootstrap history correctly, you enabled "merged 
> /usr" 
> by default in debootstrap 1.0.102. 

 Yes, that's right. #839046 was filed in Sep 2016, and uploaded in Jun 2018.

> Given the recent discussion in debian-
> devel@ (starting at [0]) and on #914897, could you (or anyone speaking as 
> with 
> a "debootstrap maintainer" hat on) state if, either of:
> 
> * you would be willing to toggle the "merged /usr" default in debootstrap in a
>   subsequent upload;
> * you maintain that the "merged /usr" default (to yes) is here to stay.

 Well, with a quick look to the thread (I cannot follow all of the email in it,
 tons of emails...), I cannot find the discussion about

  - What is the problem? (broken build for which packages? Just R?)
  - How many packages are affected?
  - Why it was caused? (just symlink to /bin or /sbin isn't enough
to deal with it?)
  - Does it cause any failure on users' machine?

 So, I want to listen above things (not thought or idea), then reply to
 your question. Please quote if someone can do it.


- -- 
Regards,

 Hideki Yamane henrich @ debian.org/iijmio-mail.jp
 http://wiki.debian.org/HidekiYamane
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEQZYJUbYxgXxV33EdBBJ4KqpAHFMFAlwDzlUACgkQBBJ4KqpA
HFOnfxAAv8s9EQwntX9SBHALIY+5X1Ma98aMhrzZ2SWDt1txznXRt18z/85oOWqs
FGLrm2QY159qWEG2lpsWhAIr7wQJBPcFH5MRQcn6pDM6pXB1ioaTsW9uhd/AMl+s
mCyvWW0xtJ1ww2EXV2hN5X0K4AAre2rajb0P4p6efeY5V9sbMQ/gZa+L2sJuL1P/
/6fK4Kxe893lVuZ3oxtOhKRkdgi1V1X63kUURofuTSZiVzeGYWAuPdnHBxADs9vK
kk6mpUFkYSeOfg45h2KQzUqeTsX5GTogWIFqEOAJ0KJGDusOiFEPWL/pus+De1E7
cyEX2i6yq3wOOQBov5/eNH2gMs9pDaOqM8hR0tjvya4aAJOa7VyFY2GzMdsEHdQe
Ay7EtzG3RLwuiQ0XrSmIyaDdlJpofCGernNgVu+dnBJb/1U4RHgneVbIELULGUYm
DGFov6FpeUQB6wc/fsaoDWQBiwwNCS2qkJnZJg5nu4ne12NqnERqoq2lIR3ivSe2
1Oi9v/ClKqNSKGLAIoRVvllZhs9W1ppwkZIqtC0mZlN05nw7Wyrj4YoRbJ4r70Rd
rdQzTntchOXbYOmdt2H6yUdpnJJoA46+OxlwykvjrUnDnzgheNMJ0wRh36LcOz50
pjQBQGGVVl/9+Tjw/vSCu+alwLwPY34YFOM8I4fh/V0OHbO4fNE=
=yD0D
-END PGP SIGNATURE-

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Simon McVittie]

On Sat, 01 Dec 2018 at 17:18:35 +, Holger Levsen wrote:
> https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
> lists these packages.
> 
> what surprises me currently, are those 3 packages which are reproducible
> in buster (even though we also vary usrmerge when testing buster).

gzip, icecc and mailagent were most recently built for buster on
2018-11-08, which might be long enough ago that the buster chroot was
not merged-/usr?

smcv

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Holger Levsen]

Hi,

Ansgar, thanks a lot for doing this!

On Sat, Dec 01, 2018 at 06:06:28PM +0100, Ansgar Burchardt wrote:
> So, I went through all reproducible build failures in unstable without
> notes and added notes for differences caused by building in merged-/usr
> vs non-merged-/usr packages.  Together with what other people tagged,
> about 60 problems were found.  (The oldest rebuild result for unstable
> is 17 days old, the merged-/usr variation was deployed before that[1].)

https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
lists these packages.

what surprises me currently, are those 3 packages which are reproducible
in buster (even though we also vary usrmerge when testing buster).


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Ansgar Burchardt]

Ansgar Burchardt writes:
> There were discussions about enabling this by default years ago, I
> don't think minor issues should be a reason to delay this change.
>
> Note that it has been delayed for after the stretch release as there
> were major issues back then (it was enabled by default for a short time
> in stretch).  It took >5 months for someone to find a problem this time
> which is pretty good for any change.

So, I went through all reproducible build failures in unstable without
notes and added notes for differences caused by building in merged-/usr
vs non-merged-/usr packages.  Together with what other people tagged,
about 60 problems were found.  (The oldest rebuild result for unstable
is 17 days old, the merged-/usr variation was deployed before that[1].)

  [1] https://bugs.debian.org/901473#43

There might be some more that as diffoscope had problems or the diff was
too large for some packages (but I'll assume that in this case
merged-/usr is not the only problem the package had).

This should cover all packages that had no other problems and were
reproducible before, that is about 80% of the archive.

Assuming the rate is similar for packages which have other reproducible
build problems, I would expect 60 / 80*20 = 15 more problems.

I haven't looked at build failures, but I would expect these to be
usually not caused by merged-/usr.

So an estimated ~80 packages which might need adjustments for building
correctly in both merged-/usr and non-merged-/usr.  I guess less than
for the average new release of GCC ;-)

The problems are usually easy to fix by passing an explicit path the the
package's configure script at build time; of course there might be some
package where it is more complicated.

Ansgar

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Ansgar Burchardt]

On Fri, 2018-11-30 at 19:40 +0100, Didier 'OdyX' Raboud wrote:
> Dear Hideki, dear src:debootstrap maintainers,
> 
> tl;dr: debootstrap maintainers; can you agree to disable "merged /usr" by 
> default now, or are you OK letting the TC decide on this subject?

There were discussions about enabling this by default years ago, I
don't think minor issues should be a reason to delay this change.

Note that it has been delayed for after the stretch release as there
were major issues back then (it was enabled by default for a short time
in stretch).  It took >5 months for someone to find a problem this time
which is pretty good for any change.

Ansgar

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Didier 'OdyX' Raboud]

Dear Hideki, dear src:debootstrap maintainers,

tl;dr: debootstrap maintainers; can you agree to disable "merged /usr" by 
default now, or are you OK letting the TC decide on this subject?

Longer version:

As you might be aware, #914897 (initially filed on src:debootstrap) has now 
been reassigned to the Technical Committee.  As, formally, the Maintainer of 
src:debootstrap is "debian-boot@l.d.o and the current Uploaders", I would like 
to make sure that the TC is not going to overrule unnecessarily.

Hideki, if I read the debootstrap history correctly, you enabled "merged /usr" 
by default in debootstrap 1.0.102.  Given the recent discussion in debian-
devel@ (starting at [0]) and on #914897, could you (or anyone speaking as with 
a "debootstrap maintainer" hat on) state if, either of:

* you would be willing to toggle the "merged /usr" default in debootstrap in a
  subsequent upload;
* you maintain that the "merged /usr" default (to yes) is here to stay.

Many thanks in advance for your consideration,

OdyX

[0] https://lists.debian.org/debian-devel/2018/11/msg00354.html

P.S. I'm aware that this might sound formal, or dismissive of Julien's
 statements.  I just _really_ don't want the TC to eventually overrule
 "the debootstrap maintainers" without need.