Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default
Hi, Hideki Yamane writes: > On Sun, 2 Dec 2018 15:15:21 + > Simon McVittie wrote: >> > - What is the problem? (broken build for which packages? Just R?) >> >> The problem we're aware of is: >> >> Some packages auto-detect the absolute path to an executable (for example >> bash or perl) and hard-code it into their output (for example the #! line >> of the bash scripts in quilt). > > Can we check and track this behavior in our packages? The Reproducible Builds project was so kind to help and now runs one build in a non-merged-/usr and a second build in a merged-/usr environment. Packages that hardcode the path to utilities, but would pick up the wrong one in a merged-/usr environment will result in a difference between the two builds and can thus be found. See [1] for an overview of issues found this way; as the entire archive was already rebuilt in this setup, there shouldn't be many more issues of this type that we don't know about[2]. Not all of these differences even cause issues as in a few packages the utility with the hardcoded path is not even used at all. Bug reports were already submitted for over half the packages, often including a simple patch (usually something like adding BASH=/bin/bash to dh_auto_configure). So we look to be on a good track to address the remaining issues. I don't think that the debootstrap default has to be reverted temporarily again to deal with this: there are only very few packages causing problems and these should have a patch soon. In addition one has to actually built one of the very few packages in a merged-/usr environment and then install them in a non-merged-/usr environment to actually trigger the problem and debootstrap already defaults to non-merged-usr for buildd chroots for now. Ansgar [1] https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html [2] https://bugs.debian.org/914897#81
Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default
Hi, Thanks Simon, it's perhaps clear for me now. On Sun, 2 Dec 2018 15:15:21 + Simon McVittie wrote: > > - What is the problem? (broken build for which packages? Just R?) > > The problem we're aware of is: > > Some packages auto-detect the absolute path to an executable (for example > bash or perl) and hard-code it into their output (for example the #! line > of the bash scripts in quilt). Can we check and track this behavior in our packages? Once disable merged-usr is good to prevent confusion but we detect such as a bug for manage non-merged-usr and merged-usr Debian system in the end, right? (or do you want to stay change in debootstrap 1.0.111 forever?) -- Hideki Yamane
Bug#914897: debating the wrong thing
Adam Borowski writes: > On Tue, Dec 04, 2018 at 06:48:45PM +0100, Ansgar Burchardt wrote: >> In very rare cases (an estimated 0.3% of the archive or so). I'm fairly >> confident that for more than 0.3% of the archive something can go wrong >> when building in non-clean environments. > > Your figure of ~80 packages counts only packages which went through a > reproducible-builds rebuild. So only all? > We later learned only a part of the archive got rebuilt since the bad > debootstrap backport. Yes, some packages were not yet rebuilt in testing, but having them rebuilt in unstable is enough. >> We had the discussion (usrmerge as default) already quite some time >> ago. Why start again at zero? As a random reference, the D-I Stretch >> RC 1 release announcement explicitly stated: >> >> +--- >> | * The switch to merged-/usr as the default setting for debootstrap >> |was reverted since it uncovered a number of important issues which >> |might not be all fixed in time for stretch. This setting is >> |expected to come back as the default at the beginning of the next >> |release cycle. >> +--- >> >> And just that happened (except a bit later). > > Except that the change went live only on 2018-11-10, then waited until > buildds recreated their chroots, then until dinstall and mirror push... No, anyone using testing/unstable to setup a new build chroot since June should have gotten a merged-/usr chroot. That no issues were found earlier is probably related to there being not so many issues. (Fun fact: there are ~3k debootstrap installations on testing/unstable, compared to 6.2k on stable and 2k on oldstable according to popcon.) Anyway, buildds are not using merged-/usr, so there is no problem with them. > and the sky started falling immediately after that. Hmm, two packages or so were reported to be broken. That is a quite high standard for "sky falling". What would you call an upload that breaks more packages? The monthly apocalypse which we deal with just fine? >> You could have easily asked the tech-ctte back then (or even before) >> instead of waiting until shortly before the Buster freeze and after >> people invested more work. > > It was only shortly before the Buster freeze that we saw this change in > action! Had the switch get flipped sooner we'd have a chance to see the > results. By now, it's much too late to fix things before the freeze > (and I don't see how they could be fixed even had we two years of > time). You keep saying that it is too late to fix anything or that it is too much work, but why do you think so? I've looked at patching packages and how many packages need changes and it does not seem much work; indeed after a week about half of the packages already have a (usually trivial) patch. If you think it is too much work or too many things break, please at least give an estimate of what you are talking about... I feel like only one side is doing any research here. > By now, all we can do is damage control. Which can be a hasty force-merge > or a hasty revert. Unless you can somehow make dpkg-dev omniscient. If we keep merged-/usr as default then we can /recommend/ people to install usrmerge to switch to merged-/usr; reducing the difference between newly-installed and existing setups is a good idea IMHO. I think I filed a report for this two years ago. Maybe we should also mention somewhere that it might be useful to not switch build environments (yet). Ansgar
Bug#914897: debating the wrong thing
On Tue, Dec 04, 2018 at 07:21:11PM +0100, Adam Borowski wrote: > Your figure of ~80 packages counts only packages which went through a > reproducible-builds rebuild. We later learned only a part of the archive > got rebuilt since the bad debootstrap backport. wrong, sigh. -- cheers, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Bug#914897: debating the wrong thing
On Tue, Dec 04, 2018 at 06:48:45PM +0100, Ansgar Burchardt wrote:
> Ian Jackson writes:
> > Ansgar Burchardt writes ("Bug#914897: debating the wrong thing"):
> >> Switching to (1) or (3a-with-no-support-in-buster) will mean merged-/usr
> >> systems would no longer be supported. In this case someone would have
> >> to write a unusrmerge program to convert systems with merged-/usr to
> >> systems with unmerged-/usr.
> >
> > Currently merged-usr is broken because it can build packages which do
> > not work on non-merged-usr systems.
>
> In very rare cases (an estimated 0.3% of the archive or so). I'm fairly
> confident that for more than 0.3% of the archive something can go wrong
> when building in non-clean environments.
Your figure of ~80 packages counts only packages which went through a
reproducible-builds rebuild. We later learned only a part of the archive
got rebuilt since the bad debootstrap backport.
> What is technically and socially wrong is reverting a change after a
> small issue was found which is already in the process of getting fixed
> for most packages.
Making packages with non-Debian origin randomly break is not a "small
issue". Remember, we can fix only packages we make ourselves -- not any of
private/PPA/company-wide packages so many of our users make.
> > When we have stopped generating more lossage, we can start to think
> > about whether we want to transition to usrmerge as default, whether to
> > make it mandatory, and if so how the transition should be handled, and
> > on what timescale.
>
> We had the discussion (usrmerge as default) already quite some time
> ago. Why start again at zero? As a random reference, the D-I Stretch
> RC 1 release announcement explicitly stated:
>
> +---
> | * The switch to merged-/usr as the default setting for debootstrap
> |was reverted since it uncovered a number of important issues which
> |might not be all fixed in time for stretch. This setting is
> |expected to come back as the default at the beginning of the next
> |release cycle.
> +---
>
> And just that happened (except a bit later).
Except that the change went live only on 2018-11-10, then waited until
buildds recreated their chroots, then until dinstall and mirror push...
and the sky started falling immediately after that.
We train DDs to not upload packages built in an unclean environment -- I
made ~1000 uploads (mostly sponsored) which did not include a _single_
unclean build. I expect most DDs to be alike, and most of us also do
source-only except to NEW.
We tend to build outside a chroot only during development. Of course we do
test those packages but almost always on the same machine we're sitting at.
Which happens to match the usrmerge status of the package being tested...
> You could have easily asked the tech-ctte back then (or even before)
> instead of waiting until shortly before the Buster freeze and after
> people invested more work.
It was only shortly before the Buster freeze that we saw this change in
action! Had the switch get flipped sooner we'd have a chance to see the
results. By now, it's much too late to fix things before the freeze
(and I don't see how they could be fixed even had we two years of time).
> There is no such crisis. There was also enough time to discuss this in
> the last years or even since June (when it was enabled by default
> again)...
Nope, it got enabled very recently. This is a case where religiously
building stuff in a clean environment bit us -- because _that_ environment
wasn't changed.
By now, all we can do is damage control. Which can be a hasty force-merge
or a hasty revert. Unless you can somehow make dpkg-dev omniscient.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Ivan was a worldly man: born in St. Petersburg, raised in
⢿⡄⠘⠷⠚⠋⠀ Petrograd, lived most of his life in Leningrad, then returned
⠈⠳⣄ to the city of his birth to die.
Bug#914897: debating the wrong thing
Ian Jackson writes:
> Ansgar Burchardt writes ("Bug#914897: debating the wrong thing"):
>> Switching to (1) or (3a-with-no-support-in-buster) will mean merged-/usr
>> systems would no longer be supported. In this case someone would have
>> to write a unusrmerge program to convert systems with merged-/usr to
>> systems with unmerged-/usr.
>
> Currently merged-usr is broken because it can build packages which do
> not work on non-merged-usr systems.
In very rare cases (an estimated 0.3% of the archive or so). I'm fairly
confident that for more than 0.3% of the archive something can go wrong
when building in non-clean environments.
> It would be quite wrong (both technically and socially) to react to
> this lack of foresight,
The reported problems are not really unexpected...
> lack of consultation,
It was discussed on -devel@ several times. I think LWN also reported
about merged-/usr developments in Debian more than once, it wasn't a
secret cabal development.
So where is the lack of consultation?
> and lack of testing, by pressing forward.
It has been tested for quite a while.
A helpful new test was recently added by the Reproducible Builds project
which allowed to identify most problematic packages.
What is technically and socially wrong is reverting a change after a
small issue was found which is already in the process of getting fixed
for most packages.
> When we have stopped generating more lossage, we can start to think
> about whether we want to transition to usrmerge as default, whether to
> make it mandatory, and if so how the transition should be handled, and
> on what timescale.
We had the discussion (usrmerge as default) already quite some time
ago. Why start again at zero? As a random reference, the D-I Stretch
RC 1 release announcement explicitly stated:
+---
| * The switch to merged-/usr as the default setting for debootstrap
|was reverted since it uncovered a number of important issues which
|might not be all fixed in time for stretch. This setting is
|expected to come back as the default at the beginning of the next
|release cycle.
+---
And just that happened (except a bit later).
You could have easily asked the tech-ctte back then (or even before)
instead of waiting until shortly before the Buster freeze and after
people invested more work.
Making it mandatory or not is an unrelated decision. That can easily
just come later.
> We need the space to discuss those options properly without being
> distracted by what is IMO currently a crisis in stretch-backports and
> buster.
There is no such crisis. There was also enough time to discuss this in
the last years or even since June (when it was enabled by default
again)...
Ansgar
Bug#914897: debating the wrong thing
Ansgar Burchardt writes ("Bug#914897: debating the wrong thing"):
> Switching to (1) or (3a-with-no-support-in-buster) will mean merged-/usr
> systems would no longer be supported. In this case someone would have
> to write a unusrmerge program to convert systems with merged-/usr to
> systems with unmerged-/usr.
Currently merged-usr is broken because it can build packages which do
not work on non-merged-usr systems. It would be quite wrong (both
technically and socially) to react to this lack of foresight, lack of
consultation, and lack of testing, by pressing forward.
What I am suggesting in this bug report is that we revert to the
status quo before the default was changed to usrmerge, that is:
[Adam:]
>> 2. supporting both merged-usr and unmerged-usr
But actually of course "supporting" it in the way that it is currently
"supported" (according to usrmerge proponents) in stretch, sid, and
buster: if you enable it you may build packages which are not
generally useable (and perhaps you may experience other bugs).
This question is urgent for buster because the longer the current
situation continues the more systems there are that build broken
packages.
It is also urgent for stretch-backports. The backports maintainers
have said that they want to keep stretch-backports in line with
buster. Regardless of the wisdom of that policy, the current
situation in stretch-backports seems very bad to me.
The easiest way to fix stretch-backports (without also generating a
need to persuade the backports maintainers to waive their usual
policies) is to revert buster.
When we have stopped generating more lossage, we can start to think
about whether we want to transition to usrmerge as default, whether to
make it mandatory, and if so how the transition should be handled, and
on what timescale.
We have at least two sketches of transitions plans. That longer-term
conversation is a much more complicated one with many more options and
many more factors.
We need the space to discuss those options properly without being
distracted by what is IMO currently a crisis in stretch-backports and
buster.
Ian.
Bug#914897: debating the wrong thing
"Alexander E. Patrakov" writes: > Ansgar Burchardt wrote: >> Making the feature default was discussed years ago which you are surely >> aware of. It's not mandatory. > > Unfortunately I have to disagree here. Merged /usr is already, > de-facto, mandatory for everyone who installs Debian Testing using the > official netinst CD. Yes, but the "make merged-/usr mandatory" refers only to require to merge /usr on upgrades; nobody has asked for there to be an installer option (and I don't think it would be useful). Ansgar
