Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Santiago Vila]

Hello Luca.

Thanks a lot for implementing this!

I'm going to answer to an old message of yours, because
I think that things have changed a little bit since then.

El 18/10/23 a las 19:17, Luca Boccassi escribió:

We can do an upload, but note that it won't have any effect on package
builds, given the buildds use stable/oldstable - and this is not
material for p-u, given the effect. [...]


That was the prudent thing to do, indeed, when the change affected
all suites (oldstable, stable, testing, unstable, etc).

However, I see that the latest git version has introduced
a small change so that only trixie and above are affected:

https://salsa.debian.org/installer-team/debootstrap/-/commit/3c71d6473f746cbe15a6fa1d1cf6950773c432ee

which I agree that it makes sense to avoid breaking builds for bookworm and 
below.

As a result, the new debootstrap behaviour currently in git would be no longer
dangerous to have in bookworm (for chroots of trixie and above), and now
I believe it would be not only suitable but highly beneficial for 
proposed-updates
(to be included in a future point release).

What would be required for that to happen? As Sebastian pointed out,
this is the right time in the release cycle to do this kind
of "potentially breaking changes", and compared to the usr-merge
changes that have been already put in proposed-updates, I would say
this one would be way less troublesome.

Thanks.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Quoting Santiago Vila (2023-10-30 21:53:59)
> El 30/10/23 a las 21:16, Johannes Schauer Marin Rodrigues escribió:
> > Quoting Luca Boccassi (2023-10-18 19:17:40)
> >> We can do an upload, but note that it won't have any effect on package
> >> builds, given the buildds use stable/oldstable
> > 
> > actually we forgot something here. The upload *does* have an effect on 
> > buildds
> > right now even before Trixie gets released because riscv buildds (in 
> > contrast
> > to the others) do run debootstrap from unstable, resulting in this recent 
> > build
> > failure of src:siridb-server:
> > 
> > https://buildd.debian.org/status/fetch.php?pkg=siridb-server=riscv64=2.0.48-1%2Bb1=1698686860=0
> > 
> > This was reported already by Santiago as #1027381 last year and Paul Gevers
> > quickly did another upload of src:siridb-server fixing this.
> > 
> > So if riscv keeps being part of the release arches for trixie, then the 
> > FTBFS
> > bugs reported by Santiago will have real effects on buildds building 
> > packages
> > for riscv going forward. So unless that change gets reverted in debootstrap,
> > this class of bugs will likely go away by itself before trixie gets 
> > released.
> 
> So, if I understood correctly, if a package has this bug, and it's already
> reported, and the maintainer does a new upload but they forget to fix the bug,
> then the package will FTBFS on riscv, and as a result, the package will not
> propagate to testing.

this will of course also happen to packages that have the bug even if it's not
reported. Otherwise: yes.

> This is actually a good thing, it means the bugs are "factually RC".

I agree.

> So, while I personally don't see a special hurry to raise the severities of
> the currently reported bugs, maybe it makes sense that we start to report
> *new* bugs like this one as serious.

I think that would make sense, given that this currently blocks package
migration to testing due to factual build failures on riscv buildds.

In #debian-release, aurel32 asked about my opinion on whether those bugs should
now be raised to RC and I agreed that they should.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Santiago Vila]

El 30/10/23 a las 21:16, Johannes Schauer Marin Rodrigues escribió:

Quoting Luca Boccassi (2023-10-18 19:17:40)

We can do an upload, but note that it won't have any effect on package
builds, given the buildds use stable/oldstable


actually we forgot something here. The upload *does* have an effect on buildds
right now even before Trixie gets released because riscv buildds (in contrast
to the others) do run debootstrap from unstable, resulting in this recent build
failure of src:siridb-server:

https://buildd.debian.org/status/fetch.php?pkg=siridb-server=riscv64=2.0.48-1%2Bb1=1698686860=0

This was reported already by Santiago as #1027381 last year and Paul Gevers
quickly did another upload of src:siridb-server fixing this.

So if riscv keeps being part of the release arches for trixie, then the FTBFS
bugs reported by Santiago will have real effects on buildds building packages
for riscv going forward. So unless that change gets reverted in debootstrap,
this class of bugs will likely go away by itself before trixie gets released.


So, if I understood correctly, if a package has this bug, and it's already
reported, and the maintainer does a new upload but they forget to fix the bug,
then the package will FTBFS on riscv, and as a result, the package will not
propagate to testing.

This is actually a good thing, it means the bugs are "factually RC".

So, while I personally don't see a special hurry to raise the severities
of the currently reported bugs, maybe it makes sense that we start to
report *new* bugs like this one as serious.

Thanks.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2023-10-18 19:17:40)
> We can do an upload, but note that it won't have any effect on package
> builds, given the buildds use stable/oldstable

actually we forgot something here. The upload *does* have an effect on buildds
right now even before Trixie gets released because riscv buildds (in contrast
to the others) do run debootstrap from unstable, resulting in this recent build
failure of src:siridb-server:

https://buildd.debian.org/status/fetch.php?pkg=siridb-server=riscv64=2.0.48-1%2Bb1=1698686860=0

This was reported already by Santiago as #1027381 last year and Paul Gevers
quickly did another upload of src:siridb-server fixing this.

So if riscv keeps being part of the release arches for trixie, then the FTBFS
bugs reported by Santiago will have real effects on buildds building packages
for riscv going forward. So unless that change gets reverted in debootstrap,
this class of bugs will likely go away by itself before trixie gets released.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Hi Luca,

Quoting Luca Boccassi (2023-10-18 19:38:16)
> Thanks - I'll merge and do an upload over the weekend then

thank you for your upload!! :D

I temporarily subscribed to debootstrap via tracker.d.o so that I hopefully
catch any bug report coming in related to this change.

I also re-triggered some autopkgtests that were flaky and where re-running them
now makes all the reverse dependencies work.

Well, not all the reverse dependencies. The autopkgtest of mmdebstrap broke as
expected, which is of course also good because it matches our expectations.

I have the fix for this locally and am currently running the mmdebstrap test
suite with the fix I posted in my last mail. This made me find another change
that was necessary. /usr/share/debootstrap/scripts/debian-common sets up
/etc/localtime if it doesn't exist yet. This code-path was probably not taken
so far because tzdata was always installed? In any case, I think this is
harmless and doesn't require any changes.

Then I worried that we forgot to update the man page after this change.
Luckily, the man page only points out that the buildd profile installs
build-essential and does not talk about priority:required packages, so that
checks out. What could maybe be added is, that the buildd variant installs
build-essential *and* apt which is the remaining exception to the rule.

I will upload mmdebstrap with the required changes after the test suite
finished in about six hours.

Thanks again!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Luca Boccassi]

On Wed, 18 Oct 2023 at 18:36, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi Luca,
>
> Quoting Luca Boccassi (2023-10-18 19:17:40)
> > On Wed, 18 Oct 2023 at 18:04, Johannes Schauer Marin Rodrigues
> >  wrote:
> > >
> > > Hi,
> > >
> > > Quoting Santiago Vila (2023-10-12 17:56:04)
> > > > Johannes has asked the RMs in this thread:
> > > >
> > > > https://lists.debian.org/debian-release/2023/10/msg00425.html
> > > >
> > > > if they are ready to consider the bugs as RC. I believe it would be 
> > > > better
> > > > if we can make the bugs "factually" RC by uploading the fixed 
> > > > debootstrap first.
> > >
> > > I do not have a strong opinion on what should happen first but in that 
> > > thread,
> > > Holger and Sam also support the idea to first upload debootstrap.
> >
> > We can do an upload, but note that it won't have any effect on package
> > builds, given the buildds use stable/oldstable - and this is not
> > material for p-u, given the effect. Of course it will affect local
> > builds, in case they are done via debootstrap, from testing/unstable
> > users.
>
> Yes, I'm aware of that. But I think having this in unstable/testing already
> will help because several maintainers replied to the bugs saying they are
> unable to reproduce it. Having debootstrap with this change in 
> unstable/testing
> will make this a) much easier and b) convince people that they need to include
> this change or otherwise their package will really FTBFS on the buildds with
> the trixie release.
>
> And this should probably go without saying but just to make sure: if this
> change causes any weird bugs, please message me so that I can supply a fix.

Thanks - I'll merge and do an upload over the weekend then

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Hi Luca,

Quoting Luca Boccassi (2023-10-18 19:17:40)
> On Wed, 18 Oct 2023 at 18:04, Johannes Schauer Marin Rodrigues
>  wrote:
> >
> > Hi,
> >
> > Quoting Santiago Vila (2023-10-12 17:56:04)
> > > Johannes has asked the RMs in this thread:
> > >
> > > https://lists.debian.org/debian-release/2023/10/msg00425.html
> > >
> > > if they are ready to consider the bugs as RC. I believe it would be better
> > > if we can make the bugs "factually" RC by uploading the fixed debootstrap 
> > > first.
> >
> > I do not have a strong opinion on what should happen first but in that 
> > thread,
> > Holger and Sam also support the idea to first upload debootstrap.
> 
> We can do an upload, but note that it won't have any effect on package
> builds, given the buildds use stable/oldstable - and this is not
> material for p-u, given the effect. Of course it will affect local
> builds, in case they are done via debootstrap, from testing/unstable
> users.

Yes, I'm aware of that. But I think having this in unstable/testing already
will help because several maintainers replied to the bugs saying they are
unable to reproduce it. Having debootstrap with this change in unstable/testing
will make this a) much easier and b) convince people that they need to include
this change or otherwise their package will really FTBFS on the buildds with
the trixie release.

And this should probably go without saying but just to make sure: if this
change causes any weird bugs, please message me so that I can supply a fix.

Thank you!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Luca Boccassi]

On Wed, 18 Oct 2023 at 18:04, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> Quoting Santiago Vila (2023-10-12 17:56:04)
> > Johannes has asked the RMs in this thread:
> >
> > https://lists.debian.org/debian-release/2023/10/msg00425.html
> >
> > if they are ready to consider the bugs as RC. I believe it would be better
> > if we can make the bugs "factually" RC by uploading the fixed debootstrap 
> > first.
>
> I do not have a strong opinion on what should happen first but in that thread,
> Holger and Sam also support the idea to first upload debootstrap.

We can do an upload, but note that it won't have any effect on package
builds, given the buildds use stable/oldstable - and this is not
material for p-u, given the effect. Of course it will affect local
builds, in case they are done via debootstrap, from testing/unstable
users.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Santiago Vila (2023-10-12 17:56:04)
> Johannes has asked the RMs in this thread:
> 
> https://lists.debian.org/debian-release/2023/10/msg00425.html
> 
> if they are ready to consider the bugs as RC. I believe it would be better
> if we can make the bugs "factually" RC by uploading the fixed debootstrap 
> first.

I do not have a strong opinion on what should happen first but in that thread,
Holger and Sam also support the idea to first upload debootstrap.

My thought process behind asking the release team about the bug severity was,
that I found it unlikely for the remaining bugs to go down to zero without the
release team declaring that those bugs are indeed of significant severity.

Paul Gevers suggested that I should NMU to DELAYED/10 without raising the
severity.

There are not many packages remaining, so I'm impartial on how this is moved
forward.

On a different topic: this change in debootstrap will break mmdebstrap because
mmdebstrap compares its own output against the output of debootstrap to verify
that it does the right thing. Here is the patch to mmdebstrap that fixes this
issue:

@@ -2955,15 +2966,15 @@ sub run_install() {
 
 my @pkgs_to_install = (@{ $options->{include} });
 if ($options->{variant} eq 'buildd') {
-push @pkgs_to_install, 'build-essential';
+push @pkgs_to_install, 'build-essential', 'apt';
 }
 if (any { $_ eq $options->{variant} }
-('required', 'important', 'standard', 'buildd')) {
+('required', 'important', 'standard')) {
 # Many of the priority:required packages are also essential:yes. We
 # make sure not to select those here to avoid useless "xxx is already
 # the newest version" messages.
 my $priority;
-if (any { $_ eq $options->{variant} } ('required', 'buildd')) {
+if (any { $_ eq $options->{variant} } ('required')) {
 $priority = '?and(?priority(required),?not(?essential))';
 } elsif ($options->{variant} eq 'important') {
 $priority = '?and(?or(?priority(required),?priority(important)),'

I verified that the patch from the MR indeed ends up doing the same as above
patch to mmdebstrap.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Santiago Vila]

El 10/10/23 a las 13:46, Luca Boccassi escribió:

Given the list of affected packages is short (and it's all about
tzdata IIRC), how about we wait until that list is down to zero (and
if you have time, maybe you could help with that?), and then merge
this change? That way we don't add instability, and fix the issue at
the same time.


Hello.

In my opinion, the main problem here is the discrepancy between
policy saying "packages must build from source after BD are installed"
and what actually happens in the buildds (packages with missing BD
not always failing in the buildds).

When there is a category of bugs which are considered important
and we want to make it serious, we report the bugs first,
we give plenty of time to fix them, and we finally raise
the severities.

We usually don't wait for the bug number to become zero,
because the number of affected packages usually decays exponentially.

In general, when the list of affected packages is short enough, that's
usually a sign that it's already ok to make the bugs RC.

Johannes has asked the RMs in this thread:

https://lists.debian.org/debian-release/2023/10/msg00425.html

if they are ready to consider the bugs as RC. I believe it would be better
if we can make the bugs "factually" RC by uploading the fixed debootstrap first.

After that, complains about this kind of bugs not happening in the buildds
or being difficult to reproduce should stop completely, and as far as I'm
concerned, the underlying problem will be solved.

In fact, we could upload deboostrap already and still have a moratorium
on already reported bugs. For example, we could agree on not raising the
severities for three months on old bugs.

This way we give even more time for people to fix those bugs, NMUs
would not be needed, but anybody trying to upload an affected package
without fixing the bug will see that it will not work in the buildds.

That was, after all, the whole idea in this bug, to optimize the way
these kind of bugs are detected by detecting them in the buildds.

I believe we would be nice enough to people if we do that (i.e. fixing
debootstrap now, and delaying severity change a little more if we
really want to give more time).

Thanks.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Luca Boccassi]

On Fri, 29 Sept 2023 at 00:42, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi Julien,
>
> thank you for your quick reply!
>
> Quoting Julien Cristau (2023-09-28 17:49:51)
> > I guess more than mixing two different things I disagree that that is
> > debootstrap's responsibility, and so I disagree that that is a valid bug.  
> > In
> > my view it's more important for debootstrap to reliably be able to create
> > chroots than some sort of philosophical purity about what is included in 
> > said
> > chroot.  Package priorities are how the archive tells debootstrap which
> > packages to install, and so since I don't see your (A) as a bug, I'm saying
> > if there's a bug it's (B) and belongs with the archive.
>
> I agree that debootstrap's reliability to create chroots is supremely
> important. But do you see a bug with my change that makes it unreliable? It
> changes what the chroot includes yes, but it does so in a reliable way unless 
> I
> missed something?

Given the list of affected packages is short (and it's all about
tzdata IIRC), how about we wait until that list is down to zero (and
if you have time, maybe you could help with that?), and then merge
this change? That way we don't add instability, and fix the issue at
the same time.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Hi Julien,

thank you for your quick reply!

Quoting Julien Cristau (2023-09-28 17:49:51)
> I guess more than mixing two different things I disagree that that is
> debootstrap's responsibility, and so I disagree that that is a valid bug.  In
> my view it's more important for debootstrap to reliably be able to create
> chroots than some sort of philosophical purity about what is included in said
> chroot.  Package priorities are how the archive tells debootstrap which
> packages to install, and so since I don't see your (A) as a bug, I'm saying
> if there's a bug it's (B) and belongs with the archive.

I agree that debootstrap's reliability to create chroots is supremely
important. But do you see a bug with my change that makes it unreliable? It
changes what the chroot includes yes, but it does so in a reliable way unless I
missed something?

I also agree that "philosophical purity" is a bad argument but "package
priorities are how the archive tells debootstrap which packages to install" is
also not a practical argument but an argument of "we've always done it this way
and that's why we will continue to do it that way". This sounds to me like
another argument of "philosophical purity" instead of one motivated by
practical considerations.

So I'd like to take a step back. There was a big thread on d-devel about this
topic and arguments have been exchanged about this forth and back. I'd like to
use this email to rephrase the argument I have already made for this change.

> I also think your argument, even if I went along with it, breaks down when
> the apt package gets included, since apt is clearly not build-essential, so
> by that logic we'd go back to the days where builds used the host system's
> apt instead of including it in the chroot.

If my argument was of "philosophical purity" then it would indeed completely
break down with making apt the exception. But I'm trying to fix a practical
problem here and thus I'm fine with leaving apt the exception for now. Whether
or not to use apt from the outside would be another discussion we could have
but I do not think it is very useful to have the discussion today. The
practical side of the matter is, that the default schroot backend of sbuild is
unable to use apt from the outside and unless official buildds switch to the
unshare backend, apt has to be included in the chroot.

What this change is trying to achieve is to make it harder for source packages
to make it into the archive that have build dependencies missing. This is my
main motivation. This is also why having apt inside the chroot is not much of a
problem because I cannot remember having seen a source package in the archive
that needed apt to build but was missing it in its Build-Depends.

Source packages with missing build dependencies make QA work on the archive
harder. Santiago is not the only one who found these bugs, Helmut filed bugs
for missing B-D on tzdata and obviously I ran into the problem myself where I
wanted to build a source package but it failed to build because it did not
declare the B-D it actually needed. The time we spend on stumbling over these
bugs, filing them and waiting for them to be fixed could be spent doing more
useful things. We would not have these bugs if source packages were build on
the official buildds in an environment that didn't have extra packages
installed.

So I propose to change the debootstrap buildd variant to install fewer
packages: to help catch this sort of bugs early. In the best case, already on
the developer's machine when they try to build the package.

As always this is a trade-off. If this change to debootstrap is not made, these
bugs will continue to keep happening and the time of some of us will continue
to be wasted on this issue. What is the other side of the coin? If this change
is made, what would break? Santiago already analyzed the archive to find source
packages that would fail and filed bugs. So we know what would break and we
informed the maintainers. What else would break? Would anything else break
other than the principle that debootstrap historically just must include the
priority:required packages just because it has always done it like that?

I really do not understand why using the Priority field and only the Priority
field is a thing that just must keep being used by debootstrap. Why is it a
problem to use the Essential field instead?

Right now, debootstrap is the odd-one-out in the archive. To my knowledge there
is no other software (other than mmdebstrap which will mirror what debootstrap
does by design) which considers the Priority field when selecting packages that
need to be installed to satisfy the build-dependencies of a source package.
Sbuild for example is fine with working on chroots that just include
essenital+apt but it will only install build-essential and not
Priority:required packages. When asking apt to satisfy build dependencies, it
will install build-essenital but not Priority:required packages. When asking

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Julien Cristau]

On Thu, Sep 28, 2023 at 12:42:27PM +0200, Santiago Vila wrote:
> El 28/9/23 a las 11:50, Julien Cristau escribió:
> > I still think that is absolutely the wrong thing to do, and makes
> > debootstrap more fragile for no good reason.
> 
> Julien, I believe you are mixing two different things here.
> 
> (A) What this bug is really about.
> 
> (B) What the effect of the bug is.
> 
> The bug (A) is that debootstrap, being the tool used to create chroots
> to build packages, has the responsibility of ensuring that
> the chroot is composed by build-essential packages only, and it
> should try hard not to install packages which are not build-essential.
> 
I guess more than mixing two different things I disagree that that is
debootstrap's responsibility, and so I disagree that that is a valid
bug.  In my view it's more important for debootstrap to reliably be able
to create chroots than some sort of philosophical purity about what is
included in said chroot.  Package priorities are how the archive tells
debootstrap which packages to install, and so since I don't see your (A)
as a bug, I'm saying if there's a bug it's (B) and belongs with the
archive.

I also think your argument, even if I went along with it, breaks down
when the apt package gets included, since apt is clearly not
build-essential, so by that logic we'd go back to the days where builds
used the host system's apt instead of including it in the chroot.

> In other words, the bug says that the algorithm followed by debootstrap
> to determine which packages should be installed is *flawed*.
> 
> Then there is the effect of the bug (B). The effect, obviously,
> is that we end up having non-build-essential packages in a chroot
> when using the buildd profile, which is definitely not ok.
> 
> Why do you suggest that we fix only the effects of the bug but not
> the bug itself? In other words: Why are you apparently mixing (A) and (B)
> as if they were the same thing?
> 
> True, the ftpmasters might change priorities so that debootstrap
> does the right thing by default, but this would be "by pure chance",
> as the algorithm would still be wrong.
> 

> Even if they change the priorities today, it would suffice that
> some day another essential package becomes non-essential but still required,
> and then we would have to wait another seven years for debootstrap
> to do the right thing again.
> 
There's no reason that would need to take seven years, so I don't know
what that point is about.

Cheers,
Julien

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Santiago Vila]

El 28/9/23 a las 11:50, Julien Cristau escribió:

I still think that is absolutely the wrong thing to do, and makes
debootstrap more fragile for no good reason.


Julien, I believe you are mixing two different things here.

(A) What this bug is really about.

(B) What the effect of the bug is.

The bug (A) is that debootstrap, being the tool used to create chroots
to build packages, has the responsibility of ensuring that
the chroot is composed by build-essential packages only, and it
should try hard not to install packages which are not build-essential.

In other words, the bug says that the algorithm followed by debootstrap
to determine which packages should be installed is *flawed*.

Then there is the effect of the bug (B). The effect, obviously,
is that we end up having non-build-essential packages in a chroot
when using the buildd profile, which is definitely not ok.

Why do you suggest that we fix only the effects of the bug but not
the bug itself? In other words: Why are you apparently mixing (A) and (B)
as if they were the same thing?

True, the ftpmasters might change priorities so that debootstrap
does the right thing by default, but this would be "by pure chance",
as the algorithm would still be wrong.

Even if they change the priorities today, it would suffice that
some day another essential package becomes non-essential but still required,
and then we would have to wait another seven years for debootstrap
to do the right thing again.

We could avoid all that by fixing debootstrap once and for good.


If you think a particular package shouldn't be priority:required
then file a bug against ftp.debian.org to change it.


That may also be true, but it's not the bug that was reported here.

Thanks.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Julien Cristau]

Hi,

I still think that is absolutely the wrong thing to do, and makes
debootstrap more fragile for no good reason.  If you think a particular
package shouldn't be priority:required then file a bug against
ftp.debian.org to change it.

Cheers,
Julien

On Sat, Sep 23, 2023 at 20:13:45 +0200, Johannes Schauer Marin Rodrigues wrote:

> Hi all,
> 
> On Sun, 25 Jun 2017 06:01:13 +0200 Johannes Schauer  wrote:
> > Quoting Cyril Brulebois (2017-06-24 20:23:25)
> > > Julien Cristau  (2016-09-12):
> > > > This is a transient situation because some Essential packages'
> > > > dependencies changed.  I'd consider this a bug in the archive, not in
> > > > debootstrap.
> > > Any reasons to keep this bug report open then? Seen no objections, so I'm
> > > tempted to close it.
> > 
> > But... the buildd variant still explicitly (and not only implicitly through
> > dependencies of essential:yes packages) installs Priority:required packages,
> > no?
> 
> as we are at the beginning of the trixie development cycle, I have opened a
> merge request against debootstrap which avoids installing priority:required
> packages with the buildd variant:
> 
> https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/106#note_430035
> 
> I've put Ansgar and Julien in CC as they were opposed to this change.
> 
> I'm putting Luca and Guillem in CC who wrote in favour of this change also in
> policy bug #1029831.
> 
> Santiago is in CC as the driver of the mass bug filing for source packages 
> that
> fail to build in a chroot environment with just Essential:yes and
> build-essential installed.
> 
> According to the last MBF from December 2022 and January 2023, there are 13
> source packages that would FTBFS after this change because they are missing an
> explicit build dependency on tzdata or mount.
> 
> As part of the thread starting at
> 9b40f6f2-4942-acfc-2f9c-4668f05d9...@debian.org a number of arguments were 
> made
> for and against this change. I still believe that the arguments for this 
> change
> weigh stronger than those against it and thus I filed that merge request 
> above.
> 
> Luca, as the debootstrap maintainer, what are your thoughts?
> 
> Thank you!
> 
> cheers, josch

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

The remaining bugs are now properly user-tagged and can be found here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian...@lists.debian.org;tag=missing-build-depends-on-not-build-essential

On Sat, 23 Sep 2023 20:13:45 +0200 Johannes Schauer Marin Rodrigues 
 wrote:
> Santiago is in CC as the driver of the mass bug filing for source packages
> that fail to build in a chroot environment with just Essential:yes and
> build-essential installed.
> 
> According to the last MBF from December 2022 and January 2023, there are 13
> source packages that would FTBFS after this change because they are missing
> an explicit build dependency on tzdata or mount.

Let me also correct my last message. As I believe that source packages should
be build with just Essential:yes and build-essential (and their respective
dependencies) installed, I do not consider Santiago's work an MBF but just
another QA bug filing against packages that FTBFS in an environment where they
should be buildable.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Luca Boccassi]

On Sat, 23 Sept 2023 at 19:13, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi all,
>
> On Sun, 25 Jun 2017 06:01:13 +0200 Johannes Schauer  wrote:
> > Quoting Cyril Brulebois (2017-06-24 20:23:25)
> > > Julien Cristau  (2016-09-12):
> > > > This is a transient situation because some Essential packages'
> > > > dependencies changed.  I'd consider this a bug in the archive, not in
> > > > debootstrap.
> > > Any reasons to keep this bug report open then? Seen no objections, so I'm
> > > tempted to close it.
> >
> > But... the buildd variant still explicitly (and not only implicitly through
> > dependencies of essential:yes packages) installs Priority:required packages,
> > no?
>
> as we are at the beginning of the trixie development cycle, I have opened a
> merge request against debootstrap which avoids installing priority:required
> packages with the buildd variant:
>
> https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/106#note_430035
>
> I've put Ansgar and Julien in CC as they were opposed to this change.
>
> I'm putting Luca and Guillem in CC who wrote in favour of this change also in
> policy bug #1029831.
>
> Santiago is in CC as the driver of the mass bug filing for source packages 
> that
> fail to build in a chroot environment with just Essential:yes and
> build-essential installed.
>
> According to the last MBF from December 2022 and January 2023, there are 13
> source packages that would FTBFS after this change because they are missing an
> explicit build dependency on tzdata or mount.
>
> As part of the thread starting at
> 9b40f6f2-4942-acfc-2f9c-4668f05d9...@debian.org a number of arguments were 
> made
> for and against this change. I still believe that the arguments for this 
> change
> weigh stronger than those against it and thus I filed that merge request 
> above.
>
> Luca, as the debootstrap maintainer, what are your thoughts?

Sounds fine by me if people are ok with it, impact is minimal and fix
is very simple and sounds correct from a conceptual point of view.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer Marin Rodrigues]

Hi all,

On Sun, 25 Jun 2017 06:01:13 +0200 Johannes Schauer  wrote:
> Quoting Cyril Brulebois (2017-06-24 20:23:25)
> > Julien Cristau  (2016-09-12):
> > > This is a transient situation because some Essential packages'
> > > dependencies changed.  I'd consider this a bug in the archive, not in
> > > debootstrap.
> > Any reasons to keep this bug report open then? Seen no objections, so I'm
> > tempted to close it.
> 
> But... the buildd variant still explicitly (and not only implicitly through
> dependencies of essential:yes packages) installs Priority:required packages,
> no?

as we are at the beginning of the trixie development cycle, I have opened a
merge request against debootstrap which avoids installing priority:required
packages with the buildd variant:

https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/106#note_430035

I've put Ansgar and Julien in CC as they were opposed to this change.

I'm putting Luca and Guillem in CC who wrote in favour of this change also in
policy bug #1029831.

Santiago is in CC as the driver of the mass bug filing for source packages that
fail to build in a chroot environment with just Essential:yes and
build-essential installed.

According to the last MBF from December 2022 and January 2023, there are 13
source packages that would FTBFS after this change because they are missing an
explicit build dependency on tzdata or mount.

As part of the thread starting at
9b40f6f2-4942-acfc-2f9c-4668f05d9...@debian.org a number of arguments were made
for and against this change. I still believe that the arguments for this change
weigh stronger than those against it and thus I filed that merge request above.

Luca, as the debootstrap maintainer, what are your thoughts?

Thank you!

cheers, josch

signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer]

Hi,

Quoting Cyril Brulebois (2017-06-24 20:23:25)
> Julien Cristau  (2016-09-12):
> > This is a transient situation because some Essential packages'
> > dependencies changed.  I'd consider this a bug in the archive, not in
> > debootstrap.
> Any reasons to keep this bug report open then? Seen no objections, so I'm
> tempted to close it.

But... the buildd variant still explicitly (and not only implicitly through
dependencies of essential:yes packages) installs Priority:required packages,
no?

cheers, josch


signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Cyril Brulebois]

Julien Cristau  (2016-09-12):
> This is a transient situation because some Essential packages'
> dependencies changed.  I'd consider this a bug in the archive, not in
> debootstrap.

Any reasons to keep this bug report open then? Seen no objections, so I'm
tempted to close it.


KiBi.


signature.asc
Description: Digital signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Julien Cristau]

On Thu, Sep  8, 2016 at 14:07:04 +0200, Johannes Schauer wrote:

> Package: debootstrap
> Version: 1.0.81
> Severity: normal
> 
> Hi,
> 
> in Debian, every binary package implicitly depends on all binary
> packages marked as Essential:yes and every source package implicitly
> build-depends on the binary package build-essential. Policy §4.2 says:
> 
>  | it must be possible to build the package and produce working binaries
>  | on a system with only essential and build-essential packages installed
>  | and also those required to satisfy the build-time relationships
>  | (including any implied relationships).
> 
> Currently, programs in Debian that facilitate building source packages
> in "clean" environments like sbuild and pbuilder use debootstrap to
> create this "minimal" environment. Specifically, they use the buildd
> variant provided by debootstrap.
> 
> Unfortunately it seems that in addition to installing the minimum
> required packages (all Essential:yes, build-essential and (unfortunately
> necessarily) apt), debootstrap also installs all packages marked as
> Priority:required (and their transitive dependencies).
> 
This is a transient situation because some Essential packages'
dependencies changed.  I'd consider this a bug in the archive, not in
debootstrap.

Cheers,
Julien

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Sven Joachim]

On 2016-09-08 22:12 +0200, Johannes Schauer wrote:

> Hi,
>
> On Thu, 08 Sep 2016 19:40:15 +0200 Sven Joachim  wrote:
>> Looking at the code in scripts/sid, it is "x_core_install mawk" which
>> fails here.  The reason is that mawk has not been downloaded,
>> debootstrap's limited dependency resolver cannot resolve base-files'
>> pre-dependency on awk.
>>
>> The good news is that with "--include=mawk" added to the commandline,
>> debootstrap succeeds and does not include tzdata or lsb-base in the
>> chroot. :-)
>>
>> So changing base-files to Pre-depend on mawk | awk seems to be the only
>> blocker here.  Would you like to file a blocking bug on base-files?
>
> I don't see why this is a bug in base-files. As far as I can see, base-files
> properly declares its pre-dependency on the virtual package awk. That
> debootstrap is unable to understand basic Debian dependency constructs (we are
> not even talking multiarch here) is a bug in debootstrap.
>
> This is also the point where I wonder how much sense it makes to have yet
> another resolver of Debian's complex dependency mechanism around. It's one of
> the reasons why I often use multistrap instead of debootstrap because the
> former uses apt which already implements all the required dependency logic.

Unlike multistrap, debootstrap also needs to work on systems where
apt or dpkg are not available.

> If debootstrap wants to depend on its own resolver, then it has to make sure
> that it is up to the task of dealing with Debian's dependency system.

I think that cdebootstrap is better in that respect, because it runs
apt-get at the end of the installation to fix up any broken
dependencies.  Somebody™ ought to implement this in debootstrap.

Cheers,
   Sven

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Sven Joachim]

On 2016-09-08 20:54 +0100, Santiago Vila wrote:

> On Thu, Sep 08, 2016 at 07:40:15PM +0200, Sven Joachim wrote:
>
>
>> | Setting up perl-base (5.22.2-5) ...
>> | dpkg: error: --install needs at least one package archive file argument
>> `
>> 
>> Looking at the code in scripts/sid, it is "x_core_install mawk" which
>> fails here.  The reason is that mawk has not been downloaded,
>> debootstrap's limited dependency resolver cannot resolve base-files'
>> pre-dependency on awk.
>> 
>> The good news is that with "--include=mawk" added to the commandline,
>> debootstrap succeeds and does not include tzdata or lsb-base in the
>> chroot. :-)
>> 
>> So changing base-files to Pre-depend on mawk | awk seems to be the only
>> blocker here.  Would you like to file a blocking bug on base-files?
>
> You don't need to change base-files for that.
>
> Package awk is essential and virtual. The dependency of base-files on
> awk is just to ensure that there is always an awk available (once that
> you already have a running system), but it's not meant to tell
> debootstrap which one is "the awk of preference".
>
> debootstrap already knows that, and the proof is that it deliberately
> installs "mawk" and no other awk implementation.

No, debootstrap does not know this, it downloads mawk because it has
priority required.

> If we switch from installing required packages to installing just the
> essential packages (which are a subset), we will miss mawk, so for all
> purposes, since we are bootstrapping, you should treat mawk as if it
> were essential.
>
> Could you please try something like this?
>
>   required="mawk $(get_debs Essential: yes)"

This works, thanks.

Cheers,
   Sven

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer]

Hi,

On Thu, 08 Sep 2016 19:40:15 +0200 Sven Joachim  wrote:
> Looking at the code in scripts/sid, it is "x_core_install mawk" which
> fails here.  The reason is that mawk has not been downloaded,
> debootstrap's limited dependency resolver cannot resolve base-files'
> pre-dependency on awk.
>
> The good news is that with "--include=mawk" added to the commandline,
> debootstrap succeeds and does not include tzdata or lsb-base in the
> chroot. :-)
>
> So changing base-files to Pre-depend on mawk | awk seems to be the only
> blocker here.  Would you like to file a blocking bug on base-files?

I don't see why this is a bug in base-files. As far as I can see, base-files
properly declares its pre-dependency on the virtual package awk. That
debootstrap is unable to understand basic Debian dependency constructs (we are
not even talking multiarch here) is a bug in debootstrap.

This is also the point where I wonder how much sense it makes to have yet
another resolver of Debian's complex dependency mechanism around. It's one of
the reasons why I often use multistrap instead of debootstrap because the
former uses apt which already implements all the required dependency logic.

If debootstrap wants to depend on its own resolver, then it has to make sure
that it is up to the task of dealing with Debian's dependency system. So in
summary, I think this is a bug in debootstrap and not in base-files.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Santiago Vila]

On Thu, Sep 08, 2016 at 07:40:15PM +0200, Sven Joachim wrote:


> | Setting up perl-base (5.22.2-5) ...
> | dpkg: error: --install needs at least one package archive file argument
> `
> 
> Looking at the code in scripts/sid, it is "x_core_install mawk" which
> fails here.  The reason is that mawk has not been downloaded,
> debootstrap's limited dependency resolver cannot resolve base-files'
> pre-dependency on awk.
> 
> The good news is that with "--include=mawk" added to the commandline,
> debootstrap succeeds and does not include tzdata or lsb-base in the
> chroot. :-)
> 
> So changing base-files to Pre-depend on mawk | awk seems to be the only
> blocker here.  Would you like to file a blocking bug on base-files?

You don't need to change base-files for that.

Package awk is essential and virtual. The dependency of base-files on
awk is just to ensure that there is always an awk available (once that
you already have a running system), but it's not meant to tell
debootstrap which one is "the awk of preference".

debootstrap already knows that, and the proof is that it deliberately
installs "mawk" and no other awk implementation.

If we switch from installing required packages to installing just the
essential packages (which are a subset), we will miss mawk, so for all
purposes, since we are bootstrapping, you should treat mawk as if it
were essential.

Could you please try something like this?

  required="mawk $(get_debs Essential: yes)"

I think that should be more than enough.

Thanks.

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Sven Joachim]

On 2016-09-08 14:07 +0200, Johannes Schauer wrote:

> Package: debootstrap
> Version: 1.0.81
> Severity: normal
>
> Hi,
>
> in Debian, every binary package implicitly depends on all binary
> packages marked as Essential:yes and every source package implicitly
> build-depends on the binary package build-essential. Policy §4.2 says:
>
>  | it must be possible to build the package and produce working binaries
>  | on a system with only essential and build-essential packages installed
>  | and also those required to satisfy the build-time relationships
>  | (including any implied relationships).
>
> Currently, programs in Debian that facilitate building source packages
> in "clean" environments like sbuild and pbuilder use debootstrap to
> create this "minimal" environment. Specifically, they use the buildd
> variant provided by debootstrap.
>
> Unfortunately it seems that in addition to installing the minimum
> required packages (all Essential:yes, build-essential and (unfortunately
> necessarily) apt), debootstrap also installs all packages marked as
> Priority:required (and their transitive dependencies).

One could argue that the priority of those required packages which are
not Essential:yes or a (transitive) dependency of an Essential:yes
package should probably be demoted.

> Thus, it can easily happen that source packages in Debian do not
> correctly declare their build dependencies on packages that are
> Priority:required because they happen to always be installed in
> virtually any environment that the source package will probably ever be
> built in (because they were all created by debootstrap).
>
> I think this is a bug in the package list installed by the buildd
> variant. Since the buildd variant is meant to provide a clean and
> minimal environment to build source packages, it should try hard not to
> install any extra packages which would then make it impossible to test
> whether a source package is policy compliant in the build dependencies
> it declares.

I had a look and tried this naive patch:

--8<---cut here---start->8---
diff --git a/scripts/sid b/scripts/sid
index 7b32ac2..8294748 100644
--- a/scripts/sid
+++ b/scripts/sid
@@ -24,6 +24,7 @@ work_out_debs () {
base="$(get_debs Priority: important)"
elif doing_variant buildd || doing_variant scratchbox; then
base="apt build-essential"
+   required="$(get_debs Essential: yes)"
elif doing_variant minbase; then
base="apt"
fi
--8<---cut here---end--->8---

This failed with the following cryptic error message:

,
| I: Installing core packages...
| W: Failure trying to run: chroot /usr/local/abfall/base-test3 dpkg 
--force-depends --install
| W: See /usr/local/abfall/base-test3/debootstrap/debootstrap.log for details
`

The debootstrap.log file contains these lines at the end:

,
| Unpacking perl-base (5.22.2-5) ...
| Setting up perl-base (5.22.2-5) ...
| dpkg: error: --install needs at least one package archive file argument
`

Looking at the code in scripts/sid, it is "x_core_install mawk" which
fails here.  The reason is that mawk has not been downloaded,
debootstrap's limited dependency resolver cannot resolve base-files'
pre-dependency on awk.

The good news is that with "--include=mawk" added to the commandline,
debootstrap succeeds and does not include tzdata or lsb-base in the
chroot. :-)

So changing base-files to Pre-depend on mawk | awk seems to be the only
blocker here.  Would you like to file a blocking bug on base-files?

Cheers,
   Sven

Bug#837060: debootstrap: Do not install packages of Priority:required for buildd variant

[Johannes Schauer]

Package: debootstrap
Version: 1.0.81
Severity: normal

Hi,

in Debian, every binary package implicitly depends on all binary
packages marked as Essential:yes and every source package implicitly
build-depends on the binary package build-essential. Policy §4.2 says:

 | it must be possible to build the package and produce working binaries
 | on a system with only essential and build-essential packages installed
 | and also those required to satisfy the build-time relationships
 | (including any implied relationships).

Currently, programs in Debian that facilitate building source packages
in "clean" environments like sbuild and pbuilder use debootstrap to
create this "minimal" environment. Specifically, they use the buildd
variant provided by debootstrap.

Unfortunately it seems that in addition to installing the minimum
required packages (all Essential:yes, build-essential and (unfortunately
necessarily) apt), debootstrap also installs all packages marked as
Priority:required (and their transitive dependencies).

Thus, it can easily happen that source packages in Debian do not
correctly declare their build dependencies on packages that are
Priority:required because they happen to always be installed in
virtually any environment that the source package will probably ever be
built in (because they were all created by debootstrap).

I think this is a bug in the package list installed by the buildd
variant. Since the buildd variant is meant to provide a clean and
minimal environment to build source packages, it should try hard not to
install any extra packages which would then make it impossible to test
whether a source package is policy compliant in the build dependencies
it declares.

Thank you!

cheers, josch