Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

El 4/1/23 a las 17:45, Julien Cristau escribió:

Like I said in the debootstrap bug, I believe we should treat a case
where a package is Priority: required but not actually required by the
Essential set as a bug in package priorities, and neither debootstrap
nor policy need to change.


I take your word that policy does not need to change and I've just closed
this report, as there is clearly not a consensus that policy needs a 
clarification.

I see that you have just downgraded the debootstrap bug to wishlist.

Have you actually asked ftpmasters to downgrade the priorities
that would have to be downgraded so that debootstrap does not install
packages that are not build-essential when using the buildd profile?

Thanks.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

El 4/1/23 a las 19:28, Sam Hartman escribió:

"Santiago" == Santiago Vila  writes:


 Santiago> I think you can't really estimate such thing. You seem to
 Santiago> imply that we have been allowing packages with missing
 Santiago> build-dependencies for a long time, but that's not
 Santiago> accurate. The *buildds* have been allowing packages with
 Santiago> missing build-dependencies for a long time, but I have
 Santiago> been reporting those bugs for a long time as well.

Thanks for the additional information.
You have not changed my mind.
I would prefer to solve this situation by increasing the build essential
set based on what I know today.


This bug report was a request to clarify policy without altering it, for those
who don't understand "packages required to build a hello world program" which
is already in policy.

But I'm starting to feel uncomfortable with the fact that the bug report is 
actually
being used to propose a policy change, which was never the intent, as it's 
something
completely different.

I fully respect those who want to change policy regarding the build-essential 
definition,
but I find it not appropriate to do that in this report, which was merely 
asking for
a clarification of current policy.

Therefore I withdraw my suggestion that current policy should be clarified by 
closing this bug,
as there seems not to be a consensus that it needs a clarification, and I 
respectfully request
that those willing to change the build-essential definition do so in another 
bug report.

Thanks.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

El 4/1/23 a las 19:54, Bill Allombert escribió:


BTW: Today I reported that kodi did not build without tzdata. But in the end
this was not a missing build-dependency of kodi, but a missing *binary* 
dependency
of one of the build-dependencies of kodi.


So is there a service that detect such missing *binary* dependency ?
This seems the missing piece.


No, there is not such a "service" (not sure if understood what do you mean by 
that).

The missing binary dependency was discovered because I am doing archive-wide 
rebuilds
following Policy 4.2, i.e. using a chroot containing only essential and 
build-essential
packages, and because kodi maintainers immediately realized about the root of 
the problem.

Thanks.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Bill Allombert]

On Wed, Jan 04, 2023 at 07:13:04PM +0100, Santiago Vila wrote:
> El 4/1/23 a las 18:23, Sam Hartman escribió:
> > I think that the
> > cost of going and adding all the build-depends on
> > required-but-not-build-essential is not worth what I estimate we'd gain
> > from having this extra information.
> 
> I think you can't really estimate such thing. You seem to imply that we have
> been allowing packages with missing build-dependencies for a long time, but 
> that's
> not accurate. The *buildds* have been allowing packages with missing 
> build-dependencies
> for a long time, but I have been reporting those bugs for a long time as well.
> 
> So it's not as if I were proposing that we start doing something that we have 
> never done.
> My only aim is that we detect such bugs earlier in the chain, in the buildds, 
> where it should be.
> 
> BTW: Today I reported that kodi did not build without tzdata. But in the end
> this was not a missing build-dependency of kodi, but a missing *binary* 
> dependency
> of one of the build-dependencies of kodi.

So is there a service that detect such missing *binary* dependency ?
This seems the missing piece. 

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Sam Hartman]

> "Santiago" == Santiago Vila  writes:


Santiago> I think you can't really estimate such thing. You seem to
Santiago> imply that we have been allowing packages with missing
Santiago> build-dependencies for a long time, but that's not
Santiago> accurate. The *buildds* have been allowing packages with
Santiago> missing build-dependencies for a long time, but I have
Santiago> been reporting those bugs for a long time as well.

Thanks for the additional information.
You have not changed my mind.
I would prefer to solve this situation by increasing the build essential
set based on what I know today.

I think this is a case where either option is technically reasonable,
and where a fairly rough consensus is appropriate to move forward.
So my hope is that as more people chime in, the policy editors
eventually judge a consensus, but I think it is fine for that consensus
to be more rough than we usually take for policy.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

El 4/1/23 a las 18:23, Sam Hartman escribió:

I think that the
cost of going and adding all the build-depends on
required-but-not-build-essential is not worth what I estimate we'd gain
from having this extra information.


I think you can't really estimate such thing. You seem to imply that we have
been allowing packages with missing build-dependencies for a long time, but 
that's
not accurate. The *buildds* have been allowing packages with missing 
build-dependencies
for a long time, but I have been reporting those bugs for a long time as well.

So it's not as if I were proposing that we start doing something that we have 
never done.
My only aim is that we detect such bugs earlier in the chain, in the buildds, 
where it should be.

BTW: Today I reported that kodi did not build without tzdata. But in the end
this was not a missing build-dependency of kodi, but a missing *binary* 
dependency
of one of the build-dependencies of kodi.

Also, several weeks ago, a bunch of ruby packages which did not build from 
source
had a common build-dependency which had a missing dependency on tzdata.

This means that smaller build environments help to detect missing binary
dependencies as well.

Thanks.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Sam Hartman]

> "Santiago" == Santiago Vila  writes:
Santiago> A minimal build essential set provides and generates
Santiago> useful information that a build essential set which is not
Santiago> so minimal does not provide.

Santiago> For example, some packages have unit tests which depend on
Santiago> the information stored on tzdata. In some cases, changes
Santiago> in tzdata causes those unit tests to fail.

Okay.
I don't find that example particularly compelling.
I absolutely agree with you we've found such bugs, but I think that the
cost of going and adding all the build-depends on
required-but-not-build-essential is not worth what I estimate we'd gain
from having this extra information.

I thin there are a few other reasons we want to keep the build-essential
set small:

* It reduces the number of packages involved in early freeze
* I suspect that there probably are bootstrapping implications.

However, neither of those reasons appear very compelling when applied to
required packages.

I appreciate the work of fixing the packages would be distributed,
although I think a significant portion of it would land on the small
number of people who do archive-wide QA.
Even if it were fully distributed, that work has a real cost.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

El 4/1/23 a las 17:16, Russ Allbery escribió:

But if you are building new Debian packages,
by definition you are not in a tiny minimal system case.  build-essential
is already somewhat arbitrary and chosen for convenience (most packages do
not require a C++ compiler).  Why not expand build-essential to what we're
largely doing in practice to fix the consistency problem (which is a real
issue) but not add work tweaking build dependencies for a bunch of
packages?


A minimal build essential set provides and generates useful information that
a build essential set which is not so minimal does not provide.

For example, some packages have unit tests which depend on the information
stored on tzdata. In some cases, changes in tzdata causes those unit tests
to fail.

The tzdata package is updated often in stable. If we were to know in advance
what packages could break, we could look at the packages which build-depend on 
it,
either directly or indirectly.

If we don't have such information, we can't even try to do that.

So, a minimal build essential set has the same benefits as a minimal essential 
set,
namely, that we know a lot better which are the real build-dependencies or the
real dependencies, as they are not hidden.

Thanks.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Julien Cristau]

On Tue, Jan  3, 2023 at 22:14:10 +0100, Santiago Vila wrote:

> Hello. This is an attempt to put the basis for fixing this bug:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837060
> 
> As an example, packages tzdata, mount or e2fsprogs are not build-essential
> and afaik have not been for a long time, but there are still people who
> believe that they are build-essential for the mere fact that they are
> priority:required.
> 
Like I said in the debootstrap bug, I believe we should treat a case
where a package is Priority: required but not actually required by the
Essential set as a bug in package priorities, and neither debootstrap
nor policy need to change.

Cheers,
Julien

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Bill Allombert]

On Wed, Jan 04, 2023 at 08:46:39AM +0100, Santiago Vila wrote:
> El 4/1/23 a las 2:32, Sam Hartman escribió:
> > > > > > > "Santiago" == Santiago Vila  writes:
> > 
> >  Santiago> As an example, packages tzdata, mount or e2fsprogs are not
> >  Santiago> build-essential and afaik have not been for a long time,
> >  Santiago> but there are still people who believe that they are
> >  Santiago> build-essential for the mere fact that they are
> >  Santiago> priority:required.
> > 
> > Why not just make all required packages build-essential?
> > I agree we should fix the class of bugs you are talking about, but it
> > seems like in some cases it might be easier to fix them by declaring
> > them not buggy.
> 
> Because required to build != required in a _running_ system

At minimum, it should not be allowed to Build-Conflicts with a required package.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Diederik de Haas]

On Tuesday, 3 January 2023 22:14:10 CET Santiago Vila wrote:
> --
>  From this definition it follows that packages of required priority are not
> necessarily build essential, as it is possible for some them not to be

... for some *of* them ... ?

> needed at all to compile, link and put in a Debian package a Hello World
> program written in C or C++.
> --



signature.asc
Description: This is a digitally signed message part.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Russ Allbery]

Santiago Vila  writes:
> El 4/1/23 a las 2:32, Sam Hartman escribió:

>> Why not just make all required packages build-essential?  I agree we
>> should fix the class of bugs you are talking about, but it seems like
>> in some cases it might be easier to fix them by declaring them not
>> buggy.

> Because required to build != required in a _running_ system

I think that's why Sam suggested making them build-essential, not
essential.

> The idea of declaring something not a bug to avoid fixing it is not very
> appealing to me. I believe we can do better than that.

I think the part that I don't understand is why we would bite off a
possibly substantial amount of work to minimize the *build-essential* set.

I understand why we want to minimize the essential set: we want to support
tiny systems and small containers and other cases where Debian shouldn't
hog a bunch of disk space.  But if you are building new Debian packages,
by definition you are not in a tiny minimal system case.  build-essential
is already somewhat arbitrary and chosen for convenience (most packages do
not require a C++ compiler).  Why not expand build-essential to what we're
largely doing in practice to fix the consistency problem (which is a real
issue) but not add work tweaking build dependencies for a bunch of
packages?  What benefit are we gaining from trying to push for that extra
bit of minimization of *build* environments?

To be clear, this is a real question, not rhetorical.  It's quite likely
that there is some benefit that I'm not seeing (such as with bootstrapping
new architectures).

-- 
Russ Allbery (r...@debian.org)  

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

El 4/1/23 a las 2:32, Sam Hartman escribió:

"Santiago" == Santiago Vila  writes:


 Santiago> As an example, packages tzdata, mount or e2fsprogs are not
 Santiago> build-essential and afaik have not been for a long time,
 Santiago> but there are still people who believe that they are
 Santiago> build-essential for the mere fact that they are
 Santiago> priority:required.

Why not just make all required packages build-essential?
I agree we should fix the class of bugs you are talking about, but it
seems like in some cases it might be easier to fix them by declaring
them not buggy.


Because required to build != required in a _running_ system

The idea of declaring something not a bug to avoid fixing it is not very 
appealing
to me. I believe we can do better than that. The proposal made in bug #837060
(namely, that debootstrap stops installing packages not build essential when
using the buildd profile) is IMO the optimal way to fix the problem at its root,
because once that packages with missing build-depends start failing in the 
buildds,
then it will be clear for everybody that there is a missing build-depends.

(The initial email in such bug by Johannes Schauer describes the problem
very well).

Thanks.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Sam Hartman]

> "Santiago" == Santiago Vila  writes:


Santiago> As an example, packages tzdata, mount or e2fsprogs are not
Santiago> build-essential and afaik have not been for a long time,
Santiago> but there are still people who believe that they are
Santiago> build-essential for the mere fact that they are
Santiago> priority:required.

Why not just make all required packages build-essential?
I agree we should fix the class of bugs you are talking about, but it
seems like in some cases it might be easier to fix them by declaring
them not buggy.

Bug#1027832: debian-policy: Please clarify that priority required packages are not automatically build essential

[Santiago Vila]

Package: debian-policy
Version: 4.6.2.0
Severity: wishlist

Hello. This is an attempt to put the basis for fixing this bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837060

As an example, packages tzdata, mount or e2fsprogs are not build-essential
and afaik have not been for a long time, but there are still people who
believe that they are build-essential for the mere fact that they are
priority:required.

Therefore I think a clarification would be useful to clear those kind
of misconceptions.

Proposed text, to be added after the paragraph which defines build essential
based on the Hello World example:

--
From this definition it follows that packages of required priority are not
necessarily build essential, as it is possible for some them not to be
needed at all to compile, link and put in a Debian package a Hello World
program written in C or C++.
--

Next step would be to add a paragraph saying tools like debootstrap when used
to create chroots for building (i.e. "buildd" profile in deboostrap) should try
to keep the list of installed packages as minimal as possible, as far as
doing so does not become disruptive (for example, apt is technically not
build-essential but it is required to install the build-dependencies).

Thanks.