Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Russ Allbery]

Simon McVittie  writes:

> Here are some updated patches for Policy, incorporating this requirement.

> I have not attempted to incorporate the corner case involving
> build-profiles. I think if we were going to do that, it would require
> documenting build-profiles first (#757760), and maybe even then it's too
> much of a corner-case to be documenting unless/until it actually
> happens.

I also second these changes.  In the name of expediency, and since I
believe all the proposed wording changes were informative, I applied these
patches for the next release and made the wording changes suggested by
Holger and Sean.  Attached are the changes as committed.

Subsequent to this work, we added the non-free-firmware archive area.
Simon's patch therefore doesn't add a statement to that section about
whether source packages in non-free-firmware are allowed to produce
binaries in non-free, or if source packages in non-free are allowed to
produce binaries in non-free-firmware, and I don't know what the answer to
that is.

Would the following wording be correct?

If a source package is in the *non-free-firmware* archive area, then
each of the binary packages that it produces must also be in the
*non-free-firmware* archive area.

Please let me know, and I will propose some follow-up wording for that.

-- 
Russ Allbery (r...@debian.org)  

diff --git a/debian/changelog b/debian/changelog
index 66d6fa0..a5e3e3e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -11,6 +11,11 @@ debian-policy (4.6.3.0) UNRELEASED; urgency=medium
 Seconded: Russ Allbery 
 Seconded: Holger Levsen 
 Closes: #1035733
+  * Policy: Source packages in main may build binary packages in contrib
+Wording: Simon McVittie 
+Seconded: Holger Levsen 
+Seconded: Russ Allbery 
+Closes: #994008
 
  -- Sean Whitton   Wed, 14 Jun 2023 16:58:40 +0100
 
diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
index c7cd340..eb8978a 100644
--- a/policy/ch-archive.rst
+++ b/policy/ch-archive.rst
@@ -136,6 +136,27 @@ In addition, the packages in *main*
 
 - must meet all policy requirements presented in this manual.
 
+If a source package is in the *main* archive area, then at least one of
+its binary packages must be in the *main* archive area, and each of the
+remaining packages must be in either the *main* or *contrib* archive
+area. Each binary package's archive area is indicated by its ``Section``
+field: see :ref:`s-subsections`.
+
+Source packages in *main* with a mixture of *main* and *contrib* binary
+packages are more complex for archive tooling to handle, and therefore
+should be limited to situations where it would be inconvenient to split
+the source package. If it is straightforward to split the source package
+into a *main* part and a *contrib* part that are built separately, then
+those parts should be represented as separate source packages.
+
+When a *main* source package has a mixture of *main* and *contrib* binary
+packages, the source package and the *main* binary packages must follow
+the requirements for *main* packages, but the *contrib* binary packages
+may follow the weaker requirements for *contrib* packages. In particular,
+source packages in *main* must not have build dependencies outside *main*,
+but the *contrib* binary packages may have runtime dependencies outside
+*main*.
+
 .. [2]
See `What Does Free Mean? `_ for
more about what we mean by free software.
@@ -192,6 +213,10 @@ Examples of packages which would be included in *contrib* are:
 - wrapper packages or other sorts of free accessories for non-free
   programs.
 
+If a source package is in the *contrib* archive area, then each of the
+binary packages that it produces must also be in the *contrib* archive
+area.
+
 .. _s-non-free:
 
 The non-free archive area
@@ -214,6 +239,10 @@ In addition, the packages in *non-free*
 - must meet all policy requirements presented in this manual that it is
   possible for them to meet.  [4]_
 
+If a source package is in the *non-free* archive area, then each of the
+binary packages that it produces must also be in the *non-free* archive
+area.
+
 .. [4]
It is possible that there are policy requirements which the package
is unable to meet, for example, if the source is unavailable. These
diff --git a/policy/upgrading-checklist.rst b/policy/upgrading-checklist.rst
index 54a473b..6009819 100644
--- a/policy/upgrading-checklist.rst
+++ b/policy/upgrading-checklist.rst
@@ -44,6 +44,13 @@ Version 4.7.0
 
 Unreleased.
 
+2.2.1
+Document that source packages in the *main* archive area may build
+binary packages in the *contrib* archive area, although this is
+discouraged unless the source package is inconvenient to split.  This
+does not relax the requirement that source packages in *main* must not
+have build dependencies outside of *main*.
+
 2.2.2
 The ``non-free-firmware`` 

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Sean Whitton]

Hello Simon,

On Fri 29 Oct 2021 at 10:51AM +01, Simon McVittie wrote:

> From a332e4e787837cac0856c9c36d6e87e9f19197e2 Mon Sep 17 00:00:00 2001
> From: Simon McVittie 
> Date: Thu, 9 Sep 2021 15:43:20 +0100
> Subject: [PATCH 1/2] archive: Point out that mixed main/contrib source
>  packages can exist

Thank you for the patches.  Would you mind combining them into a single
commit?  That way it makes more sense to say that applying this does in
fact resolve the whole bug.

> Signed-off-by: Simon McVittie 

Please drop this trailer, as it has no semantics for policy.git (we have
no DEVELOPER-CERTIFICATE or similar).

> +If a source package is in the *main* archive area, then at least one of
> +the binary packages that it produces must be in the *main* archive area,
> +and each of the remaining packages must be in either the *main* or *contrib*
> +archive area. Each binary package's archive area is indicated by its
> +``Section`` field: see :ref:`s-subsections`.

Minor suggestion: "binary packages it produces" could be just "its
binary packages".

> +Source packages in *main* with a mixture of *main* and *contrib* binary
> +packages should be limited to situations where it would be inconvenient
> +to split the source package. [...]

How about saying that this is for technical reasons, not anything
philosophical?

> +In particular, build-dependencies outside *main* are not allowed in
> +these source packages, but the *contrib* binary packages may have runtime
> +dependencies outside *main*.

Please rephrase this in terms of "must" rather than "not allowed".

> A source package in non-free cannot produce contrib binary packages,
> because we want main + contrib to be self-contained: if you download
> all main or contrib source packages, that should give you the source
> code of all main and contrib binary packages.

I wonder if this idea that we want main+contrib to be self-contained
should be included in the text somehow?  Or is it obvious?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Holger Levsen]

On Fri, Oct 29, 2021 at 10:51:44AM +0100, Simon McVittie wrote:
> Here are some updated patches for Policy, incorporating this requirement.

thanks for your work on this, Simon.
 
> I have not attempted to incorporate the corner case involving
> build-profiles. I think if we were going to do that, it would require
> documenting build-profiles first (#757760), and maybe even then it's too
> much of a corner-case to be documenting unless/until it actually happens.

sounds good.

> From a332e4e787837cac0856c9c36d6e87e9f19197e2 Mon Sep 17 00:00:00 2001
> From: Simon McVittie 
> Date: Thu, 9 Sep 2021 15:43:20 +0100
> Subject: [PATCH 1/2] archive: Point out that mixed main/contrib source
>  packages can exist
> 
> Most source packages produce only binary packages in the same archive
> area, but a few source packages in main (such as bumblebee) produce
> a mixture of main and contrib binary packages.
> 
> If an upstream project is in this situation (for example a program with
> optional plugins that have non-free dependencies) it isn't entirely
> obvious how to package it; clarify that a single source package in main
> is considered to be appropriate in this case, as long as no non-free
> build-dependencies are required.
> 
> Signed-off-by: Simon McVittie 
> Closes: #994008
> ---
>  policy/ch-archive.rst | 21 +
>  1 file changed, 21 insertions(+)
> 
> diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
> index ab04261..3d40f55 100644
> --- a/policy/ch-archive.rst
> +++ b/policy/ch-archive.rst
> @@ -130,6 +130,27 @@ In addition, the packages in *main*
>  
>  - must meet all policy requirements presented in this manual.
>  
> +If a source package is in the *main* archive area, then at least one of
> +the binary packages that it produces must be in the *main* archive area,
> +and each of the remaining packages must be in either the *main* or *contrib*
> +archive area. Each binary package's archive area is indicated by its
> +``Section`` field: see :ref:`s-subsections`.
> +
> +Source packages in *main* with a mixture of *main* and *contrib* binary
> +packages should be limited to situations where it would be inconvenient
> +to split the source package. If it is straightforward to split the source
> +package into a *main* part and a *contrib* part that are compiled
> +separately, then those parts should be represented as separate source
> +packages.
> +
> +When a *main* source package has a mixture of *main* and *contrib*
> +binary packages, the source package and the *main* binary packages must
> +follow the requirements for *main* packages, but the *contrib* binary
> +packages may follow the weaker requirements for *contrib* packages.
> +In particular, build-dependencies outside *main* are not allowed in
> +these source packages, but the *contrib* binary packages may have runtime
> +dependencies outside *main*.
> +
>  .. _s-contrib:
>  
>  The contrib archive area
> -- 
> 2.33.1

seconded.

maybe it would be better to replace 'complied' with 'build' in the patch
above? Obviously I'd also second this patch with that replacement...


> From 14cd80454fc2ef8122315a1edcc05eed43106583 Mon Sep 17 00:00:00 2001
> From: Simon McVittie 
> Date: Thu, 9 Sep 2021 15:53:20 +0100
> Subject: [PATCH 2/2] archive: Clarify binaries produced by contrib and
>  non-free source
> 
> A source package outside main cannot produce main binary packages, because
> we want main to be self-contained: if you download all main source
> packages, that should give you the source code of all main binary
> packages.
> 
> A source package in contrib cannot produce non-free binary packages,
> because by definition contrib only contains free software (with non-free
> dependencies, but those are not part of the source code).
> 
> A source package in non-free cannot produce contrib binary packages,
> because we want main + contrib to be self-contained: if you download
> all main or contrib source packages, that should give you the source
> code of all main and contrib binary packages.
> 
> Signed-off-by: Simon McVittie 
> ---
>  policy/ch-archive.rst | 8 
>  1 file changed, 8 insertions(+)
> 
> diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
> index 3d40f55..0979d87 100644
> --- a/policy/ch-archive.rst
> +++ b/policy/ch-archive.rst
> @@ -177,6 +177,10 @@ Examples of packages which would be included in 
> *contrib* are:
>  -  wrapper packages or other sorts of free accessories for non-free
> programs.
>  
> +If a source package is in the *contrib* archive area, then each of the
> +binary packages that it produces must also be in the *contrib* archive
> +area.
> +
>  .. _s-non-free:
>  
>  The non-free archive area
> @@ -199,6 +203,10 @@ In addition, the packages in *non-free*
>  -  must meet all policy requirements presented in this manual that it is
> possible for them to meet.  [#]_
>  
> +If a source package is in the *non-free* archive area, then each of the
> +binary packages that it produces must 

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Simon McVittie]

On Thu, 09 Sep 2021 at 21:15:06 +0200, Ansgar wrote:
> On Thu, 2021-09-09 at 17:39 +0100, Simon McVittie wrote:
> > In the form of a table, the allowed source/binary combinations are:
> > 
> >   |    binary   |
> >   | main  contrib  non-free |
> >  -|-|
> >  main |  yes    yes  -  |
> >  source  contrib  |   - yes  -  |
> >  non-free |   -  -  yes |
> > 
> > ftp team: is this correct?
> 
> Yes. But source packages in main must also produce at least one binary
> package in main[1].

Here are some updated patches for Policy, incorporating this requirement.

I have not attempted to incorporate the corner case involving
build-profiles. I think if we were going to do that, it would require
documenting build-profiles first (#757760), and maybe even then it's too
much of a corner-case to be documenting unless/until it actually happens.

> I personally would prefer if we would avoid using this feature too much
> if possible.

I've added some wording to try to express that.

smcv
>From a332e4e787837cac0856c9c36d6e87e9f19197e2 Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Thu, 9 Sep 2021 15:43:20 +0100
Subject: [PATCH 1/2] archive: Point out that mixed main/contrib source
 packages can exist

Most source packages produce only binary packages in the same archive
area, but a few source packages in main (such as bumblebee) produce
a mixture of main and contrib binary packages.

If an upstream project is in this situation (for example a program with
optional plugins that have non-free dependencies) it isn't entirely
obvious how to package it; clarify that a single source package in main
is considered to be appropriate in this case, as long as no non-free
build-dependencies are required.

Signed-off-by: Simon McVittie 
Closes: #994008
---
 policy/ch-archive.rst | 21 +
 1 file changed, 21 insertions(+)

diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
index ab04261..3d40f55 100644
--- a/policy/ch-archive.rst
+++ b/policy/ch-archive.rst
@@ -130,6 +130,27 @@ In addition, the packages in *main*
 
 - must meet all policy requirements presented in this manual.
 
+If a source package is in the *main* archive area, then at least one of
+the binary packages that it produces must be in the *main* archive area,
+and each of the remaining packages must be in either the *main* or *contrib*
+archive area. Each binary package's archive area is indicated by its
+``Section`` field: see :ref:`s-subsections`.
+
+Source packages in *main* with a mixture of *main* and *contrib* binary
+packages should be limited to situations where it would be inconvenient
+to split the source package. If it is straightforward to split the source
+package into a *main* part and a *contrib* part that are compiled
+separately, then those parts should be represented as separate source
+packages.
+
+When a *main* source package has a mixture of *main* and *contrib*
+binary packages, the source package and the *main* binary packages must
+follow the requirements for *main* packages, but the *contrib* binary
+packages may follow the weaker requirements for *contrib* packages.
+In particular, build-dependencies outside *main* are not allowed in
+these source packages, but the *contrib* binary packages may have runtime
+dependencies outside *main*.
+
 .. _s-contrib:
 
 The contrib archive area
-- 
2.33.1

>From 14cd80454fc2ef8122315a1edcc05eed43106583 Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Thu, 9 Sep 2021 15:53:20 +0100
Subject: [PATCH 2/2] archive: Clarify binaries produced by contrib and
 non-free source

A source package outside main cannot produce main binary packages, because
we want main to be self-contained: if you download all main source
packages, that should give you the source code of all main binary
packages.

A source package in contrib cannot produce non-free binary packages,
because by definition contrib only contains free software (with non-free
dependencies, but those are not part of the source code).

A source package in non-free cannot produce contrib binary packages,
because we want main + contrib to be self-contained: if you download
all main or contrib source packages, that should give you the source
code of all main and contrib binary packages.

Signed-off-by: Simon McVittie 
---
 policy/ch-archive.rst | 8 
 1 file changed, 8 insertions(+)

diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
index 3d40f55..0979d87 100644
--- a/policy/ch-archive.rst
+++ b/policy/ch-archive.rst
@@ -177,6 +177,10 @@ Examples of packages which would be included in *contrib* are:
 -  wrapper packages or other sorts of free accessories for non-free
programs.
 
+If a source package is in the *contrib* archive area, then each of the
+binary packages that it produces must also be in the *contrib* archive
+area.
+
 .. _s-non-free:
 
 The 

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Ansgar]

On Thu, 2021-09-09 at 12:48 -0700, Sean Whitton wrote:
> > But source packages in main must also produce at least one binary
> > package in main[1].
> 
> Let's include that point in Policy.
> 
> > [1]: Probably with the default build profile if we care about corner
> >    cases.
> 
> Are we sure about this?  Is this a bug in current dak or a general
> problem with the semantics we're discussing?

We expect all packages uploaded to Debian to use the default build
profile (as a policy). Not doing so will probably cause various issues
(such as a binNMU reverting to the default profile thus changing the
package). So for the archive a binary only built in non-default build
profiles should just be ignored (for binary-NEW, choice of archive
area, ...) as it shouldn't appear in Debian; packages are free to do
whatever they want with build profiles.

This behavior was asked for in https://bugs.debian.org/913965

If we allowed packages to use non-default build profiles then we would
probably have to revisit this.

(I would expect this corner case to not happen in practice.)

> > I personally would prefer if we would avoid using this feature too much
> > if possible. It is simpler to understand when archive areas are self-
> > contained (IMHO). Outside Debian archive areas are used differently,
> > e.g., for different "branches" or similar; sources building binary
> > packages across multiple archive areas also find strange corner cases
> > now and then that are not handled correctly.
> 
> It would be good if we included in Policy the idea that for technical
> reasons, it is best to use separate source packages (only) when that is
> not otherwise too inconvenient.
> 
> I am not sure we have a consensus on avoiding using this feature for the
> sake of simplicity of understanding, so let's exclude that idea for now.

Sure, that seems reasonable enough.

Ansgar

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Sean Whitton]

Hello,

On Thu 09 Sep 2021 at 09:15PM +02, Ansgar wrote:

> On Thu, 2021-09-09 at 17:39 +0100, Simon McVittie wrote:
>>
>> ftp team: is this correct?
>
> Yes.

Indeed.

> But source packages in main must also produce at least one binary
> package in main[1].

Let's include that point in Policy.

> [1]: Probably with the default build profile if we care about corner
>cases.

Are we sure about this?  Is this a bug in current dak or a general
problem with the semantics we're discussing?

> I personally would prefer if we would avoid using this feature too much
> if possible. It is simpler to understand when archive areas are self-
> contained (IMHO). Outside Debian archive areas are used differently,
> e.g., for different "branches" or similar; sources building binary
> packages across multiple archive areas also find strange corner cases
> now and then that are not handled correctly.

It would be good if we included in Policy the idea that for technical
reasons, it is best to use separate source packages (only) when that is
not otherwise too inconvenient.

I am not sure we have a consensus on avoiding using this feature for the
sake of simplicity of understanding, so let's exclude that idea for now.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Ansgar]

Hi,

On Thu, 2021-09-09 at 17:39 +0100, Simon McVittie wrote:
> In the form of a table, the allowed source/binary combinations are:
> 
>   |    binary   |
>   | main  contrib  non-free |
>  -|-|
>  main |  yes    yes  -  |
>  source  contrib  |   - yes  -  |
>  non-free |   -  -  yes |
> 
> ftp team: is this correct?

Yes. But source packages in main must also produce at least one binary
package in main[1].

  [1]: Probably with the default build profile if we care about corner
   cases.

I personally would prefer if we would avoid using this feature too much
if possible. It is simpler to understand when archive areas are self-
contained (IMHO). Outside Debian archive areas are used differently,
e.g., for different "branches" or similar; sources building binary
packages across multiple archive areas also find strange corner cases
now and then that are not handled correctly.

Ansgar

Bug#994008: debian-policy: Clarify relationship between source and binary packages' archive areas

[Simon McVittie]

Package: debian-policy
Version: 4.6.0.1
Severity: wishlist
Tags: patch
X-Debbugs-Cc: ftpmas...@debian.org

If you have an upstream project consisting of Free source code with a
mixture of Free and non-Free dependencies, it isn't currently clear how
that should be packaged.

For example, if gnome-software added a steam plugin that was itself Free
Software, but had Depends: steam, then it would be in this situation:
the program and the flatpak and snap plugins would still be suitable
for main, but the steam plugin would need to go in contrib.

Based on the precedent seen in the bumblebee package, I believe the
ftp team's intention is that packages like this are allowed to be a
single source package in main, which builds binary packages for main
and contrib, even though this means the contrib binary packages don't
match their source package's archive area.

I think I remember talking to a ftp team member about this in person
and coming to the conclusion that there are good reasons for none of
the other possible source/binary archive area mismatches to be allowed.

In the form of a table, the allowed source/binary combinations are:

  |binary   |
  | main  contrib  non-free |
 -|-|
 main |  yesyes  -  |
 source  contrib  |   - yes  -  |
 non-free |   -  -  yes |

ftp team: is this correct?

If it is, I attach proposed patches for Policy. Their commit messages
attempt to capture the rationale for why the other situations are not
allowed; corrections/clarifications welcome.

Thanks,
smcv
>From eda0f325301bd514e5ac94328dd4b5a01960634a Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Thu, 9 Sep 2021 15:43:20 +0100
Subject: [PATCH 1/2] archive: Point out that mixed main/contrib source
 packages can exist

Most source packages produce only binary packages in the same archive
area, but a few source packages in main (such as bumblebee) produce
a mixture of main and contrib binary packages.

If an upstream project is in this situation (for example a program with
optional plugins that have non-free dependencies) it isn't entirely
obvious how to package it; clarify that a single source package in main
is considered to be appropriate in this case, as long as no non-free
build-dependencies are required.

Signed-off-by: Simon McVittie 
---
 policy/ch-archive.rst | 13 +
 1 file changed, 13 insertions(+)

diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
index ab04261..7829601 100644
--- a/policy/ch-archive.rst
+++ b/policy/ch-archive.rst
@@ -130,6 +130,19 @@ In addition, the packages in *main*
 
 - must meet all policy requirements presented in this manual.
 
+If a source package is in the *main* archive area, then each of the
+binary packages that it produces must be in either the *main* or *contrib*
+archive area. Each binary package's archive area is indicated by its
+``Section`` field: see :ref:`s-subsections`.
+
+When a *main* source package has a mixture of *main* and *contrib*
+binary packages, the source package and the *main* binary packages must
+follow the requirements for *main* packages, but the *contrib* binary
+packages may follow the weaker requirements for *contrib* packages.
+In particular, build-dependencies outside *main* are not allowed in
+these source packages, but the *contrib* binary packages may have runtime
+dependencies outside *main*.
+
 .. _s-contrib:
 
 The contrib archive area
-- 
2.33.0

>From 0b855a4a0dbb348269388985fc45c7887376a245 Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Thu, 9 Sep 2021 15:53:20 +0100
Subject: [PATCH 2/2] archive: Clarify binaries produced by contrib and
 non-free source

A source package outside main cannot produce main binary packages, because
we want main to be self-contained: if you download all main source
packages, that should give you the source code of all main binary
packages.

A source package in contrib cannot produce non-free binary packages,
because by definition contrib only contains free software (with non-free
dependencies, but those are not part of the source code).

A source package in non-free cannot produce contrib binary packages,
because we want main + contrib to be self-contained: if you download
all main or contrib source packages, that should give you the source
code of all main and contrib binary packages.

Signed-off-by: Simon McVittie 
---
 policy/ch-archive.rst | 8 
 1 file changed, 8 insertions(+)

diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
index 7829601..35b44bb 100644
--- a/policy/ch-archive.rst
+++ b/policy/ch-archive.rst
@@ -169,6 +169,10 @@ Examples of packages which would be included in *contrib* are:
 -  wrapper packages or other sorts of free accessories for non-free
programs.
 
+If a source package is in the *contrib* archive area, then each of the
+binary packages that it produces must also be in the