Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Holger Levsen]

On Tue, Jun 13, 2023 at 11:04:07PM +0100, Luca Boccassi wrote:
> ---
>  policy/ap-pkg-alternatives.rst |  3 +++
>  policy/ap-pkg-diversions.rst   |  3 +++
>  policy/ch-binary.rst   | 35 ++
>  3 files changed, 41 insertions(+)
> 
> diff --git a/policy/ap-pkg-alternatives.rst b/policy/ap-pkg-alternatives.rst
> index ffa2163..6f7780f 100644
> --- a/policy/ap-pkg-alternatives.rst
> +++ b/policy/ap-pkg-alternatives.rst
> @@ -24,3 +24,6 @@ See the :manpage:`update-alternatives(8)` man page for 
> details.
>  If ``update-alternatives`` does not seem appropriate you may wish to
>  consider using diversions instead.
>  
> +Do not use alternatives for ``systemd`` configuration files. See
> +:doc:`ch-binary` for more information.
> +
> diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
> index fe360d1..d299d04 100644
> --- a/policy/ap-pkg-diversions.rst
> +++ b/policy/ap-pkg-diversions.rst
> @@ -81,3 +81,6 @@ when the file does not exist.
>  Do not attempt to divert a conffile, as ``dpkg`` does not handle it
>  well.
>  
> +Do not use diversions for files that have their own native override 
> mechanisms,
> +such as ``systemd`` unit files. See :doc:`ch-binary` for more information.
> +
> diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> index e517f26..19635e7 100644
> --- a/policy/ch-binary.rst
> +++ b/policy/ch-binary.rst
> @@ -371,6 +371,41 @@ against earlier versions of something that previously 
> did not use
>  ``update-alternatives``; this is an exception to the usual rule that
>  versioned conflicts should be avoided.)
>  
> +Diversions are primarily intended as a tool for local administrators or local
> +packages to override the behavior of Debian. While there are some 
> circumstances
> +where one Debian package may need to divert a file of another Debian package,
> +those circumstances are rare and diversions should only be used as a last 
> resort
> +when no other suitable mechanism exists. Diversion of a file in one Debian
> +package by another Debian package should be coordinated between the 
> maintainers
> +of those packages. Maintainers should strongly prefer using other overriding
> +mechanisms, instead of diversions, whenever those other mechanisms are
> +sufficient to accomplish the same goal. In other words, diversions in 
> packages
> +should be considered a last resort.
> +
> +One specific case of that rule is that configuration files used by
> +``systemd`` components, such as `units,
> +`_
> +`udev rules,
> +`_
> +`tmpfiles.d,
> +`_
> +`modules-load.d,
> +`_,
> +`sysusers
> +`_
> +and other such files, including those specific to systemd daemons
> +(e.g.:  `/etc/systemd/system.conf).
> +`_
> +must not be diverted by any Debian package. Instead, use `masking and 
> drop-ins
> +`_.
> +
> +Alternatives must never be used for ``systemd`` configuration files. The
> +alternatives system does not know how to apply changes to services when 
> updating
> +alternatives, so the resulting behavior would be confusing and unpredictable.
> +Instead, `aliases
> +`_
> +can be used to provide alternative implementations of the same named unit.
> +
 
seconded.

and wondering, if there should be a recommendation similar to consulting 
debian-devl@l.d.o
when introducing epochs, or...

..the other way round: should it be explicitly spelled out that, unlike for 
epochs, 
in general there's no need to consult -devel for diversions for packages as 
they are 
generally ment for local admins and only in very very very rare cases...

(maybe be even more verbose about "other overriding mechanisms"?!)

but in any case seconded, things can always be improved, and this is good.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Punk ist nicht tot.
Punk trägt Maske, ist solidarisch und schützt sich und andere.
(@Kreuzpirat)


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Wed, 14 Jun 2023 at 16:27, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > I.e.: if the attached version works, then that's good enough for me.
>
> Seconded.  Thank you for your work on multiple revisions of this patch!

Thanks for your patience and feedback!

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Luca Boccassi  writes:

> I.e.: if the attached version works, then that's good enough for me.

Seconded.  Thank you for your work on multiple revisions of this patch!

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

> diff --git a/policy/ap-pkg-alternatives.rst b/policy/ap-pkg-alternatives.rst
> index ffa2163..6f7780f 100644
> --- a/policy/ap-pkg-alternatives.rst
> +++ b/policy/ap-pkg-alternatives.rst
> @@ -24,3 +24,6 @@ See the :manpage:`update-alternatives(8)` man page for 
> details.
>  If ``update-alternatives`` does not seem appropriate you may wish to
>  consider using diversions instead.
>
> +Do not use alternatives for ``systemd`` configuration files. See
> +:doc:`ch-binary` for more information.
> +
> diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
> index fe360d1..d299d04 100644
> --- a/policy/ap-pkg-diversions.rst
> +++ b/policy/ap-pkg-diversions.rst
> @@ -81,3 +81,6 @@ when the file does not exist.
>  Do not attempt to divert a conffile, as ``dpkg`` does not handle it
>  well.
>
> +Do not use diversions for files that have their own native override 
> mechanisms,
> +such as ``systemd`` unit files. See :doc:`ch-binary` for more information.
> +
> diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> index e517f26..19635e7 100644
> --- a/policy/ch-binary.rst
> +++ b/policy/ch-binary.rst
> @@ -371,6 +371,41 @@ against earlier versions of something that previously 
> did not use
>  ``update-alternatives``; this is an exception to the usual rule that
>  versioned conflicts should be avoided.)
>
> +Diversions are primarily intended as a tool for local administrators or local
> +packages to override the behavior of Debian. While there are some 
> circumstances
> +where one Debian package may need to divert a file of another Debian package,
> +those circumstances are rare and diversions should only be used as a last 
> resort
> +when no other suitable mechanism exists. Diversion of a file in one Debian
> +package by another Debian package should be coordinated between the 
> maintainers
> +of those packages. Maintainers should strongly prefer using other overriding
> +mechanisms, instead of diversions, whenever those other mechanisms are
> +sufficient to accomplish the same goal. In other words, diversions in 
> packages
> +should be considered a last resort.

(When applying the patch I'll probably edit this down to reduce redundancy.)

> +One specific case of that rule is that configuration files used by
> +``systemd`` components, such as `units,
> +`_
> +`udev rules,
> +`_
> +`tmpfiles.d,
> +`_
> +`modules-load.d,
> +`_,
> +`sysusers
> +`_
> +and other such files, including those specific to systemd daemons
> +(e.g.:  `/etc/systemd/system.conf).
> +`_
> +must not be diverted by any Debian package. Instead, use `masking and 
> drop-ins
> +`_.
> +
> +Alternatives must never be used for ``systemd`` configuration files. The
> +alternatives system does not know how to apply changes to services when 
> updating
> +alternatives, so the resulting behavior would be confusing and unpredictable.
> +Instead, `aliases
> +`_
> +can be used to provide alternative implementations of the same named unit.
> +
>  .. _s-maintscriptprompt:
>
>  Prompting in maintainer scripts

Seconded.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 13 Jun 2023 at 22:59, Luca Boccassi  wrote:
>
> On Tue, 13 Jun 2023 at 22:49, Russ Allbery  wrote:
> >
> > Luca Boccassi  writes:
> >
> > > That essentially means it's fine to use diversions and ship releases
> > > using them, so that's exactly what will happen as per Murphy's law.
> >
> > I think we're reaching a consensus that "must" is appropriate for the
> > systemd configuration files, so this discussion is about how to phrase the
> > general guidance that isn't specific to systemd.
> >
> > I'm therefore not really understanding the argument that you're trying to
> > make here.  Are you trying to say that we should just delete everything in
> > Policy that isn't a "must" because it serves no useful purpose?  If not,
> > then why are you taking this incredibly aggressive position that general
> > guidance is pointless unless it says "must"?
>
> Waait, maybe I misunderstood - I thought the change was requested for
> all additions, including systemd files? Is that not the case? (It
> would be obvious what it refers to on a merge request :-P )
> If I can keep a 'must' for systemd config files, then I'm ok with changing.

I.e.: if the attached version works, then that's good enough for me.

Kind regards,
Luca Boccassi
From ed947ae43dc48d4bee1b609fe3942ff1a01e7a6b Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Mon, 8 May 2023 03:21:14 +0100
Subject: [PATCH] Forbid using dpkg-divert/alternatives when there are native
 mechanisms

The supported mechanism for augmenting, changing, overriding and
disabling systemd configuration files is natively supported and fully
integrated in Debian, via drop-ins, hierarchical overrides, and
masking. dpkg-divert is not integrated in systemd tools so its use
is completely hidden in logs and status interfaces, and it is specific
to Debian and thus diverges from what users expect as implemented by
all other distros, going against one of the core goals of the systemd
project which is to provide a uniform interface regardless of distro
vendor or flavour.

Clarify that diversions and alternatives should only be used when
needed, with cooperation, and must not be used when there are native
mechanisms to obtain the same goals, and use systemd files as an
example.
---
 policy/ap-pkg-alternatives.rst |  3 +++
 policy/ap-pkg-diversions.rst   |  3 +++
 policy/ch-binary.rst   | 35 ++
 3 files changed, 41 insertions(+)

diff --git a/policy/ap-pkg-alternatives.rst b/policy/ap-pkg-alternatives.rst
index ffa2163..6f7780f 100644
--- a/policy/ap-pkg-alternatives.rst
+++ b/policy/ap-pkg-alternatives.rst
@@ -24,3 +24,6 @@ See the :manpage:`update-alternatives(8)` man page for details.
 If ``update-alternatives`` does not seem appropriate you may wish to
 consider using diversions instead.
 
+Do not use alternatives for ``systemd`` configuration files. See
+:doc:`ch-binary` for more information.
+
diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
index fe360d1..d299d04 100644
--- a/policy/ap-pkg-diversions.rst
+++ b/policy/ap-pkg-diversions.rst
@@ -81,3 +81,6 @@ when the file does not exist.
 Do not attempt to divert a conffile, as ``dpkg`` does not handle it
 well.
 
+Do not use diversions for files that have their own native override mechanisms,
+such as ``systemd`` unit files. See :doc:`ch-binary` for more information.
+
diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
index e517f26..19635e7 100644
--- a/policy/ch-binary.rst
+++ b/policy/ch-binary.rst
@@ -371,6 +371,41 @@ against earlier versions of something that previously did not use
 ``update-alternatives``; this is an exception to the usual rule that
 versioned conflicts should be avoided.)
 
+Diversions are primarily intended as a tool for local administrators or local
+packages to override the behavior of Debian. While there are some circumstances
+where one Debian package may need to divert a file of another Debian package,
+those circumstances are rare and diversions should only be used as a last resort
+when no other suitable mechanism exists. Diversion of a file in one Debian
+package by another Debian package should be coordinated between the maintainers
+of those packages. Maintainers should strongly prefer using other overriding
+mechanisms, instead of diversions, whenever those other mechanisms are
+sufficient to accomplish the same goal. In other words, diversions in packages
+should be considered a last resort.
+
+One specific case of that rule is that configuration files used by
+``systemd`` components, such as `units,
+`_
+`udev rules,
+`_
+`tmpfiles.d,
+`_
+`modules-load.d,
+`_,

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 13 Jun 2023 at 22:49, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > That essentially means it's fine to use diversions and ship releases
> > using them, so that's exactly what will happen as per Murphy's law.
>
> I think we're reaching a consensus that "must" is appropriate for the
> systemd configuration files, so this discussion is about how to phrase the
> general guidance that isn't specific to systemd.
>
> I'm therefore not really understanding the argument that you're trying to
> make here.  Are you trying to say that we should just delete everything in
> Policy that isn't a "must" because it serves no useful purpose?  If not,
> then why are you taking this incredibly aggressive position that general
> guidance is pointless unless it says "must"?

Waait, maybe I misunderstood - I thought the change was requested for
all additions, including systemd files? Is that not the case? (It
would be obvious what it refers to on a merge request :-P )
If I can keep a 'must' for systemd config files, then I'm ok with changing.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Luca Boccassi  writes:

> That essentially means it's fine to use diversions and ship releases
> using them, so that's exactly what will happen as per Murphy's law.

I think we're reaching a consensus that "must" is appropriate for the
systemd configuration files, so this discussion is about how to phrase the
general guidance that isn't specific to systemd.

I'm therefore not really understanding the argument that you're trying to
make here.  Are you trying to say that we should just delete everything in
Policy that isn't a "must" because it serves no useful purpose?  If not,
then why are you taking this incredibly aggressive position that general
guidance is pointless unless it says "must"?

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Tue 13 Jun 2023 at 08:45PM +01, Luca Boccassi wrote:

> That essentially means it's fine to use diversions and ship releases
> using them, so that's exactly what will happen as per Murphy's law.
> [...]

I don't believe you've made any new arguments in the message to which
I'm replying.  Based on that, and the other activity this bug has seen,
I don't think it would be premature for us to start gathering formal
seconds of a version of your patch with my "should strongly prefer"
wording substituted for one part of it.
Would you like to prepare that patch, or shall one of Russ or I do so?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 13 Jun 2023 08:39:24 -0700 Russ Allbery  wrote:
> Sean Whitton  writes:
> 
> > I still find myself feeling queasy about adding this must-with-
caveat.
> > It feels like with the caveat you're trying to get something
between
> > "must" and "should", but then rather than build down from "must",
why
> > not build up from "should"?  I would like to propose:
> 
> > Maintainers should strongly prefer using other overriding
> > mechanisms, instead of diversions, whenever those other
mechanisms
> > are sufficient to accomplish the same goal.  In other words,
> > diversions in packages should be considered a last resort.
> 
> This sounds good to me.  The argument for building up from should
instead
> of down from must seems compelling.

That essentially means it's fine to use diversions and ship releases
using them, so that's exactly what will happen as per Murphy's law.

> I think that this accomplishes everything, in terms of guidance for
> our
> maintainers, that the must-with-caveat wording does.  But it avoids
> saying that using diversions over a native mechanism in and of itself
> renders a package RC-buggy, which doesn't seem right to me.
> 
> My own experience with diversions is limited, and so I accept that I
> may
> well be naively underestimating the badness of the edge cases.

Why would it not be RC-buggy? Not RC-buggy means we are happy if it
ships in a release. What does that buy us? Why wouldn't we want to
direct maintainers toward the better alternative, that is current
practice as of today, and instead let them reintroduce a mechanism that
we agree is inferior and was just removed from the distribution?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Sean Whitton  writes:

> I still find myself feeling queasy about adding this must-with-caveat.
> It feels like with the caveat you're trying to get something between
> "must" and "should", but then rather than build down from "must", why
> not build up from "should"?  I would like to propose:

> Maintainers should strongly prefer using other overriding
> mechanisms, instead of diversions, whenever those other mechanisms
> are sufficient to accomplish the same goal.  In other words,
> diversions in packages should be considered a last resort.

This sounds good to me.  The argument for building up from should instead
of down from must seems compelling.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Tue 06 Jun 2023 at 12:16PM +01, Luca Boccassi wrote:

> My understanding is that "should" is effectively optional, ie: it is
> not enough to make a package rc-buggy, and while they are generally
> followed, there is no actual hard requirement to do so. That is not
> enough for me and for this use case. So again my goal here is to make
> an hypothetical future dpkg-divert or alternative of a unit file
> (wherever it is shipped from) instantly and unambiguously rc-buggy,
> with no leeway (again there are no longer any such cases in the
> archive as of bookworm, so nothing is affected). If you can suggest an
> alternative appropriate wording that can achieve the same effect, that
> would be fine to me - I don't mind how it's written, as long as it
> meets this requirement.

We have more than two bug severities in the BTS and more than two levels
of normative language in Policy.  It's simply untrue that all but one of
each of those is optional.  Debian's normative landscape is more complex
than that.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Tue 06 Jun 2023 at 07:56PM -07, Russ Allbery wrote:

> Sean Whitton  writes:
>
>> I think what's a bit peculiar here is using "must" for a case where
>> there might be package-specific exceptions.  In other cases, Policy uses
>> "should" for these cases.  Typically "must" rules are simple and
>> completely determinate.
>
> I prefer that too, but in this case, it feels like must is appropriate
> for at least systemd configuration files.

I've spent some time thinking about this, and I now agree that we should
add a "must" for systemd unit files, and the other classes of systemd
configuration mentioned in the patch.

> And also, just intuitively, I feel like must is correct when people
> are using diversions rather than a native mechanism.  Diversions add
> weird edge cases and we really shouldn't be using them lightly.
>
> The wording I proposed and that Luca has now adopted therefore uses
> must with a caveat.  Does that sound okay to you?

I still find myself feeling queasy about adding this must-with-caveat.
It feels like with the caveat you're trying to get something between
"must" and "should", but then rather than build down from "must", why
not build up from "should"?  I would like to propose:

Maintainers should strongly prefer using other overriding
mechanisms, instead of diversions, whenever those other mechanisms
are sufficient to accomplish the same goal.  In other words,
diversions in packages should be considered a last resort.

I think that this accomplishes everything, in terms of guidance for our
maintainers, that the must-with-caveat wording does.  But it avoids
saying that using diversions over a native mechanism in and of itself
renders a package RC-buggy, which doesn't seem right to me.

My own experience with diversions is limited, and so I accept that I may
well be naively underestimating the badness of the edge cases.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

> diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> index e517f26..4ceec3f 100644
> --- a/policy/ch-binary.rst
> +++ b/policy/ch-binary.rst
> +
> +Alternatives must never be used for ``systemd`` configuration files.

"Must not" rather than "must never" is our standard phrasing,
and "must not" is not any normatively weaker than "must never".

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 11:34:17PM +0100, Luca Boccassi wrote:
> On Wed, 7 Jun 2023 00:01:42 +0200 Bill Allombert 
> > This is beside the point. Your problematic statement was
> > "The whole project is moving toward git and Salsa ".
> > This is not conducive of productive interaction.
> 
> It was not problematic at all, unless you are deliberately trying to
> see problems where there are none. It was an hyperbole: a huge number
> of packages and teams are using effectively Salsa workflows for
> collaboration today. Are you denying this?

This is not important. What matters is that this statement could be read as
excluding from the project or belittling developers that are not moving toward
Salsa.  This is inappropriate. Even assuming it was not your intent, such
statements creates friction in the project by making people uncomfortable
without any benefit, and so should be avoided.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Holger Levsen]

On Wed, Jun 07, 2023 at 08:19:08AM -0700, Russ Allbery wrote:
> I would like to add more documentation like this around systemd-related
> things to Policy because systemd is complicated and has a lot of options,
> so people who aren't deeply familiar with it will easily miss best
> practices in its reams of very good but overwhelming documentation, and
> Debian should be opinionated about steering people towards best practices
> for our primary init system, just as we were very opinionated about how to
> write init scripts for sysvinit.
[...]
> I definitely agree that Policy should have a whole section devoted to
> systemd and its configuration files, but I don't want to try to boil the
> ocean in one bug.  But yes, we should be working towards that, IMO.

yes! (to everything in said email.) thank you.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Only change is constant.


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Wed, Jun 07, 2023 at 08:19:08AM -0700, Russ Allbery wrote:
> > In general, policy proscription are only useful when the description of
> > a better mechanism is provided.  But there is no place for that in this
> > section.
> 
> I'm not sure I understand this statement, since describing a better
> mechanism is exactly the point of that language about systemd.  We link
> directly to those better mechanisms (masking, drop-ins, and, for
> alternatives, aliases).

What I meant was that this section in not the right place for systemd or other
software configuration detail, because nobody will look for systemd
configuration detail in the dpkg-divert section.

> I definitely agree that Policy should have a whole section devoted to
> systemd and its configuration files, but I don't want to try to boil the
> ocean in one bug.  But yes, we should be working towards that, IMO.

Yes that what I wanted to say.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Bill Allombert  writes:
> On Tue, Jun 06, 2023 at 07:56:14PM -0700, Russ Allbery wrote:

>> I prefer that too, but in this case, it feels like must is appropriate
>> for at least systemd configuration files.  And also, just intuitively,
>> I feel like must is correct when people are using diversions rather
>> than a native mechanism.  Diversions add weird edge cases and we really
>> shouldn't be using them lightly.

>> The wording I proposed and that Luca has now adopted therefore uses
>> must with a caveat.  Does that sound okay to you?

> I do not think appropriate for the policy to list systemd or any other
> packages specifically in this section.

My rationale for listing systemd specifically is:

1. This is a common case where one package wants to change the behavior of
   another and I expect it to become increasingly common as people make
   more use of systemd security features. For example, I would expect
   cases where the default systemd configuration for a service disallows
   access to large swaths of the file system, and then installing a plugin
   for that package grants additional access to specific paths used by
   that plugin.

2. We understand what people are trying to do in this case and can offer
   very specific guidance, whereas we can't in general for arbitrary
   packages. This makes Policy more useful for packagers; instead of
   seeing a general rule that they can't use diversions but being left on
   their own to figure out how to solve their problem, Policy can
   explicitly say "here are the mechanisms to use for this case, they can
   do everything you would want to do with diversions."

I would like to add more documentation like this around systemd-related
things to Policy because systemd is complicated and has a lot of options,
so people who aren't deeply familiar with it will easily miss best
practices in its reams of very good but overwhelming documentation, and
Debian should be opinionated about steering people towards best practices
for our primary init system, just as we were very opinionated about how to
write init scripts for sysvinit.

> If a package set up a diversion that breaks another, then it is buggy,
> whatever policy say. If the diversion does not cause any breakage, there
> is no purpose for policy to declare it a RC bug.

I don't really agree with this, and I thought we had reached some
agreement in the other branch of this thread that this isn't quite
complete and it's okay to ban diversions if there's a better mechanism
available.

I think that if diversions and some other mechanism work equally well
technically, we should forbid using diversions and require using the other
mechanism.  This is in part based on the long threads that came out of the
/usr-merge discussions, which made clear that diversions are a very
complicated feature with a lot of edge cases, and it's possible to get
into trouble with them because they're manipulating the packaging system
at a very low level.  If there's some alternative that's less invasive, I
do feel like using diversions is a fairly serious bug because it adds to
the instability of the system.

In isolation, you could talk me into it being an important bug rather than
an RC bug, but that feels like a lot of nuance here and must feels cleaner
and more straightforward.

That being said, I'd rather back down to should than remove the
systemd-specific details, since I think the latter are quite valuable.

> In general, policy proscription are only useful when the description of
> a better mechanism is provided.  But there is no place for that in this
> section.

I'm not sure I understand this statement, since describing a better
mechanism is exactly the point of that language about systemd.  We link
directly to those better mechanisms (masking, drop-ins, and, for
alternatives, aliases).

I definitely agree that Policy should have a whole section devoted to
systemd and its configuration files, but I don't want to try to boil the
ocean in one bug.  But yes, we should be working towards that, IMO.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Wed, 7 Jun 2023 12:23:15 +0200 Bill Allombert 
wrote:
> On Tue, Jun 06, 2023 at 07:56:14PM -0700, Russ Allbery wrote:
> > Sean Whitton  writes:
> > 
> > > I think what's a bit peculiar here is using "must" for a case
where
> > > there might be package-specific exceptions.  In other cases,
Policy uses
> > > "should" for these cases.  Typically "must" rules are simple and
> > > completely determinate.
> > 
> > I prefer that too, but in this case, it feels like must is
appropriate for
> > at least systemd configuration files.  And also, just intuitively,
I feel
> > like must is correct when people are using diversions rather than a
native
> > mechanism.  Diversions add weird edge cases and we really shouldn't
be
> > using them lightly.
> > 
> > The wording I proposed and that Luca has now adopted therefore uses
must
> > with a caveat.  Does that sound okay to you?
> 
> I do not think appropriate for the policy to list systemd or any
> other packages specifically in this section.

There are already mentions of systemd and unit files in policy before
this change, in other sections. For what reason are those fine and this
is not?

> If a package set up a diversion that breaks another, then it is
buggy,
> whatever policy say. If the diversion does not cause any breakage,
there is
> no purpose for policy to declare it a RC bug.

Earlier you wrote:

> Policy is for promoting interoperability and documenting current
practices.

This is what this change is doing - promoting interoperability by
forbidding known broken and unsupported practices, and instead steering
toward known working and supported practices. It also documents what
the current practice w.r.t. overriding and aliasing is as of Bookworm.

> In general, policy proscription are only useful when the description
of a
> better mechanism is provided.  But there is no place for that in this
section.

It is explained how to use a better mechanism? With links to all the
relevant documentation et cetera. Are you saying it's not exhaustive
enough and you want more details added? I am wary of excessively
redefining and duplicating existing documentation, especially because
it will naturally evolve (in backward-compatible ways) and any such
copy would get out of date and be confusing. 

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 07:56:14PM -0700, Russ Allbery wrote:
> Sean Whitton  writes:
> 
> > I think what's a bit peculiar here is using "must" for a case where
> > there might be package-specific exceptions.  In other cases, Policy uses
> > "should" for these cases.  Typically "must" rules are simple and
> > completely determinate.
> 
> I prefer that too, but in this case, it feels like must is appropriate for
> at least systemd configuration files.  And also, just intuitively, I feel
> like must is correct when people are using diversions rather than a native
> mechanism.  Diversions add weird edge cases and we really shouldn't be
> using them lightly.
> 
> The wording I proposed and that Luca has now adopted therefore uses must
> with a caveat.  Does that sound okay to you?

I do not think appropriate for the policy to list systemd or any
other packages specifically in this section.

If a package set up a diversion that breaks another, then it is buggy,
whatever policy say. If the diversion does not cause any breakage, there is
no purpose for policy to declare it a RC bug.

In general, policy proscription are only useful when the description of a
better mechanism is provided.  But there is no place for that in this section.

It would more suitable to have a separate section or document defining the
interface between systemd and other packages, that would explain how to avoid
diversion by providing better solutions rather than direct proscriptions, with
more details than just "use native systemd configuration files".

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 07:52:35PM -0700, Russ Allbery wrote:
> Bill Allombert  writes:
> > On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
> 
> >> If you prefer, I can reword the general rule to be stricter, ie:
> >> "packages must not use diversions where native mechanisms are
> >> available" or so. Would this be better?
> 
> > "native mechanisms" seems to vague.
> 
> I suggested "must not be used when a suitable override mechanism that
> accomplishes the same goal is already available."  Does that sound okay?
> It's a bit weaker and opens the door to arguments about whether the native
> override mechanism accomplishes the same goal, but I think that's a
> feature here rather than a bug.

It is fine with me. It is in line with the notion that diversion are a last
resort, but not precluded entirely.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Luca Boccassi  writes:

> There is a native alternative: aliases. We already do that today in the
> archive for the well-known 'dbus' unit. The dbus package ships the
> reference implementation, and dbus-broker which is an alternative
> implementation ships its own unit, which also lists 'dbus.service' as an
> alias. I have listed this in addition to your suggested paragraph.

Thanks!  This looks good to me.  I've reviewed all the bug traffic and
feel like this addresses the concerns, but not exactly in the way that
Sean, Bill, and Sam preferred.  I'd therefore like to see if this works
for them or if they still feel like it's too strong before I second it.
But if I were the only one deciding, I'd second this wording.  (Probably
not that surprising since you took all of my wording suggestions.)

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Sean Whitton  writes:

> I think what's a bit peculiar here is using "must" for a case where
> there might be package-specific exceptions.  In other cases, Policy uses
> "should" for these cases.  Typically "must" rules are simple and
> completely determinate.

I prefer that too, but in this case, it feels like must is appropriate for
at least systemd configuration files.  And also, just intuitively, I feel
like must is correct when people are using diversions rather than a native
mechanism.  Diversions add weird edge cases and we really shouldn't be
using them lightly.

The wording I proposed and that Luca has now adopted therefore uses must
with a caveat.  Does that sound okay to you?

I think must is too strong for alternatives, where I think there are more
likely to be package-specific exceptions.  But I'd like to take a harder
line on diversions, and I do think that matches how we use diversions
between two Debian packages (work around some weird situation where we
have no better option).

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Bill Allombert  writes:
> On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:

>> If you prefer, I can reword the general rule to be stricter, ie:
>> "packages must not use diversions where native mechanisms are
>> available" or so. Would this be better?

> "native mechanisms" seems to vague.

I suggested "must not be used when a suitable override mechanism that
accomplishes the same goal is already available."  Does that sound okay?
It's a bit weaker and opens the door to arguments about whether the native
override mechanism accomplishes the same goal, but I think that's a
feature here rather than a bug.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 06 Jun 2023 16:46:04 -0700 Russ Allbery  wrote:
> Luca Boccassi  writes:
> > I'm not suggesting that you stop using emails to send your changes
- I'm
> > simply asking to reconsider making policy work like the vast
majority of
> > other parts of Debian, and _also_ accepts merge requests on Salsa,
in
> > _addition_ to emails.
> 
> I have considered it several times, and I considered it again as part
of
> this recent conversation before writing my reply.

Sure, it's your package so it's up to you.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Luca Boccassi  writes:

> It depends on your pain threshold.

Salsa is worse for me for working on Policy in pretty much every respect
than email with patches.  That's not a statement about my pain threshold.
It's a fundamental disagreement with you about which tools work better for
this specific type of maintenance.

> For me it's much easier and simpler and less painful to do all of that
> on Salsa, as opposed to wade through a bunch of random emails and
> scrolling up and down through the BTS.

I understand that you don't like email.  The Policy process is optimized
for the Policy editors, rather than for the preferences of people who are
working on a specific change that they care about.  I appreciate that this
can be annoying, but it's fairly typical for open source work.  See, for
example, the Linux kernel, which likewise doesn't take merge requests, no
matter how inconvenient that may be for contributors.

> I'm sure there were some suggestions earlier in the thread, that I now
> have forgotten about, and they will be lost because there's no way I'm
> going back wading through dozens of random replies to figure out what is
> applied and what is not.

That's fine; this is part of the job of the Policy editors.

> I'm not suggesting that you stop using emails to send your changes - I'm
> simply asking to reconsider making policy work like the vast majority of
> other parts of Debian, and _also_ accepts merge requests on Salsa, in
> _addition_ to emails.

I have considered it several times, and I considered it again as part of
this recent conversation before writing my reply.

> Then the submitter can choose. I suspect you are going to get a lot more
> contributions this way,

Maximizing contribution of merge requests doesn't help the bottleneck for
Policy changes.  Policy changes are often complex and require thought and
research.  Most of the patches we get are incorrect or incomplete to start
with.  The bottleneck is time from people with a lot of in-depth Debian
knowledge to think deeply about a problem.

If the people who routinely provide detailed and incredibly useful
analyses of Policy problems say that they would rather use Salsa to do so,
that's important feedback and I would like to hear that feedback.  But the
goal here is not to maximize development speed.  It's to think hard about
a problem and get the answer correct.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Wed, 7 Jun 2023 00:02:06 +0200 Bill Allombert 
wrote:
> On Tue, Jun 06, 2023 at 03:16:02PM +0100, Luca Boccassi wrote:
> > On Tue, 6 Jun 2023 15:23:35 +0200 Bill Allombert
,
> > Luca Boccassi  wrote:
> > > On Tue, Jun 06, 2023 at 01:38:51PM +0100, Luca Boccassi wrote:
> > > > > The diversion system is made precisely to work around other
> > packages
> > > > behavior,
> > > > > this is a feature not a bug. That it should only be used as
last
> > > > resort, I
> > > > > think everyone agree. But when it is, it should not be a RC
bug.
> > > > 
> > > > This is a technical matter, I'm not sure what 'consensus' means
in
> > this
> > > > context? Things _will not work_ as expected by shoe-horning
dpkg's
> > > > overrides onto systemd mechanisms, they _will_ break in weird
and
> > > > unexpected ways, and we as maintainers _will not support it_ -
> > whether
> > > > somebody else agrees or disagrees with this won't change any of
it.
> > > 
> > > Consensus is the way the Debian Policy update process works.
> > > But you do not need changes in Policy to report bugs about
package
> > that breaks
> > > others, there is the "grave" severity already.
> > 
> > That does not help, given currently policy allows it, without
changes
> > they could just say "policy allows me, so go fix it yourself". What
> > then?
> 
> That simply not how Policy works.
> Policy is for promoting interoperability and documenting current
practices.
> "Policy is not a stick to beat people with" as Manoj used to say.
> 
> If you are suggesting a policy change so that you can report RC bugs
on other
> packages, you are on the wrong track.

No, I am suggesting a policy change so that we do not end up in a messy
and unmaintainable situation, which I thought was one of the goals. The
current practice is that packages are not using diversions and
alternatives to take over systemd files. As I have already specified a
number of times, the number of packages that need changes following
this change is zero - because I have _already_ done the required work
to make it so. After having done this work for Bookwrom, I am
translating this current practice into policy, to ensure we don't
regress.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Wed, 7 Jun 2023 00:01:42 +0200 Bill Allombert 
wrote:
> On Tue, Jun 06, 2023 at 09:36:31PM +0100, Luca Boccassi wrote:
> > On Tue, 6 Jun 2023 20:51:46 +0200 Dominik George
> >  wrote:
> > > > Ok, how about: "the whole project, minus
> > naturesha...@debian.org who
> > > > appears to be unfamiliar with the concept of hyperboles, is
moving
> > > > toward git and Salsa". Better?
> > > 
> > > No.
> > > 
> > > Your "hyperbole" very much read as "Come on, minority who cares
about
> > > the mail workflow, you're weird anachronists, get onto the Salsa
> > train already!"
> > > 
> > > So that's what I am criticizing.
> > 
> > Sure, and you just coincidentally happened to leave out the part
that
> > says the exact opposite:
> 
> This is beside the point. Your problematic statement was
> "The whole project is moving toward git and Salsa ".
> This is not conducive of productive interaction.

It was not problematic at all, unless you are deliberately trying to
see problems where there are none. It was an hyperbole: a huge number
of packages and teams are using effectively Salsa workflows for
collaboration today. Are you denying this?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 03:16:02PM +0100, Luca Boccassi wrote:
> On Tue, 6 Jun 2023 15:23:35 +0200 Bill Allombert ,
> Luca Boccassi  wrote:
> > On Tue, Jun 06, 2023 at 01:38:51PM +0100, Luca Boccassi wrote:
> > > > The diversion system is made precisely to work around other
> packages
> > > behavior,
> > > > this is a feature not a bug. That it should only be used as last
> > > resort, I
> > > > think everyone agree. But when it is, it should not be a RC bug.
> > > 
> > > This is a technical matter, I'm not sure what 'consensus' means in
> this
> > > context? Things _will not work_ as expected by shoe-horning dpkg's
> > > overrides onto systemd mechanisms, they _will_ break in weird and
> > > unexpected ways, and we as maintainers _will not support it_ -
> whether
> > > somebody else agrees or disagrees with this won't change any of it.
> > 
> > Consensus is the way the Debian Policy update process works.
> > But you do not need changes in Policy to report bugs about package
> that breaks
> > others, there is the "grave" severity already.
> 
> That does not help, given currently policy allows it, without changes
> they could just say "policy allows me, so go fix it yourself". What
> then?

That simply not how Policy works.
Policy is for promoting interoperability and documenting current practices.
"Policy is not a stick to beat people with" as Manoj used to say.

If you are suggesting a policy change so that you can report RC bugs on other
packages, you are on the wrong track.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 09:36:31PM +0100, Luca Boccassi wrote:
> On Tue, 6 Jun 2023 20:51:46 +0200 Dominik George
>  wrote:
> > > Ok, how about: "the whole project, minus
> naturesha...@debian.org who
> > > appears to be unfamiliar with the concept of hyperboles, is moving
> > > toward git and Salsa". Better?
> > 
> > No.
> > 
> > Your "hyperbole" very much read as "Come on, minority who cares about
> > the mail workflow, you're weird anachronists, get onto the Salsa
> train already!"
> > 
> > So that's what I am criticizing.
> 
> Sure, and you just coincidentally happened to leave out the part that
> says the exact opposite:

This is beside the point. Your problematic statement was
"The whole project is moving toward git and Salsa ".
This is not conducive of productive interaction.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 at 17:12, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > --- a/policy/ap-pkg-alternatives.rst
> > +++ b/policy/ap-pkg-alternatives.rst
> > @@ -24,3 +24,7 @@ See the :manpage:`update-alternatives(8)` man page for 
> > details.
> >  If ``update-alternatives`` does not seem appropriate you may wish to
> >  consider using diversions instead.
> >
> > +Do not attempt to use alternatives for files belonging or used by 
> > components
> > +that support native overriding mechanisms, such as ``systemd`` unit files. 
> > Read
> > +:doc:`ch-binary` for more information.
>
> "Do not attempt" in US English reads a little weirdly (I know you're
> copying an existing UK English phrasing). I instead suggest:
>
> Do not use alternatives for ``systemd`` configuration files. See
> :doc:`ch-binary` for more information.
>
> (See below for why this is systemd-specific.) This doesn't use our normal
> normative language conventions, which I think is correct since this is a
> non-normative appendix.
>
> > diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
> > index fe360d1..09367d7 100644
> > --- a/policy/ap-pkg-diversions.rst
> > +++ b/policy/ap-pkg-diversions.rst
> > @@ -81,3 +81,7 @@ when the file does not exist.
> >  Do not attempt to divert a conffile, as ``dpkg`` does not handle it
> >  well.
> >
> > +Do not attempt to divert files belonging or used by components that support
> > +native overriding mechanisms, such as ``systemd`` unit files. Read
> > +:doc:`ch-binary` for more information.
>
> Similarly here:
>
> Do not use diversions for files that have their own native override
> mechanisms, such as ``systemd`` unit files. See :doc:`ch-binary` for
> more information.

Sounds all good to me, applied, I simply had copied the previous
paragraph's style.

> > diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> > index e517f26..e36d028 100644
> > --- a/policy/ch-binary.rst
> > +++ b/policy/ch-binary.rst
> > @@ -371,6 +371,37 @@ against earlier versions of something that previously 
> > did not use
> >  ``update-alternatives``; this is an exception to the usual rule that
> >  versioned conflicts should be avoided.)
> >
> > +Diversions and alternatives should be used primarily as a tool for local
> > +administrators and local packages to override the behaviour of Debian. Its 
> > use
> > +between Debian packages should be rare, should involve coordination 
> > between the
> > +packages and their maintainers, and must only be used to solve problems 
> > that
> > +cannot be handled through other facilities or native mechanisms.
>
> I think this not correct for alternatives. They are intended for use
> between Debian packages, so we need to distinguish between alternatives
> and diversions here. This might be clearer if we created a subsection for
> alternatives and diversions and added a bit of additional context from the
> appendices (and ideally removed the appendices). That's not something to
> do on this bug, just noting mostly for Sean and future work.

While I disagree on that, because alternatives make a mess of
image-based systems and are a pain to handle (requires moving parts
via maint scripts to be effective), it is also a bit off topic so I
have applied your suggestion as-is.

> I think there's also a bit more justification than we need. The
> justification is useful for discussing the Policy bug, but once we've
> decided on an approach, I think we can just point at the correct
> alternative mechanism.
>
> How about something like this:
>
> Diversions are primarily intended as a tool for local administrators
> or local packages to override the behavior of Debian. While there are
> some circumstances where one Debian package may need to divert a file
> of another Debian package, those circumstances are rare and diversions
> should only be used as a last resort when no other suitable mechanism
> exists. Diversion of a file in one Debian package by another Debian
> package should be coordinated between the maintainers of those
> packages.
>
> Diversions must not be used when a suitable override mechanism that
> accomplishes the same goal is already available.
>
> One specific case of that rule is that configuration files used by
> ``systemd`` components, such as `units,
> 
> `_
> `udev rules,
> 
> `_
> `tmpfiles.d,
> 
> `_
> `modules-load.d,
> 
> `_,
> `sysusers
> 
> `_
> and other such files, 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 06 Jun 2023 09:19:30 -0700 Russ Allbery  wrote:
> Luca Boccassi  writes:
> 
> > Snarks aside, allowing merge requests to be open on Salsa in
_addition_
> > to attachments to the BTS, as the vast majority of other packages
> > already do, doesn't take away anything from you, but would add
quite a
> > lot for the rest of us, who find ourselves very limited and very
much
> > barrier-ized by clunky, old and painful email-based processes.
> 
> Merge requests are useful when the goal is to be able to merge the
merge
> request as-is at the end of some code review process. My guess is
that
> this happens about 20% of the time with Policy. The other 80% of the
time
> I tweak wording or phrasing or formatting when applying the change,
and
> also add an upgrading-checklist entry and other bookkeeping that I
don't
> want to have to explain to everyone proposing new wording.
> 
> Policy is not a program and we're not taking source code changes. The
goal
> of the discussion is to converge on the semantics of what we are
going to
> say, which often requires multi-paragraph discussions where the
normal
> tools of email such as quoting are more useful. I find trying to have
an
> email-style discussion in Salsa to be annoying and tedious. It also
> fragments the record, whereas having it in the bug means I can, at
any
> time, load the entire bug into Gnus and re-read the discussion of how
we
> arrived at the decision in the same tool that I use for reading all
other
> Debian discussions.

It depends on your pain threshold. For me it's much easier and simpler
and less painful to do all of that on Salsa, as opposed to wade through
a bunch of random emails and scrolling up and down through the BTS. I'm
sure there were some suggestions earlier in the thread, that I now have
forgotten about, and they will be lost because there's no way I'm going
back wading through dozens of random replies to figure out what is
applied and what is not. On Salsa, this would be an 'unresolved' review
comment that I can click 'solved' once handled.

I'm not suggesting that you stop using emails to send your changes -
I'm simply asking to reconsider making policy work like the vast
majority of other parts of Debian, and _also_ accepts merge requests on
Salsa, in _addition_ to emails. Then the submitter can choose. I
suspect you are going to get a lot more contributions this way, that is
certainly the experience from other teams that have enabled Salsa from
what I can hear (eg: kernel team).

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 20:51:46 +0200 Dominik George
 wrote:
> > Ok, how about: "the whole project, minus
naturesha...@debian.org who
> > appears to be unfamiliar with the concept of hyperboles, is moving
> > toward git and Salsa". Better?
> 
> No.
> 
> Your "hyperbole" very much read as "Come on, minority who cares about
> the mail workflow, you're weird anachronists, get onto the Salsa
train already!"
> 
> So that's what I am criticizing.

Sure, and you just coincidentally happened to leave out the part that
says the exact opposite:

> Snarks aside, allowing merge requests to be open on Salsa in
> _addition_ to attachments to the BTS, as the vast majority of other
> packages already do, doesn't take away anything from you, but would
> add quite a lot for the rest of us, who find ourselves very limited
> and very much barrier-ized by clunky, old and painful email-based
> processes.

Which means you are actually doing what you accuse me of - you are not
content with being able to keep using mail workflows for yourself, you
want to actively stop everybody from being able to use Salsa too.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Dominik George]

> Ok, how about: "the whole project, minus naturesha...@debian.org who
> appears to be unfamiliar with the concept of hyperboles, is moving
> toward git and Salsa". Better?

No.

Your "hyperbole" very much read as "Come on, minority who cares about
the mail workflow, you're weird anachronists, get onto the Salsa train already!"

So that's what I am criticizing.

-nik


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Luca Boccassi  writes:

> Snarks aside, allowing merge requests to be open on Salsa in _addition_
> to attachments to the BTS, as the vast majority of other packages
> already do, doesn't take away anything from you, but would add quite a
> lot for the rest of us, who find ourselves very limited and very much
> barrier-ized by clunky, old and painful email-based processes.

Merge requests are useful when the goal is to be able to merge the merge
request as-is at the end of some code review process. My guess is that
this happens about 20% of the time with Policy. The other 80% of the time
I tweak wording or phrasing or formatting when applying the change, and
also add an upgrading-checklist entry and other bookkeeping that I don't
want to have to explain to everyone proposing new wording.

Policy is not a program and we're not taking source code changes. The goal
of the discussion is to converge on the semantics of what we are going to
say, which often requires multi-paragraph discussions where the normal
tools of email such as quoting are more useful. I find trying to have an
email-style discussion in Salsa to be annoying and tedious. It also
fragments the record, whereas having it in the bug means I can, at any
time, load the entire bug into Gnus and re-read the discussion of how we
arrived at the decision in the same tool that I use for reading all other
Debian discussions.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Luca Boccassi  writes:

> --- a/policy/ap-pkg-alternatives.rst
> +++ b/policy/ap-pkg-alternatives.rst
> @@ -24,3 +24,7 @@ See the :manpage:`update-alternatives(8)` man page for 
> details.
>  If ``update-alternatives`` does not seem appropriate you may wish to
>  consider using diversions instead.
>  
> +Do not attempt to use alternatives for files belonging or used by components
> +that support native overriding mechanisms, such as ``systemd`` unit files. 
> Read
> +:doc:`ch-binary` for more information.

"Do not attempt" in US English reads a little weirdly (I know you're
copying an existing UK English phrasing). I instead suggest:

Do not use alternatives for ``systemd`` configuration files. See
:doc:`ch-binary` for more information.

(See below for why this is systemd-specific.) This doesn't use our normal
normative language conventions, which I think is correct since this is a
non-normative appendix.

> diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
> index fe360d1..09367d7 100644
> --- a/policy/ap-pkg-diversions.rst
> +++ b/policy/ap-pkg-diversions.rst
> @@ -81,3 +81,7 @@ when the file does not exist.
>  Do not attempt to divert a conffile, as ``dpkg`` does not handle it
>  well.
>  
> +Do not attempt to divert files belonging or used by components that support
> +native overriding mechanisms, such as ``systemd`` unit files. Read
> +:doc:`ch-binary` for more information.

Similarly here:

Do not use diversions for files that have their own native override
mechanisms, such as ``systemd`` unit files. See :doc:`ch-binary` for
more information.

> diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> index e517f26..e36d028 100644
> --- a/policy/ch-binary.rst
> +++ b/policy/ch-binary.rst
> @@ -371,6 +371,37 @@ against earlier versions of something that previously 
> did not use
>  ``update-alternatives``; this is an exception to the usual rule that
>  versioned conflicts should be avoided.)
>  
> +Diversions and alternatives should be used primarily as a tool for local
> +administrators and local packages to override the behaviour of Debian. Its 
> use
> +between Debian packages should be rare, should involve coordination between 
> the
> +packages and their maintainers, and must only be used to solve problems that
> +cannot be handled through other facilities or native mechanisms.

I think this not correct for alternatives. They are intended for use
between Debian packages, so we need to distinguish between alternatives
and diversions here. This might be clearer if we created a subsection for
alternatives and diversions and added a bit of additional context from the
appendices (and ideally removed the appendices). That's not something to
do on this bug, just noting mostly for Sean and future work.

I think there's also a bit more justification than we need. The
justification is useful for discussing the Policy bug, but once we've
decided on an approach, I think we can just point at the correct
alternative mechanism.

How about something like this:

Diversions are primarily intended as a tool for local administrators
or local packages to override the behavior of Debian. While there are
some circumstances where one Debian package may need to divert a file
of another Debian package, those circumstances are rare and diversions
should only be used as a last resort when no other suitable mechanism
exists. Diversion of a file in one Debian package by another Debian
package should be coordinated between the maintainers of those
packages.

Diversions must not be used when a suitable override mechanism that
accomplishes the same goal is already available.

One specific case of that rule is that configuration files used by
``systemd`` components, such as `units,

`_
`udev rules,
`_
`tmpfiles.d,

`_
`modules-load.d,

`_,
`sysusers

`_
and other such files, including those specific to systemd daemons
(e.g.:  `/etc/systemd/system.conf).

`_
must not be diverted by any Debian package. Instead, use masking and
drop-ins.

"masking" and "drop-ins" here should ideally be links to the relevant
documentation.

We then have to figure out what to say about alternatives. I think the
above rule is too strong for alternatives; it's often convenient to use
slave alternatives even in places where there may be a native override
mechanism, 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 15:23:35 +0200 Bill Allombert ,
Luca Boccassi  wrote:
> On Tue, Jun 06, 2023 at 01:38:51PM +0100, Luca Boccassi wrote:
> > > The diversion system is made precisely to work around other
packages
> > behavior,
> > > this is a feature not a bug. That it should only be used as last
> > resort, I
> > > think everyone agree. But when it is, it should not be a RC bug.
> > 
> > This is a technical matter, I'm not sure what 'consensus' means in
this
> > context? Things _will not work_ as expected by shoe-horning dpkg's
> > overrides onto systemd mechanisms, they _will_ break in weird and
> > unexpected ways, and we as maintainers _will not support it_ -
whether
> > somebody else agrees or disagrees with this won't change any of it.
> 
> Consensus is the way the Debian Policy update process works.
> But you do not need changes in Policy to report bugs about package
that breaks
> others, there is the "grave" severity already.

That does not help, given currently policy allows it, without changes
they could just say "policy allows me, so go fix it yourself". What
then?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 01:38:51PM +0100, Luca Boccassi wrote:
> > The diversion system is made precisely to work around other packages
> behavior,
> > this is a feature not a bug. That it should only be used as last
> resort, I
> > think everyone agree. But when it is, it should not be a RC bug.
> 
> This is a technical matter, I'm not sure what 'consensus' means in this
> context? Things _will not work_ as expected by shoe-horning dpkg's
> overrides onto systemd mechanisms, they _will_ break in weird and
> unexpected ways, and we as maintainers _will not support it_ - whether
> somebody else agrees or disagrees with this won't change any of it.

Consensus is the way the Debian Policy update process works.
But you do not need changes in Policy to report bugs about package that breaks
others, there is the "grave" severity already.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 14:03:41 +0200 Bill Allombert 
wrote:
> On Tue, Jun 06, 2023 at 12:16:39PM +0100, Luca Boccassi wrote:
> > > > local administrators and local packages to override the
behaviour of
> > > > Debian. Its use between Debian packages should be rare, should
involve
> > > > coordination between the packages and their maintainers, and
must only
> > > > be used to solve problems that cannot be handled through other
> > > > facilities or native mechanisms.  In other words, packages in
Debian
> > > > must not divert a file from another package unless this is
arranged
> > > > cooperatively between the packages to solve some specific and
unusual
> > > > problem.
> > >
> > > How about:
> > >
> > > Diversions and alternatives are primarily tools for local
> > > administrators and local packages to override Debian's
default
> > > behaviour.  Maintainers should prefer to use other, package-
specific
> > > facilities for overriding configuration, such as systemd's
unit file
> > > overrides, wherever possible.
> > >
> > > If diversions and alternatives are to be used, maintainers
should
> > > co-ordinate with the maintainers of all the packages
involved.  For
> > > example, configuration files used by systemd components
should not
> > > be diverted with dpkg-divert or the alternatives system
without
> > > agreement between not only the maintainers of the packages
that ship
> > > the files, but also the maintainers of the relevant systemd
> > > components.
> > 
> > As above, as long as this wording makes any offending package
> > rc-buggy, it's fine by me, but my understanding is that using
'should'
> > doesn't guarantee that. Please correct me if I'm wrong here.
> 
> I do not think there is consensus that this should be a RC bug
outside of
> a case by case basis. It is already awkward to mention systemd
explicitly
> in this paragraph.
> 
> The diversion system is made precisely to work around other packages
behavior,
> this is a feature not a bug. That it should only be used as last
resort, I
> think everyone agree. But when it is, it should not be a RC bug.

This is a technical matter, I'm not sure what 'consensus' means in this
context? Things _will not work_ as expected by shoe-horning dpkg's
overrides onto systemd mechanisms, they _will_ break in weird and
unexpected ways, and we as maintainers _will not support it_ - whether
somebody else agrees or disagrees with this won't change any of it.

It means that, when such bug happens, and it turns out there's a
divert/override in the way, the bug will be an immediate close+wontfix
and the reporter will be told they are on their own. And that's a best
case scenario where it is all apparent, while actually it's more likely
this will be hidden away and require a large waste of time to figure
out. Isn't it better to ensure this cannot happen in the first place?
Once more: there are no such cases as of bookworm in the archive, so
the impact today is nil. Wouldn't it be prudent to ensure it stays that
way in the strictest possible terms?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Tue, Jun 06, 2023 at 12:16:39PM +0100, Luca Boccassi wrote:
> > > local administrators and local packages to override the behaviour of
> > > Debian. Its use between Debian packages should be rare, should involve
> > > coordination between the packages and their maintainers, and must only
> > > be used to solve problems that cannot be handled through other
> > > facilities or native mechanisms.  In other words, packages in Debian
> > > must not divert a file from another package unless this is arranged
> > > cooperatively between the packages to solve some specific and unusual
> > > problem.
> >
> > How about:
> >
> > Diversions and alternatives are primarily tools for local
> > administrators and local packages to override Debian's default
> > behaviour.  Maintainers should prefer to use other, package-specific
> > facilities for overriding configuration, such as systemd's unit file
> > overrides, wherever possible.
> >
> > If diversions and alternatives are to be used, maintainers should
> > co-ordinate with the maintainers of all the packages involved.  For
> > example, configuration files used by systemd components should not
> > be diverted with dpkg-divert or the alternatives system without
> > agreement between not only the maintainers of the packages that ship
> > the files, but also the maintainers of the relevant systemd
> > components.
> 
> As above, as long as this wording makes any offending package
> rc-buggy, it's fine by me, but my understanding is that using 'should'
> doesn't guarantee that. Please correct me if I'm wrong here.

I do not think there is consensus that this should be a RC bug outside of
a case by case basis. It is already awkward to mention systemd explicitly
in this paragraph.

The diversion system is made precisely to work around other packages behavior,
this is a feature not a bug. That it should only be used as last resort, I
think everyone agree. But when it is, it should not be a RC bug.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 at 12:25, Dominik George  wrote:
>
> > The whole project is moving toward git and Salsa
>
> Sorry for the noise, but as you are clearly misattributing this to me (I am 
> part of the project, so "the whole project" includes me):
>
> I am not, and do not want to, move bugs and patches to Git and Salsa. I 
> consider it a huge advantage of Debian that I can contribute limitless with 
> something as barrierfree as an e-mail.
>
> If you voice your opinion, please do not impose it on me. Thanks!

Ok, how about: "the whole project, minus naturesha...@debian.org who
appears to be unfamiliar with the concept of hyperboles, is moving
toward git and Salsa". Better?

Snarks aside, allowing merge requests to be open on Salsa in
_addition_ to attachments to the BTS, as the vast majority of other
packages already do, doesn't take away anything from you, but would
add quite a lot for the rest of us, who find ourselves very limited
and very much barrier-ized by clunky, old and painful email-based
processes.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Dominik George]

> The whole project is moving toward git and Salsa

Sorry for the noise, but as you are clearly misattributing this to me (I am 
part of the project, so "the whole project" includes me):

I am not, and do not want to, move bugs and patches to Git and Salsa. I 
consider it a huge advantage of Debian that I can contribute limitless with 
something as barrierfree as an e-mail.

If you voice your opinion, please do not impose it on me. Thanks!

-nik

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 at 11:58, Sean Whitton  wrote:
>
> Hello,
>
> On Mon 05 Jun 2023 at 12:59AM +01, Luca Boccassi wrote:
>
> > On Sun, 4 Jun 2023 19:39:49 +0200 Bill Allombert 
> > wrote:
> >> On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
> >> > If you prefer, I can reword the general rule to be stricter, ie:
> >> > "packages must not use diversions where native mechanisms are
> >> > available" or so. Would this be better?
> >>
> >> "native mechanisms" seems to vague.
> >
> > Well, I originally had written precisely what I meant, but was told on
> > this thread to "add normative language for only the general case"
> > instead, so I've done that. That's always going to sound vague. I
> > cannot do both at the same time.
> >
> > As you can see from the latest revision, right now the rule is general
> > but it's immediately followed by a very concrete and documented
> > example. Open to precise suggestions on how to improve it.
>
> I think what's a bit peculiar here is using "must" for a case where
> there might be package-specific exceptions.  In other cases, Policy uses
> "should" for these cases.  Typically "must" rules are simple and
> completely determinate.
>
> Maintainers do take our "should"s seriously -- let's use that here.
> It can always be strengthened later.

My understanding is that "should" is effectively optional, ie: it is
not enough to make a package rc-buggy, and while they are generally
followed, there is no actual hard requirement to do so. That is not
enough for me and for this use case. So again my goal here is to make
an hypothetical future dpkg-divert or alternative of a unit file
(wherever it is shipped from) instantly and unambiguously rc-buggy,
with no leeway (again there are no longer any such cases in the
archive as of bookworm, so nothing is affected). If you can suggest an
alternative appropriate wording that can achieve the same effect, that
would be fine to me - I don't mind how it's written, as long as it
meets this requirement.

> > Diversions and alternatives should be used primarily as a tool for
> > local administrators and local packages to override the behaviour of
> > Debian. Its use between Debian packages should be rare, should involve
> > coordination between the packages and their maintainers, and must only
> > be used to solve problems that cannot be handled through other
> > facilities or native mechanisms.  In other words, packages in Debian
> > must not divert a file from another package unless this is arranged
> > cooperatively between the packages to solve some specific and unusual
> > problem.
>
> How about:
>
> Diversions and alternatives are primarily tools for local
> administrators and local packages to override Debian's default
> behaviour.  Maintainers should prefer to use other, package-specific
> facilities for overriding configuration, such as systemd's unit file
> overrides, wherever possible.
>
> If diversions and alternatives are to be used, maintainers should
> co-ordinate with the maintainers of all the packages involved.  For
> example, configuration files used by systemd components should not
> be diverted with dpkg-divert or the alternatives system without
> agreement between not only the maintainers of the packages that ship
> the files, but also the maintainers of the relevant systemd
> components.

As above, as long as this wording makes any offending package
rc-buggy, it's fine by me, but my understanding is that using 'should'
doesn't guarantee that. Please correct me if I'm wrong here.

> I would also like to suggest "their own mechanisms for overriding parts
> of configuration" over "native overriding mechanisms" in the appendices.

Sure that sounds good.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Tue 06 Jun 2023 at 11:51AM +01, Luca Boccassi wrote:

> Well, the README says:
>
> "Please submit a bug to the BTS, either with patches attached, or a
> reference to a git branch that is publically fetchable."
>
> The whole project is moving toward git and Salsa,

Git, certainly, but not all of us love Salsa.

> and it is very annoying to have to do drive-by contributions via
> email, it really sucks as a process for contributors, so please
> consider re-evaluating this in the future. Patch attached.

You are very fast!
I just posted another message with some wording review.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Mon 05 Jun 2023 at 12:59AM +01, Luca Boccassi wrote:

> On Sun, 4 Jun 2023 19:39:49 +0200 Bill Allombert 
> wrote:
>> On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
>> > If you prefer, I can reword the general rule to be stricter, ie:
>> > "packages must not use diversions where native mechanisms are
>> > available" or so. Would this be better?
>>
>> "native mechanisms" seems to vague.
>
> Well, I originally had written precisely what I meant, but was told on
> this thread to "add normative language for only the general case"
> instead, so I've done that. That's always going to sound vague. I
> cannot do both at the same time.
>
> As you can see from the latest revision, right now the rule is general
> but it's immediately followed by a very concrete and documented
> example. Open to precise suggestions on how to improve it.

I think what's a bit peculiar here is using "must" for a case where
there might be package-specific exceptions.  In other cases, Policy uses
"should" for these cases.  Typically "must" rules are simple and
completely determinate.

Maintainers do take our "should"s seriously -- let's use that here.
It can always be strengthened later.

> Diversions and alternatives should be used primarily as a tool for
> local administrators and local packages to override the behaviour of
> Debian. Its use between Debian packages should be rare, should involve
> coordination between the packages and their maintainers, and must only
> be used to solve problems that cannot be handled through other
> facilities or native mechanisms.  In other words, packages in Debian
> must not divert a file from another package unless this is arranged
> cooperatively between the packages to solve some specific and unusual
> problem.

How about:

Diversions and alternatives are primarily tools for local
administrators and local packages to override Debian's default
behaviour.  Maintainers should prefer to use other, package-specific
facilities for overriding configuration, such as systemd's unit file
overrides, wherever possible.

If diversions and alternatives are to be used, maintainers should
co-ordinate with the maintainers of all the packages involved.  For
example, configuration files used by systemd components should not
be diverted with dpkg-divert or the alternatives system without
agreement between not only the maintainers of the packages that ship
the files, but also the maintainers of the relevant systemd
components.

I would also like to suggest "their own mechanisms for overriding parts
of configuration" over "native overriding mechanisms" in the appendices.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 6 Jun 2023 at 11:45, Sean Whitton  wrote:
>
> Hello,
>
> On Sun 04 Jun 2023 at 01:35PM +01, Luca Boccassi wrote:
>
> > In the interest of speeding things up a bit, I've done some rewording
> > as suggested - moved to the exiting chapter, and use the systemd files
> > only as an example:
> >
> > https://salsa.debian.org/bluca/policy/-/commit/5058bd2f8c742c3d8695e2c98ee3a597d431ffd7
> >
> > Off-topic - any reasons MRs are disabled on the policy repo? It would
> > be much nicer and quicker to use the Gitlab review process I think,
> > like we do for other packages.
>
> It's actually on-topic -- can you post your proposed patch to this bug
> for inline review, please?  This is documented in README.md.  The main
> reason we have MRs disabled is that we want a complete record of the
> discussion that led up to a Policy change to be recorded in the BTS.
>
> A secondary reason is that I strongly disprefer doing patch review using
> an interface other than my mail client.

Well, the README says:

"Please submit a bug to the BTS, either with patches attached, or a
reference to a git branch that is publically fetchable."

The whole project is moving toward git and Salsa, and it is very
annoying to have to do drive-by contributions via email, it really
sucks as a process for contributors, so please consider re-evaluating
this in the future. Patch attached.

Kind regards,
Luca Boccassi
From 5058bd2f8c742c3d8695e2c98ee3a597d431ffd7 Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Mon, 8 May 2023 03:21:14 +0100
Subject: [PATCH] Forbid using dpkg-divert/alternatives when there are native
 mechanisms

The supported mechanism for augmenting, changing, overriding and
disabling systemd configuration files is natively supported and fully
integrated in Debian, via drop-ins, hierarchical overrides, and
masking. dpkg-divert is not integrated in systemd tools so its use
is completely hidden in logs and status interfaces, and it is specific
to Debian and thus diverges from what users expect as implemented by
all other distros, going against one of the core goals of the systemd
project which is to provide a uniform interface regardless of distro
vendor or flavour.

Clarify that diversions and alternatives should only be used when
needed, with cooperation, and must not be used when there are native
mechanisms to obtain the same goals, and use systemd files as an
example.
---
 policy/ap-pkg-alternatives.rst |  4 
 policy/ap-pkg-diversions.rst   |  4 
 policy/ch-binary.rst   | 31 +++
 3 files changed, 39 insertions(+)

diff --git a/policy/ap-pkg-alternatives.rst b/policy/ap-pkg-alternatives.rst
index ffa2163..1cfd3e6 100644
--- a/policy/ap-pkg-alternatives.rst
+++ b/policy/ap-pkg-alternatives.rst
@@ -24,3 +24,7 @@ See the :manpage:`update-alternatives(8)` man page for details.
 If ``update-alternatives`` does not seem appropriate you may wish to
 consider using diversions instead.
 
+Do not attempt to use alternatives for files belonging or used by components
+that support native overriding mechanisms, such as ``systemd`` unit files. Read
+:doc:`ch-binary` for more information.
+
diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
index fe360d1..09367d7 100644
--- a/policy/ap-pkg-diversions.rst
+++ b/policy/ap-pkg-diversions.rst
@@ -81,3 +81,7 @@ when the file does not exist.
 Do not attempt to divert a conffile, as ``dpkg`` does not handle it
 well.
 
+Do not attempt to divert files belonging or used by components that support
+native overriding mechanisms, such as ``systemd`` unit files. Read
+:doc:`ch-binary` for more information.
+
diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
index e517f26..e36d028 100644
--- a/policy/ch-binary.rst
+++ b/policy/ch-binary.rst
@@ -371,6 +371,37 @@ against earlier versions of something that previously did not use
 ``update-alternatives``; this is an exception to the usual rule that
 versioned conflicts should be avoided.)
 
+Diversions and alternatives should be used primarily as a tool for local
+administrators and local packages to override the behaviour of Debian. Its use
+between Debian packages should be rare, should involve coordination between the
+packages and their maintainers, and must only be used to solve problems that
+cannot be handled through other facilities or native mechanisms.
+In other words, packages in Debian must not divert a file from another package
+unless this is arranged cooperatively between the packages to solve some
+specific and unusual problem.
+
+For example, configuration files that are used by ``systemd`` components, such
+as `units,
+`_
+`udev rules,
+`_
+`tmpfiles.d,
+`_
+`modules-load.d,

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Sun 04 Jun 2023 at 01:35PM +01, Luca Boccassi wrote:

> In the interest of speeding things up a bit, I've done some rewording
> as suggested - moved to the exiting chapter, and use the systemd files
> only as an example:
>
> https://salsa.debian.org/bluca/policy/-/commit/5058bd2f8c742c3d8695e2c98ee3a597d431ffd7
>
> Off-topic - any reasons MRs are disabled on the policy repo? It would
> be much nicer and quicker to use the Gitlab review process I think,
> like we do for other packages.

It's actually on-topic -- can you post your proposed patch to this bug
for inline review, please?  This is documented in README.md.  The main
reason we have MRs disabled is that we want a complete record of the
discussion that led up to a Policy change to be recorded in the BTS.

A secondary reason is that I strongly disprefer doing patch review using
an interface other than my mail client.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Sun, 4 Jun 2023 19:39:49 +0200 Bill Allombert 
wrote:
> On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
> > If you prefer, I can reword the general rule to be stricter, ie:
> > "packages must not use diversions where native mechanisms are
> > available" or so. Would this be better?
> 
> "native mechanisms" seems to vague.

Well, I originally had written precisely what I meant, but was told on
this thread to "add normative language for only the general case"
instead, so I've done that. That's always going to sound vague. I
cannot do both at the same time.

As you can see from the latest revision, right now the rule is general
but it's immediately followed by a very concrete and documented
example. Open to precise suggestions on how to improve it.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Bill Allombert]

On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
> If you prefer, I can reword the general rule to be stricter, ie:
> "packages must not use diversions where native mechanisms are
> available" or so. Would this be better?

"native mechanisms" seems to vague.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Sun, 4 Jun 2023 at 12:25, Luca Boccassi  wrote:
>
> On Sun, 4 Jun 2023 at 11:54, Sean Whitton  wrote:
> >
> > Hello Luca,
> >
> > On Mon 08 May 2023 at 08:07PM +01, Luca Boccassi wrote:
> >
> > > The specific difference, for which I think an explicit call out is
> > > needed, is because these config files are shipped by some packages but
> > > are not used _by_ them, they are consumed by systemd (or udev, or
> > > kmod, etc). Specifically, if package A ships a.service, and package B
> > > overrides it, even if the maintainers of A and B agree, that's still
> > > not good enough for me, as they are really affecting systemd, which is
> > > the consumer and the provider of the interface they are using, and
> > > ultimately the first port of call for bug reports. This is especially
> > > true for udev.
> > >
> > > So in my latest revision of the patch, the general rule is as
> > > requested by Russ and as you mention it, but there is an explicit,
> > > stricter rule to cover this case, which is important to me. Policy
> > > calls out core component software in many places, such as dpkg, and
> > > systemd is already mentioned in other parts of the policy, so it did
> > > not seem too far-fetched to me.
> >
> > I'm afraid I'm not convinced.  I'd second a patch where systemd is used
> > as an example of the rule, as I suggested.
>
> The existing policy is too weak for this case, ie: it's a "should". It
> needs to be a "must" for these specific cases. Also the existing
> policy only covers diverting from other packages, not from 'self' -
> that needs to be forbidden too. There was one such example,
> iptables-persistent, and it has been fixed in Bookworm, so to be clear
> this is a zero-net-effect policy change, ie, no packages will suddenly
> become rc-buggy, as the two existing instances have already been
> fixed.
>
> If you prefer, I can reword the general rule to be stricter, ie:
> "packages must not use diversions where native mechanisms are
> available" or so. Would this be better?
>
> > Thank you for the additional commit regarding kmod.  It is good to have
> > been made aware of issue, but let's discuss it in a separate bug after
> > making this change -- the considerations might be quite different.
> >
> > On Tue 09 May 2023 at 12:31AM +01, Luca Boccassi wrote:
> >
> > > On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery  wrote:
> > >
> > >> Oh, thank you!  I had completely forgotten that we said something
> > >> about this under maintainer scripts.
> > >>
> > >> That doesn't entirely cover this case (because systemd and udev may
> > >> not be "that package" in this sense), but it covers much of the
> > >> general case.
> > >
> > > Would you like me to reword/move the new snippet?
> >
> > Yes, thank you.  I will review the new version.
>
> Any specific suggestions? IE, where it should be, etc.

In the interest of speeding things up a bit, I've done some rewording
as suggested - moved to the exiting chapter, and use the systemd files
only as an example:

https://salsa.debian.org/bluca/policy/-/commit/5058bd2f8c742c3d8695e2c98ee3a597d431ffd7

Off-topic - any reasons MRs are disabled on the policy repo? It would
be much nicer and quicker to use the Gitlab review process I think,
like we do for other packages.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Sun, 4 Jun 2023 at 11:54, Sean Whitton  wrote:
>
> Hello Luca,
>
> On Mon 08 May 2023 at 08:07PM +01, Luca Boccassi wrote:
>
> > The specific difference, for which I think an explicit call out is
> > needed, is because these config files are shipped by some packages but
> > are not used _by_ them, they are consumed by systemd (or udev, or
> > kmod, etc). Specifically, if package A ships a.service, and package B
> > overrides it, even if the maintainers of A and B agree, that's still
> > not good enough for me, as they are really affecting systemd, which is
> > the consumer and the provider of the interface they are using, and
> > ultimately the first port of call for bug reports. This is especially
> > true for udev.
> >
> > So in my latest revision of the patch, the general rule is as
> > requested by Russ and as you mention it, but there is an explicit,
> > stricter rule to cover this case, which is important to me. Policy
> > calls out core component software in many places, such as dpkg, and
> > systemd is already mentioned in other parts of the policy, so it did
> > not seem too far-fetched to me.
>
> I'm afraid I'm not convinced.  I'd second a patch where systemd is used
> as an example of the rule, as I suggested.

The existing policy is too weak for this case, ie: it's a "should". It
needs to be a "must" for these specific cases. Also the existing
policy only covers diverting from other packages, not from 'self' -
that needs to be forbidden too. There was one such example,
iptables-persistent, and it has been fixed in Bookworm, so to be clear
this is a zero-net-effect policy change, ie, no packages will suddenly
become rc-buggy, as the two existing instances have already been
fixed.

If you prefer, I can reword the general rule to be stricter, ie:
"packages must not use diversions where native mechanisms are
available" or so. Would this be better?

> Thank you for the additional commit regarding kmod.  It is good to have
> been made aware of issue, but let's discuss it in a separate bug after
> making this change -- the considerations might be quite different.
>
> On Tue 09 May 2023 at 12:31AM +01, Luca Boccassi wrote:
>
> > On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery  wrote:
> >
> >> Oh, thank you!  I had completely forgotten that we said something
> >> about this under maintainer scripts.
> >>
> >> That doesn't entirely cover this case (because systemd and udev may
> >> not be "that package" in this sense), but it covers much of the
> >> general case.
> >
> > Would you like me to reword/move the new snippet?
>
> Yes, thank you.  I will review the new version.

Any specific suggestions? IE, where it should be, etc.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello Luca,

On Mon 08 May 2023 at 08:07PM +01, Luca Boccassi wrote:

> The specific difference, for which I think an explicit call out is
> needed, is because these config files are shipped by some packages but
> are not used _by_ them, they are consumed by systemd (or udev, or
> kmod, etc). Specifically, if package A ships a.service, and package B
> overrides it, even if the maintainers of A and B agree, that's still
> not good enough for me, as they are really affecting systemd, which is
> the consumer and the provider of the interface they are using, and
> ultimately the first port of call for bug reports. This is especially
> true for udev.
>
> So in my latest revision of the patch, the general rule is as
> requested by Russ and as you mention it, but there is an explicit,
> stricter rule to cover this case, which is important to me. Policy
> calls out core component software in many places, such as dpkg, and
> systemd is already mentioned in other parts of the policy, so it did
> not seem too far-fetched to me.

I'm afraid I'm not convinced.  I'd second a patch where systemd is used
as an example of the rule, as I suggested.

Thank you for the additional commit regarding kmod.  It is good to have
been made aware of issue, but let's discuss it in a separate bug after
making this change -- the considerations might be quite different.

On Tue 09 May 2023 at 12:31AM +01, Luca Boccassi wrote:

> On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery  wrote:
>
>> Oh, thank you!  I had completely forgotten that we said something
>> about this under maintainer scripts.
>>
>> That doesn't entirely cover this case (because systemd and udev may
>> not be "that package" in this sense), but it covers much of the
>> general case.
>
> Would you like me to reword/move the new snippet?

Yes, thank you.  I will review the new version.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Mon 08 May 2023 at 12:52PM -07, Russ Allbery wrote:

> Sean Whitton  writes:
>> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:
>
>>> In other words, dpkg-divert is primarily for local administrators,
>>> non-Policy-compliant local packages that are doing unusual things, and
>>> the occasional rare problem that requires special coordination between
>>> packages, not something that Debian packages should be doing to other
>>> packages without explicit coordination.
>
>>> The rule about systemd and udev files doesn't entirely fall out of that
>>> statement,
>
>> Hmm, could you explain why?
>
> It didn't fall out of the above statement because the systemd unit file
> may not be shipped with the systemd package, but by some other random
> package, so you could have an explicit coordination with the package that
> provides the unit file but still be doing something that the systemd
> maintainers don't want to support.
>
> I think it does fall out of the somewhat squishier statement that you
> shouldn't use diversions when there's some other available mechanism to
> accomplish the same goal.

Thanks, I see.  I agree that we should have the latter statement.

>> I don't mean to dismiss the significant impact on the systemd
>> maintainers that's being claimed, but specifically calling out udev and
>> systemd configuration files seems strangely specific, for Policy, to me.
>
> I think they're a special case of the general rule that if there's some
> mechanism other than diversions to do what you want, you should use it
> instead, but it's such a common special case that we should call it out
> explicitly, particularly since a lot of people right now don't seem to
> know about masking or drop-ins.  So in other words, I think I basically
> agree with this, but I think it's worth spending some words on systemd and
> udev, more as a communication strategy than anything else.

Okay cool, I'm glad you're happy with my "For example ..." thing.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Wed, 10 May 2023 23:40:44 +0100 Luca Boccassi 
wrote:
> On Tue, 09 May 2023 00:31:20 +0100 Luca Boccassi 
> wrote:
> > On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery 
> wrote:
> > > Guillem Jover  writes:
> > > > On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> > > 
> > > >> […] I suspect Policy should say something stronger and more
> > general,
> > > >> namely that no package in Debian should divert a file from
> another
> > > >> package unless this is arranged cooperatively between the
> packages
> > to
> > > >> solve some specific (unusual) problem.
> > > 
> > > > ,--- §3.9 ---
> > > > You should not use "dpkg-divert" on a file belonging to another
> > > > package without consulting the maintainer of that package
first.
> > When
> > > > adding or removing diversions, package maintainer scripts must
> > provide
> > > > the "--package" flag to "dpkg-divert" and must not use "--
local".
> > > > `---
> > > 
> > > Oh, thank you!  I had completely forgotten that we said something
> > about
> > > this under maintainer scripts.
> > > 
> > > That doesn't entirely cover this case (because systemd and udev
may
> > not be
> > > "that package" in this sense), but it covers much of the general
> > case.
> > 
> > Would you like me to reword/move the new snippet?
> 
> Gentle ping. Any suggestion on how to reword/change the proposal?

With the only occurrence now removed, this policy change should now be
impactless: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035667

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Tue, 09 May 2023 00:31:20 +0100 Luca Boccassi 
wrote:
> On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery 
wrote:
> > Guillem Jover  writes:
> > > On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> > 
> > >> […] I suspect Policy should say something stronger and more
> general,
> > >> namely that no package in Debian should divert a file from
another
> > >> package unless this is arranged cooperatively between the
packages
> to
> > >> solve some specific (unusual) problem.
> > 
> > > ,--- §3.9 ---
> > > You should not use "dpkg-divert" on a file belonging to another
> > > package without consulting the maintainer of that package first.
> When
> > > adding or removing diversions, package maintainer scripts must
> provide
> > > the "--package" flag to "dpkg-divert" and must not use "--local".
> > > `---
> > 
> > Oh, thank you!  I had completely forgotten that we said something
> about
> > this under maintainer scripts.
> > 
> > That doesn't entirely cover this case (because systemd and udev may
> not be
> > "that package" in this sense), but it covers much of the general
> case.
> 
> Would you like me to reword/move the new snippet?

Gentle ping. Any suggestion on how to reword/change the proposal?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery  wrote:
> Guillem Jover  writes:
> > On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> 
> >> […] I suspect Policy should say something stronger and more
general,
> >> namely that no package in Debian should divert a file from another
> >> package unless this is arranged cooperatively between the packages
to
> >> solve some specific (unusual) problem.
> 
> > ,--- §3.9 ---
> > You should not use "dpkg-divert" on a file belonging to another
> > package without consulting the maintainer of that package first.
When
> > adding or removing diversions, package maintainer scripts must
provide
> > the "--package" flag to "dpkg-divert" and must not use "--local".
> > `---
> 
> Oh, thank you!  I had completely forgotten that we said something
about
> this under maintainer scripts.
> 
> That doesn't entirely cover this case (because systemd and udev may
not be
> "that package" in this sense), but it covers much of the general
case.

Would you like me to reword/move the new snippet?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Guillem Jover  writes:
> On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:

>> […] I suspect Policy should say something stronger and more general,
>> namely that no package in Debian should divert a file from another
>> package unless this is arranged cooperatively between the packages to
>> solve some specific (unusual) problem.

> ,--- §3.9 ---
> You should not use "dpkg-divert" on a file belonging to another
> package without consulting the maintainer of that package first. When
> adding or removing diversions, package maintainer scripts must provide
> the "--package" flag to "dpkg-divert" and must not use "--local".
> `---

Oh, thank you!  I had completely forgotten that we said something about
this under maintainer scripts.

That doesn't entirely cover this case (because systemd and udev may not be
"that package" in this sense), but it covers much of the general case.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Guillem Jover]

On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> […] I suspect Policy should say something stronger and more general,
> namely that no package in Debian should divert a file from another package
> unless this is arranged cooperatively between the packages to solve some
> specific (unusual) problem.

,--- §3.9 ---
You should not use "dpkg-divert" on a file belonging to another
package without consulting the maintainer of that package first. When
adding or removing diversions, package maintainer scripts must provide
the "--package" flag to "dpkg-divert" and must not use "--local".
`---

> To me, this feels similar to the case of one
> package modifying the configuration files of another package, where we
> explicitly prohibit one package modifying the configuration of another
> package except through an interface provided by the package whose
> configuration is being modified.

,--- §10.7.4 ---
If the configuration file cannot be shared as described above, the
packages must be marked as conflicting with each other. Two packages
that specify the same file as a "conffile" must conflict. This is an
instance of the general rule about not sharing files. Neither
alternatives nor diversions are likely to be appropriate in this case;
in particular, "dpkg" does not handle diverted "conffile"s well.
`---

Regards,
Guillem

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Sean Whitton  writes:
> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:

>> In other words, dpkg-divert is primarily for local administrators,
>> non-Policy-compliant local packages that are doing unusual things, and
>> the occasional rare problem that requires special coordination between
>> packages, not something that Debian packages should be doing to other
>> packages without explicit coordination.

>> The rule about systemd and udev files doesn't entirely fall out of that
>> statement,

> Hmm, could you explain why?

It didn't fall out of the above statement because the systemd unit file
may not be shipped with the systemd package, but by some other random
package, so you could have an explicit coordination with the package that
provides the unit file but still be doing something that the systemd
maintainers don't want to support.

I think it does fall out of the somewhat squishier statement that you
shouldn't use diversions when there's some other available mechanism to
accomplish the same goal.

> I don't mean to dismiss the significant impact on the systemd
> maintainers that's being claimed, but specifically calling out udev and
> systemd configuration files seems strangely specific, for Policy, to me.

I think they're a special case of the general rule that if there's some
mechanism other than diversions to do what you want, you should use it
instead, but it's such a common special case that we should call it out
explicitly, particularly since a lot of people right now don't seem to
know about masking or drop-ins.  So in other words, I think I basically
agree with this, but I think it's worth spending some words on systemd and
udev, more as a communication strategy than anything else.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 20:00, Sean Whitton  wrote:
>
> Hello,
>
> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:
>
> > In other words, dpkg-divert is primarily for local administrators,
> > non-Policy-compliant local packages that are doing unusual things, and the
> > occasional rare problem that requires special coordination between
> > packages, not something that Debian packages should be doing to other
> > packages without explicit coordination.
> >
> > The rule about systemd and udev files doesn't entirely fall out of that
> > statement,
>
> Hmm, could you explain why?
>
> > so we can still include a specific statement about them, noting that
> > drop-ins and masking make dpkg-divert unnecessary (and those
> > facilities produce better tool behavior) and therefore it should not
> > be used.
> >
> > So, ideally, the way I'd prefer to move forward is for us to add a new
> > section to the main Policy manual on diversions (probably 10.11), document
> > that this is primarily a tool for local administrators and local packages
> > to override the behavior of Debian, and that its use between Debian
> > packages should be rare, should involve coordination between the packages,
> > and should only be used to solve problems that cannot be handled through
> > other facilities such as alternatives or package-specific tools like
> > systemd's support for drop-ins and masking.  And then explicitly call out
> > systemd and udev configuration as cases where dpkg-divert should not be
> > used, alongside conffiles and critical system files.
>
> I don't mean to dismiss the significant impact on the systemd
> maintainers that's being claimed, but specifically calling out udev and
> systemd configuration files seems strangely specific, for Policy, to me.
>
> The general prohibition seems justified, and then I would be inclined to
> use the systemd and udev configuration files case as an example, which
> explains and justifies the existence of the rule.  So instead of
>
> [don't use dpkg-divert on other packages's files]  In particular,
> packages should not divert systemd and udev configuration files.
> Doing so leads to confusing command output.
>
> it seems more natural to me to do something like
>
> [don't use dpkg-divert on other packages's files]  For example,
> diverting another package's systemd units or udev configuration
> files can result in confusing command output.
>
> If this was a mistake that maintainers seemed to habitually make, the
> former would make sense, but so far I think we have just one or a few
> concrete examples, and so the correct thing to do seems to me to be to
> add normative language for only the general case.

Hi,

The specific difference, for which I think an explicit call out is
needed, is because these config files are shipped by some packages but
are not used _by_ them, they are consumed by systemd (or udev, or
kmod, etc). Specifically, if package A ships a.service, and package B
overrides it, even if the maintainers of A and B agree, that's still
not good enough for me, as they are really affecting systemd, which is
the consumer and the provider of the interface they are using, and
ultimately the first port of call for bug reports. This is especially
true for udev.

So in my latest revision of the patch, the general rule is as
requested by Russ and as you mention it, but there is an explicit,
stricter rule to cover this case, which is important to me. Policy
calls out core component software in many places, such as dpkg, and
systemd is already mentioned in other parts of the policy, so it did
not seem too far-fetched to me.

I am of course open to re-wording, adjustments, etc as deemed necessary.

Changeset at: https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:

> In other words, dpkg-divert is primarily for local administrators,
> non-Policy-compliant local packages that are doing unusual things, and the
> occasional rare problem that requires special coordination between
> packages, not something that Debian packages should be doing to other
> packages without explicit coordination.
>
> The rule about systemd and udev files doesn't entirely fall out of that
> statement,

Hmm, could you explain why?

> so we can still include a specific statement about them, noting that
> drop-ins and masking make dpkg-divert unnecessary (and those
> facilities produce better tool behavior) and therefore it should not
> be used.
>
> So, ideally, the way I'd prefer to move forward is for us to add a new
> section to the main Policy manual on diversions (probably 10.11), document
> that this is primarily a tool for local administrators and local packages
> to override the behavior of Debian, and that its use between Debian
> packages should be rare, should involve coordination between the packages,
> and should only be used to solve problems that cannot be handled through
> other facilities such as alternatives or package-specific tools like
> systemd's support for drop-ins and masking.  And then explicitly call out
> systemd and udev configuration as cases where dpkg-divert should not be
> used, alongside conffiles and critical system files.

I don't mean to dismiss the significant impact on the systemd
maintainers that's being claimed, but specifically calling out udev and
systemd configuration files seems strangely specific, for Policy, to me.

The general prohibition seems justified, and then I would be inclined to
use the systemd and udev configuration files case as an example, which
explains and justifies the existence of the rule.  So instead of

[don't use dpkg-divert on other packages's files]  In particular,
packages should not divert systemd and udev configuration files.
Doing so leads to confusing command output.

it seems more natural to me to do something like

[don't use dpkg-divert on other packages's files]  For example,
diverting another package's systemd units or udev configuration
files can result in confusing command output.

If this was a mistake that maintainers seemed to habitually make, the
former would make sense, but so far I think we have just one or a few
concrete examples, and so the correct thing to do seems to me to be to
add normative language for only the general case.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 18:31, Russ Allbery  wrote:
>
> Sam Hartman  writes:
>
> > In cases where the change being made is permitted by policy, I think you
> > have made a compelling case to vastly prefer the native systemd and udev
> > mechanisms to dpkg-divert.  I don't think that your case is strong
> > enough to forbid dpkg-divert.
>
> > As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> > It just gives surprising results to someone coming from the systemd
> > universe.
>
> I think (and please correct me, Luca, if I'm wrong) that Luca is trying to
> declare, on behalf of the systemd maintainers, that this method of
> disabling a systemd configuration file is unsupported and may not work.
> To me that does warrant a Policy "should not" even if in specific
> situations it works currently, because it implies that this approach is
> fragile and may well stop working or cause bugs in the future with no
> further notification (since that's essentially the definition of
> unsupported).
>
> Also, even apart from that, I personally would support a Policy "should
> not" for using diversions in any case where another mechanism is available
> that solves the same problem.  Diversions are a low-level tool with a lot
> of sharp edges and should be used very carefully and avoided when there is
> some other approach available.

Yes, that is precisely what I meant.

I have applied your suggestions and added a 10.10 chapter that has a
generic 'should' rule, and a more strict 'must' rule for systemd
files. I am pushing to the same branch, if you prefer me to attach
directly to the mail too let me know and I can do that, otherwise the
branch is on Salsa:

https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

I kept the wording for the dpkg-divert appendix too, because people
are bound to find it when googling, so as long as it's there I think
it should mention this too. Once it gets removed/reworked it can go.

On request from Marco, the kmod maintainer, I've also added the same
constraint for modprobe.d/ files, for exactly the same reason, as kmod
supports overrides, drop-ins and so on. I've kept it as a separate
commit on top of the other changes, given I am not involved with kmod
directly.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Sam Hartman  writes:

> In cases where the change being made is permitted by policy, I think you
> have made a compelling case to vastly prefer the native systemd and udev
> mechanisms to dpkg-divert.  I don't think that your case is strong
> enough to forbid dpkg-divert.

> As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> It just gives surprising results to someone coming from the systemd
> universe.

I think (and please correct me, Luca, if I'm wrong) that Luca is trying to
declare, on behalf of the systemd maintainers, that this method of
disabling a systemd configuration file is unsupported and may not work.
To me that does warrant a Policy "should not" even if in specific
situations it works currently, because it implies that this approach is
fragile and may well stop working or cause bugs in the future with no
further notification (since that's essentially the definition of
unsupported).

Also, even apart from that, I personally would support a Policy "should
not" for using diversions in any case where another mechanism is available
that solves the same problem.  Diversions are a low-level tool with a lot
of sharp edges and should be used very carefully and avoided when there is
some other approach available.

> But consider a package that diverts several resources, several of them
> systemd and several of them not systemd.  The maintainer might
> legitimately want to use the same mechanism for all the
> overriding/masking so that systemd resources and non-systemd resources
> were handled the same.

I'm not really convinced by this the way that I would be if we were
talking about alternatives.  With alternatives, the slave links mean that
managing a group of similar changes together is important, but dpkg-divert
has no equivalent and every diversion already has to be maintained
separately.  Given that, I think the burden of asking people to use
masking instead of diversion for systemd configuration files is a fairly
minor request, so I weigh the problems on the systemd side higher than
what feels like a modest convenience.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 16:48, Russ Allbery  wrote:
>
> I think your X-Debbugs-Cc was syntactically invalid and thus didn't work.
> I manually added in the other addresses in this reply.

Thanks - email is hard!

> Luca Boccassi  writes:
>
> > It has come to my attention that there is one package in Debian using
> > dpkg-divert to mask a systemd configuration file (an udev rule).
> > Speaking as one of the maintainers, both upstream and downstream, I find
> > this greatly undesirable for several reasons that I will outline
> > later. Hence I would like to propose explicitly mentioning that
> > dpkg-divert must not be used for systemd configuration files (units,
> > rules, etc), and instead the supported workflow (drop-ins, masking, etc)
> > must be used, both by packages and administrators. This is already
> > standard practice, and again there is only one instance that needs
> > correcting as far as I understand, and I have already provided a bug and
> > a MR for that [1][2]. So the impact of this policy change should be
> > minimal, and it's mostly to ensure more such instances are accidentally
> > added in the future.
>
> > I have a draft policy update, that adds a paragraph to the dpkg-divert
> > section of the policy. It is attached here, and also available on Salsa
> > on my fork [3].
>
> The part of Policy that you edited with this patch is basically
> unmaintained and should ideally be removed in favor of actual Policy.  (I
> had started looking at that a long time ago and then never finished.)  All
> of those appendices from the old packaging manual predate better
> documentation maintained elsewhere (such as in the dpkg package) and are
> ambiguous with regards to whether they set requirements for Debian
> packages, document things for local administrators, or something else.
> The Policy manual warns that they may not be normative, and people often
> don't think to read them (for good reason).
>
> In the case of diversions, while I certainly agree with your proposed
> rule, I suspect Policy should say something stronger and more general,
> namely that no package in Debian should divert a file from another package
> unless this is arranged cooperatively between the packages to solve some
> specific (unusual) problem.  To me, this feels similar to the case of one
> package modifying the configuration files of another package, where we
> explicitly prohibit one package modifying the configuration of another
> package except through an interface provided by the package whose
> configuration is being modified.

I'd like this to go a step further - rules, units and so on can (and
must!) be shipped by other packages, not just from src:systemd.
But as I mentioned in the other reply, bugs come to the systemd bug
tracker most often, and make our lives more difficult, even if it's a
third package that ships the configuration.
So, I'd very much want to make the rule stronger for this use case,
and forbid it even if the respective maintainers agree between
themselves that package A's unit should be diverted by package B's,
because ultimately both A and B are feeding configuration into
systemd/udev/etc, and this is just not a supported mechanism to apply
such changes.

> In other words, dpkg-divert is primarily for local administrators,
> non-Policy-compliant local packages that are doing unusual things, and the
> occasional rare problem that requires special coordination between
> packages, not something that Debian packages should be doing to other
> packages without explicit coordination.
>
> The rule about systemd and udev files doesn't entirely fall out of that
> statement, so we can still include a specific statement about them, noting
> that drop-ins and masking make dpkg-divert unnecessary (and those
> facilities produce better tool behavior) and therefore it should not be
> used.
>
> So, ideally, the way I'd prefer to move forward is for us to add a new
> section to the main Policy manual on diversions (probably 10.11), document
> that this is primarily a tool for local administrators and local packages
> to override the behavior of Debian, and that its use between Debian
> packages should be rare, should involve coordination between the packages,
> and should only be used to solve problems that cannot be handled through
> other facilities such as alternatives or package-specific tools like
> systemd's support for drop-ins and masking.  And then explicitly call out
> systemd and udev configuration as cases where dpkg-divert should not be
> used, alongside conffiles and critical system files.

Ok, I can look at adding 10.11 - I very naively searched for existing
paragraphs mentioning diverts and found the one I extended, I did not
realize it was not proper part of Policy, thanks for the pointer.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 16:39, Sam Hartman  wrote:
>
> > "Luca" == Luca Boccassi  writes:
>
> Luca> It has come to my attention that there is one package in
> Luca> Debian using dpkg-divert to mask a systemd configuration file
> Luca> (an udev rule).  Speaking as one of the maintainers, both
> Luca> upstream and downstream, I find this greatly undesirable for
> Luca> several reasons that I will outline later. Hence I would like
> Luca> to propose explicitly mentioning that dpkg- divert must not be
> Luca> used for systemd configuration files (units, rules, etc), and
> Luca> instead the supported workflow (drop-ins, masking, etc) must
> Luca> be used, both by packages and administrators.
>
> First, I thought there was already a prohibition about one package
> mucking with another package's configuration.
> In many cases it sounds like it's already against policy for one package
> to change the default systemd or udev configuration--at least for
> packages in the archive.
> (I am skepticle that amazon-ec2-utils complies with policy in general).
>
>
> In cases where the change being made is permitted by policy, I think you
> have made a compelling case to vastly prefer the native systemd and udev
> mechanisms to dpkg-divert.
> I don't think that your case is strong enough to forbid dpkg-divert.
>
> As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> It just gives surprising results to someone coming from the systemd
> universe.
>
> But consider a package that diverts several resources, several of them
> systemd and several of them not systemd.
> The maintainer might legitimately want to use the same mechanism for all
> the overriding/masking  so that systemd resources and non-systemd
> resources were handled the same.
>
> There's a real trade off there, and we generally leave those to
> maintainers.
>
> So, I'd support policy advice (ENCOURAGED) rather than policy
> requirements (MUST) in this case.
>
> I do think that if a maintainer violates this advice without a good
> reason, important is a more appropriate bug severity than wishlist, but
> unfortunately we don't have a good way to specify that in policy
> language.
>
> I would not support policy recommendation language (RECOMMENDED/SHOULD)
> because that has a connotation that failing to follow the recommendation
> is always a bug, and I don't think that's true here.

The problem is that when there are udev/systemd bugs they get directed
to the systemd team, not to the package shipping a divert, because
diverts are essentially invisible to normal users. It is already very
difficult and very time consuming to support these packages,
especially udev because we are essentially dealing with the
intersection of an infinite combination of hardware and software, and
anything that makes our lives harder is detrimental to the project at
large. If there was a significant, or even any benefit at all, then it
could be discussed, but I fail to see any. I do not find the
theoretical point about multiple diversions very compelling - we use
different mechanisms for different things all the time, but more
importantly such a case simply has never surfaced in the 10 past years
or so since we ship systemd by default, and longer since we ship udev.

Moreover, as upstream developer, I can guarantee you that masking via
diversion like this is NOT supported, and there could be more bugs
lurking, and we categorically do not intend to support or help with
such cases. I have no intention to spend any time investigating
whether such bugs exist and document them. Therefore, given there is
only one case in the whole distro so impact is minimal, the best
option to me seems to flat out forbid it, so that we never get into
that bad situation in the first place.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

I think your X-Debbugs-Cc was syntactically invalid and thus didn't work.
I manually added in the other addresses in this reply.

Luca Boccassi  writes:

> It has come to my attention that there is one package in Debian using
> dpkg-divert to mask a systemd configuration file (an udev rule).
> Speaking as one of the maintainers, both upstream and downstream, I find
> this greatly undesirable for several reasons that I will outline
> later. Hence I would like to propose explicitly mentioning that
> dpkg-divert must not be used for systemd configuration files (units,
> rules, etc), and instead the supported workflow (drop-ins, masking, etc)
> must be used, both by packages and administrators. This is already
> standard practice, and again there is only one instance that needs
> correcting as far as I understand, and I have already provided a bug and
> a MR for that [1][2]. So the impact of this policy change should be
> minimal, and it's mostly to ensure more such instances are accidentally
> added in the future.

> I have a draft policy update, that adds a paragraph to the dpkg-divert
> section of the policy. It is attached here, and also available on Salsa
> on my fork [3].

The part of Policy that you edited with this patch is basically
unmaintained and should ideally be removed in favor of actual Policy.  (I
had started looking at that a long time ago and then never finished.)  All
of those appendices from the old packaging manual predate better
documentation maintained elsewhere (such as in the dpkg package) and are
ambiguous with regards to whether they set requirements for Debian
packages, document things for local administrators, or something else.
The Policy manual warns that they may not be normative, and people often
don't think to read them (for good reason).

In the case of diversions, while I certainly agree with your proposed
rule, I suspect Policy should say something stronger and more general,
namely that no package in Debian should divert a file from another package
unless this is arranged cooperatively between the packages to solve some
specific (unusual) problem.  To me, this feels similar to the case of one
package modifying the configuration files of another package, where we
explicitly prohibit one package modifying the configuration of another
package except through an interface provided by the package whose
configuration is being modified.

In other words, dpkg-divert is primarily for local administrators,
non-Policy-compliant local packages that are doing unusual things, and the
occasional rare problem that requires special coordination between
packages, not something that Debian packages should be doing to other
packages without explicit coordination.

The rule about systemd and udev files doesn't entirely fall out of that
statement, so we can still include a specific statement about them, noting
that drop-ins and masking make dpkg-divert unnecessary (and those
facilities produce better tool behavior) and therefore it should not be
used.

So, ideally, the way I'd prefer to move forward is for us to add a new
section to the main Policy manual on diversions (probably 10.11), document
that this is primarily a tool for local administrators and local packages
to override the behavior of Debian, and that its use between Debian
packages should be rare, should involve coordination between the packages,
and should only be used to solve problems that cannot be handled through
other facilities such as alternatives or package-specific tools like
systemd's support for drop-ins and masking.  And then explicitly call out
systemd and udev configuration as cases where dpkg-divert should not be
used, alongside conffiles and critical system files.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> It has come to my attention that there is one package in
Luca> Debian using dpkg-divert to mask a systemd configuration file
Luca> (an udev rule).  Speaking as one of the maintainers, both
Luca> upstream and downstream, I find this greatly undesirable for
Luca> several reasons that I will outline later. Hence I would like
Luca> to propose explicitly mentioning that dpkg- divert must not be
Luca> used for systemd configuration files (units, rules, etc), and
Luca> instead the supported workflow (drop-ins, masking, etc) must
Luca> be used, both by packages and administrators.

First, I thought there was already a prohibition about one package
mucking with another package's configuration.
In many cases it sounds like it's already against policy for one package
to change the default systemd or udev configuration--at least for
packages in the archive.
(I am skepticle that amazon-ec2-utils complies with policy in general).


In cases where the change being made is permitted by policy, I think you
have made a compelling case to vastly prefer the native systemd and udev
mechanisms to dpkg-divert.
I don't think that your case is strong enough to forbid dpkg-divert.

As far as I can tell reading your reasoning, dpkg-divert *works fine*.
It just gives surprising results to someone coming from the systemd
universe.

But consider a package that diverts several resources, several of them
systemd and several of them not systemd.
The maintainer might legitimately want to use the same mechanism for all
the overriding/masking  so that systemd resources and non-systemd
resources were handled the same.

There's a real trade off there, and we generally leave those to
maintainers.

So, I'd support policy advice (ENCOURAGED) rather than policy
requirements (MUST) in this case.

I do think that if a maintainer violates this advice without a good
reason, important is a more appropriate bug severity than wishlist, but
unfortunately we don't have a good way to specify that in policy
language.

I would not support policy recommendation language (RECOMMENDED/SHOULD)
because that has a connotation that failing to follow the recommendation
is always a bug, and I don't think that's true here.

--Sam

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

Package: debian-policy
X-Debbugs-CC: bi...@debian.org pkg-systemd-maintain...@lists.alioth.debian.org

It has come to my attention that there is one package in Debian using
dpkg-divert to mask a systemd configuration file (an udev rule).
Speaking as one of the maintainers, both upstream and downstream, I
find this greatly undesirable for several reasons that I will outline
later. Hence I would like to propose explicitly mentioning that dpkg-
divert must not be used for systemd configuration files (units, rules,
etc), and instead the supported workflow (drop-ins, masking, etc) must
be used, both by packages and administrators. This is already standard
practice, and again there is only one instance that needs correcting as
far as I understand, and I have already provided a bug and a MR for
that [1][2]. So the impact of this policy change should be minimal, and
it's mostly to ensure more such instances are accidentally added in the
future.

I have a draft policy update, that adds a paragraph to the dpkg-divert
section of the policy. It is attached here, and also available on Salsa
on my fork [3].

The full reasoning below is what I provided on the MR for the one
existing instance, and I'll copy it mostly unchanged as I hope it's
exhaustive enough. It uses the one existing instance I found as
concrete example. This is not intended to single out the maintainer or
assign blame, but merely to illustrate the point with a concrete and
real use case. Quoting from the MR:

One of the main goals behind the systemd (and its udev component)
project is to unify how the low-level userspace components of a Linux
distro work, so that the exact same mechanisms, patterns, behaviours
and interfaces apply to a multitude of use cases, implementations and
tools. A core part of this is the configuration system. The
configuration system supports a complex schema of main contents,
overrides, drop-ins, masking and aliasing. This system is used and
understood by all systemd components, across all Linux distributions,
with the same interface, look and feel, so that users can feel at home
and know how to work the system regardless of the vendor, and so that
programs can rely on a stable and common interface that doesn't have to
be endlessly customized depending on which vendor or distribution it is
used by. The concept of 'masking' a configuration is well understood,
ubiquitous and fully supported by all the tooling, including in input
_and_ output, and logging, and so on. By using the supported mechanism
for masking we ensure that there are no surprises for users, coders and
vendors. When an unsupported masking mechanism is used, as in this
case, the fact that the original item is masked is completely hidden to
the systemd components, and thus to the interface provided to the user.
This causes at best confusion and misunderstanding, and at worst bugs
that will inevitably fall on the systemd maintainers, causing increased
workload for an already over stretched team.

A simple and obvious example of what I am referring to is already
included in the git commit for this change. Consider what happens when
udev parses the vanilla configuration (ie: without amazon-ec2-utils
installed):

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

The user/system/log is clearly notified that the file is parsed, there
are no errors, and thus is used.

Now consider what happens when the current version of amazon-ec2-utils
is installed:

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

It looks exactly the same. But something extremely different is
actually happening, in fact the opposite! The file is empty, so the
vanilla rule is effectively masked, but nothing and nobody is notified
of this very important fact when udev is running. One would have to
query dpkg-divert to figure this out, but this is something that even
someone like me, who can reasonably consider myself a proficient Debian
user, would not think of looking at.

Now finally consider what happens if amazon-ec2-utils with my proposed
change is installed:

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Skipping overridden file '/lib/udev/rules.d/60-cdrom_id.rules'.
Skipping empty file: /etc/udev/rules.d/60-cdrom_id.rules

There is an extremely clear, obvious and expected indication that the
original rule is ignored because it is overridden, _and_ that the
override is empty, as it's a masking operation. Now udev is a bit older
than the other components, so "masking" is not named explicitly as it
would be for example by systemctl, but it is the same operation and the
same result and the same interface. Something I should probably improve
a bit upstream, I will take a note of this.

Here's another example with systemctl and units. I have masked systemd-
homed.service via `systemctl mask` and this is what the interface shows
me when I check the status:

$ systemctl status