Bug#801065: Documenting how to not fail postinst on service fails to start

[Sam Hartman]

> "Wouter" == Wouter Verhelst  writes:

Wouter> On Wed, Feb 15, 2023 at 02:38:10PM -0500, Marvin Renich wrote:
>> > > > - the service fails to start in the postinst.
>> 
>> This implies that "the service is running" is part of "the
>> service is configured", which is where I disagree.

Wouter> What Steve said is that if

Wouter> - The service fails to start, *AND* - The service was
Wouter> previously running (or this is a new install)

Wouter> *THEN*

I think I disagree with Steve that postinst should fail on a new
install.

I think that failing postinst on a failed restart during upgrade is more
commonly the correct answer than ignoring the issue.
I also agree that it should be RECOMMENDED that if the restart fails
that be flagged to the admin somehow.

But in the case of krb5 and I think a few other services, there is not a
good way to detect at install time *whether* the service is sufficiently
configured to run from a systemd unit.  I could for example include a
ConditionPathExists on the Kerberos database.  That's wrong though
because in an LDAP deployment there will be no such database.

--Sam

Bug#801065: Documenting how to not fail postinst on service fails to start

[Wouter Verhelst]

On Wed, Feb 15, 2023 at 02:38:10PM -0500, Marvin Renich wrote:
> > > >  - the service fails to start in the postinst.
> 
> This implies that "the service is running" is part of "the service is
> configured", which is where I disagree.

What Steve said is that if

- The service fails to start, *AND*
- The service was previously running (or this is a new install)

*THEN*

this is something that should make postinst fail.

The two preconditions are linked, and should not be looked at
separately.

If the service was *not* previously running, then that is a different
situation.

But if the service was previously running and now a restart fails, then
obviously[1] this is a problem with the upgrade that should be looked at
by the admin, which means it must be flagged to the admin somehow.

As I mentioned in the TC discussion, one can reasonably debate what the
best way is to flag such problems, but I think it's not reasonable to
say "let's just push it under the mat, it doesn't matter".

We currently only have one sure way of telling the admin that there is a
problem, and that is "fail postinst".

As such, I think any suggestion that we ignore a restart failure of a
service that was running before the dpkg run should be accompanied by a
plan on *how* to flag this failure to the admin in a way that the admin
will know that things failed. In the absence of that, the status quo of
"postinst should fail on a restart of a service" should probably be
retained.

[1] barring extreme corner cases of the style "the admin made broken
changes but forgot to try a restart"

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

Bug#801065: Documenting how to not fail postinst on service fails to start

[Marvin Renich]

* Steve Langasek  [230214 13:09]:
> On Mon, Feb 13, 2023 at 09:03:34AM -0500, Marvin Renich wrote:
> > * Steve Langasek  [230212 00:03]:
> > > FWIW I think that it's the wrong thing to do if the "circumstances" 
> > > include
> > > reverse-dependencies on the package which expect to interact with the
> > > service the package provides, as these packages may themselves do such
> > > interaction in the maintainer script, resulting in cascading damage.
> 
> > > And the decision for whether there are reverse-dependencies on your 
> > > package
> > > is non-local and asynchronous.
> 
> > > Therefore I think it's always wrong for a package's postinst to exit 0 if:
> 
> > >  - it ships a service,
> > >  - it is a new install or an upgrade on a system where the service was
> > >previously started successfully, and
> > >  - the service fails to start in the postinst.
> 
> > I have to strongly disagree with this.  Suppose package A ships a
> > service, and package B depends on A and requires A's service to be
> > running in order for package B to be useful.  If I install A and disable
> > its service, and then install B, I would be very disappointed if B's
> > postinst script failed and left B in what dpkg considers an unconfigured
> > state.
> 
> > Succeeding the install and requiring the user to manually run some
> > documented configuration steps is _so_ much more user friendly than
> > leaving a broken package (from dpkg's POV).  I intentionally disabled A,
> > so when I want to use B, I will take the necessary steps.  I would
> > expect that attempting to use B without completing the configuration
> > would produce a useful error message.
> 
> Up until now the conversation has been about the semantics of exit codes for
> package A's maintainer script.
> 
> You are now arguing that a package *B* is not allowed to fail on install if
> the conditions for its full configuration have not been met.
> 
> And I absolutely disagree!  Your rationale that it's user-unfriendly to
> leave a package in an unconfigured state, if followed to its logical
> conclusion, applies to *any* package.  We might as well not care about
> postinst exit codes at all!

You are right, I went off on a tangent.  I agree that if a package
installation script is unable to leave the package's own configuration
in a "usable" state (barring explicit misconfiguration prior to upgrade
by the sysadmin), that it should fail.

The condition in the original bug report was that the problem that
prevented the service from starting or restarting was orthogonal to
anything the installation script was doing.  In that case, the
installation script will leave the filesystem in an identical state
regardless of whether starting the service succeeds or fails.  The
package is installed and perfectly usable if and when the outside
condition, unrelated to the package installation, is fixed.

In these cases, there is nothing more that the installation scripts can
or should do, so there is no need to leave the package in a dpkg
half-configured state.

I think the real problem is that the ideas of "configured" and "running"
have been conflated specifically for services.  The same type of issue
with a user program would not cause a failure of installation.

The part of your message that I disagree with is

> > >  - the service fails to start in the postinst.

This implies that "the service is running" is part of "the service is
configured", which is where I disagree.

...Marvin

Bug#801065: Documenting how to not fail postinst on service fails to start

[Steve Langasek]

On Mon, Feb 13, 2023 at 09:03:34AM -0500, Marvin Renich wrote:
> * Steve Langasek  [230212 00:03]:
> > FWIW I think that it's the wrong thing to do if the "circumstances" include
> > reverse-dependencies on the package which expect to interact with the
> > service the package provides, as these packages may themselves do such
> > interaction in the maintainer script, resulting in cascading damage.

> > And the decision for whether there are reverse-dependencies on your package
> > is non-local and asynchronous.

> > Therefore I think it's always wrong for a package's postinst to exit 0 if:

> >  - it ships a service,
> >  - it is a new install or an upgrade on a system where the service was
> >previously started successfully, and
> >  - the service fails to start in the postinst.

> I have to strongly disagree with this.  Suppose package A ships a
> service, and package B depends on A and requires A's service to be
> running in order for package B to be useful.  If I install A and disable
> its service, and then install B, I would be very disappointed if B's
> postinst script failed and left B in what dpkg considers an unconfigured
> state.

> Succeeding the install and requiring the user to manually run some
> documented configuration steps is _so_ much more user friendly than
> leaving a broken package (from dpkg's POV).  I intentionally disabled A,
> so when I want to use B, I will take the necessary steps.  I would
> expect that attempting to use B without completing the configuration
> would produce a useful error message.

Up until now the conversation has been about the semantics of exit codes for
package A's maintainer script.

You are now arguing that a package *B* is not allowed to fail on install if
the conditions for its full configuration have not been met.

And I absolutely disagree!  Your rationale that it's user-unfriendly to
leave a package in an unconfigured state, if followed to its logical
conclusion, applies to *any* package.  We might as well not care about
postinst exit codes at all!

This is not hyperbolic.  By imposing restrictions on the ability of packages
to raise error conditions, you are forcing the propagation of erroneous
system state up the dependency stack, breaking untold numbers of assumptions
in the reverse-dependencies.  Keeping the error handling *local* and forcing
the admin to correct the error before proceeding is part of what makes
Debian's system robust.

On 'apt dist-upgrade', having a failing maintainer script can be problematic
wrt the package manager's full state.  But there is no such counterargument
wrt robustness for the simple case of an 'apt install foo' failing its
initial configuration because the system preconditions are not met.

> Package relationships and the idea of "properly configured" have gotten
> much more complex (in a relatively small set of packages) since early
> Debian days.  Packages whose configuration has complex requirements
> should be installable without complete configuration and should be
> resilient and provide good user feedback when run before being properly
> set up.

> I also think that such packages are the exception, rather than the rule,
> and they are usually being administered by people capable of dealing
> with postponing the configuration step.

They are also capable of dealing with 'apt -f install'.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Bug#801065: Documenting how to not fail postinst on service fails to start

[Marvin Renich]

* Steve Langasek  [230212 00:03]:
> FWIW I think that it's the wrong thing to do if the "circumstances" include
> reverse-dependencies on the package which expect to interact with the
> service the package provides, as these packages may themselves do such
> interaction in the maintainer script, resulting in cascading damage.
> 
> And the decision for whether there are reverse-dependencies on your package
> is non-local and asynchronous.
> 
> Therefore I think it's always wrong for a package's postinst to exit 0 if:
> 
>  - it ships a service,
>  - it is a new install or an upgrade on a system where the service was
>previously started successfully, and
>  - the service fails to start in the postinst.

I have to strongly disagree with this.  Suppose package A ships a
service, and package B depends on A and requires A's service to be
running in order for package B to be useful.  If I install A and disable
its service, and then install B, I would be very disappointed if B's
postinst script failed and left B in what dpkg considers an unconfigured
state.

Succeeding the install and requiring the user to manually run some
documented configuration steps is _so_ much more user friendly than
leaving a broken package (from dpkg's POV).  I intentionally disabled A,
so when I want to use B, I will take the necessary steps.  I would
expect that attempting to use B without completing the configuration
would produce a useful error message.

Package relationships and the idea of "properly configured" have gotten
much more complex (in a relatively small set of packages) since early
Debian days.  Packages whose configuration has complex requirements
should be installable without complete configuration and should be
resilient and provide good user feedback when run before being properly
set up.

I also think that such packages are the exception, rather than the rule,
and they are usually being administered by people capable of dealing
with postponing the configuration step.

...Marvin

Bug#801065: Documenting how to not fail postinst on service fails to start

[Ansgar]

On Sat, 2023-02-11 at 21:12 -0800, Russ Allbery wrote:
> An interesting problem case is a package whose point is to run a service,
> but which requires mandatory and not-automatable setup before the service
> can usefully run.  After package installation, the service cannot start.
> So the options are either attempt to start the service as normal in the
> postinst but ignore the failure, or add some more complex logic to
> postinst to attempt to determine whether the service has been set up
> properly and only attempt to start the service if it has.

Just do not enable the systemd unit for the service by default for such
services.

If an admin has configured and enabled then the service will be
(re)started. If the admin has not done anything then the service will
not be started and there is no error to ignore.

You need to do that for "automatable setup" too if the automation means
debconf as the admin might not have answered to the questions, for
example when the package is installed in an automated setup ;-)

Ansgar

Bug#801065: Documenting how to not fail postinst on service fails to start

[Russ Allbery]

Steve Langasek  writes:

> Therefore I think it's always wrong for a package's postinst to exit 0 if:

>  - it ships a service,
>  - it is a new install or an upgrade on a system where the service was
>previously started successfully, and
>  - the service fails to start in the postinst.

An interesting problem case is a package whose point is to run a service,
but which requires mandatory and not-automatable setup before the service
can usefully run.  After package installation, the service cannot start.
So the options are either attempt to start the service as normal in the
postinst but ignore the failure, or add some more complex logic to
postinst to attempt to determine whether the service has been set up
properly and only attempt to start the service if it has.

I think our packaging system doesn't handle this case that well.  I can
make good arguments for several possible behavior choices.  But obviously
one cannot have package installation fail because the service cannot be
started when the package has to be installed so that you can configure it
so that the service can start.

-- 
Russ Allbery (r...@debian.org)  

Bug#801065: Documenting how to not fail postinst on service fails to start

[Steve Langasek]

On Wed, Feb 08, 2023 at 03:49:48PM +, Holger Levsen wrote:
> On Wed, Feb 08, 2023 at 08:39:36AM -0700, Sam Hartman wrote:
> > > "Holger" == Holger Levsen  writes:
> > Holger> I don't think there has been consent on the issue, thus I'm
> > Holger> tagging it moreinfo.
> > My reading of the TC and debian-devel discussion was that this was at
> > least a reasonable thing for maintainers to do,

> can you give pointers?

> > and whether it should be done depended on the circumstances.

> I do agree with that. I'm more against a general recommendation, depending
> on the circumstances, it's the right thing to do.

FWIW I think that it's the wrong thing to do if the "circumstances" include
reverse-dependencies on the package which expect to interact with the
service the package provides, as these packages may themselves do such
interaction in the maintainer script, resulting in cascading damage.

And the decision for whether there are reverse-dependencies on your package
is non-local and asynchronous.

Therefore I think it's always wrong for a package's postinst to exit 0 if:

 - it ships a service,
 - it is a new install or an upgrade on a system where the service was
   previously started successfully, and
 - the service fails to start in the postinst.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Bug#801065: Documenting how to not fail postinst on service fails to start

[Holger Levsen]

retitle -1 turn #904558 into advice - how postinst should deal with failures
thanks

On Wed, Feb 08, 2023 at 09:26:58AM -0700, Sam Hartman wrote:
> The TC bug is 904558.

thank you very much for this pointer, that's a pretty good discussion,
which resulted in

-

So, the TC declines to rule on what should maintscripts do when failing 
to
(re)start a service (or otherwise encountering a similarly serious
problem).

-
(read the full result at 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904558#137 )

so I still think "#801065 discourage failing install or upgrade when service
fails to start" is the wrong recommendation for dev-ref.

That said, I'd still appreciate a patch that conveys the gist of #904558.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

The apocalypse is here now, it’s just not equally distributed.


signature.asc
Description: PGP signature

Bug#801065: Documenting how to not fail postinst on service fails to start

[Sam Hartman]

The TC bug is 904558.
Busy with day job now.

Bug#801065: Documenting how to not fail postinst on service fails to start

[Sam Hartman]

> "Holger" == Holger Levsen  writes:

Holger> I do agree with that. I'm more against a general
Holger> recommendation, depending on the circumstances, it's the
Holger> right thing to do.

My recollection is this came before the TC, but I'm blanking on the bug
number.
But it seems like the sticking point is the general recommendation.

I'd love to figure out how to do this though and document for other
people (I have a Ubuntu bug open against krb5 that basically boils down
to this issue).

>> Holger, would you support adding a comment to 6.4 explaining how
>> to do it?

Holger> surely.

>> I'd write text but I'm honestly baffled how to accomplish this
>> for systemd units with dh.
 
Holger> :)

Marvin, do you have any clue how to accomplish?

Bug#801065: Documenting how to not fail postinst on service fails to start

[Holger Levsen]

On Wed, Feb 08, 2023 at 08:39:36AM -0700, Sam Hartman wrote:
> > "Holger" == Holger Levsen  writes:
> Holger> I don't think there has been consent on the issue, thus I'm
> Holger> tagging it moreinfo.
> My reading of the TC and debian-devel discussion was that this was at
> least a reasonable thing for maintainers to do,

can you give pointers?

> and whether it should be done depended on the circumstances.

I do agree with that. I'm more against a general recommendation, depending
on the circumstances, it's the right thing to do.

> Holger, would you support adding a comment to 6.4 explaining how to do
> it?

surely.

> I'd write text but I'm honestly baffled how to accomplish this for
> systemd units with dh.
 
:)


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Bottled water companies don't produce water, they produce plastic bottles.


signature.asc
Description: PGP signature

Bug#801065: Documenting how to not fail postinst on service fails to start

[Sam Hartman]

> "Holger" == Holger Levsen  writes:

Holger> I don't think there has been consent on the issue, thus I'm
Holger> tagging it moreinfo.

My reading of the TC and debian-devel discussion was that this was at
least a reasonable thing for maintainers to do,
and whether it should be done depended on the circumstances.

Holger, would you support adding a comment to 6.4 explaining how to do
it?
I'd write text but I'm honestly baffled how to accomplish this for
systemd units with dh.

--Sam