Re: /usr/-only image

[Russ Allbery]

Luca Boccassi  writes:

> Perhaps 'modifications' was the wrong term, I meant the whole system
> that handles the configuration. Correct me if I'm wrong, but AFAIK that
> is all Debian-specific. Arch, Fedora and Suse do not have this issue.

Speaking as the author of several PAM modules, Debian's PAM configuration
system is also vastly superior to that of Arch, Fedora, and SuSE, which
require that I as upstream provide complicated and tedious installation
documentation for how people can configure my modules.  It's a stark
contrast with Debian, where I can just ship a configuration file and have
everything happen automatically and correctly despite requiring some quite
complex PAM syntax.

-- 
Russ Allbery (r...@debian.org)  

Re: /usr/-only image

[Luca Boccassi]

On Sat, 16 Sept 2023 at 00:46, Steve Langasek  wrote:
>
> On Fri, Sep 15, 2023 at 07:44:45PM +0100, Luca Boccassi wrote:
> > In fact, Marco yesterday told me the only blocker to boot a minimal
> > Debian image with only /usr is PAM, and that's exclusively because of
> > downstream-specific changes - upstream not only has supported the
> > hermetic-usr config model for years, but the upstream maintainer is
> > one of the main drivers of the generic effort at SUSE.
>
> That's not accurate at all.  Debian carries no patches to the code for
> handling paths to pam config files.
>
> pam-auth-update is also not a "downstream change" to pam, it's integration
> with the OS that has been done in /etc.

Perhaps 'modifications' was the wrong term, I meant the whole system
that handles the configuration. Correct me if I'm wrong, but AFAIK
that is all Debian-specific. Arch, Fedora and Suse do not have this
issue.

Re: /usr/-only image

[Gioele Barabucci]

On 15/09/23 20:44, Luca Boccassi wrote:

In fact, Marco yesterday told me the only blocker to boot a minimal
Debian image with only /usr is PAM


Related: For compat >= 14 dh_installpam will install PAM modules in 
/usr, not /etc:


https://salsa.debian.org/debian/debhelper/-/merge_requests/63

--
Gioele Barabucci

Re: /usr/-only image

[Steve Langasek]

On Fri, Sep 15, 2023 at 07:44:45PM +0100, Luca Boccassi wrote:
> In fact, Marco yesterday told me the only blocker to boot a minimal
> Debian image with only /usr is PAM, and that's exclusively because of
> downstream-specific changes - upstream not only has supported the
> hermetic-usr config model for years, but the upstream maintainer is
> one of the main drivers of the generic effort at SUSE.

That's not accurate at all.  Debian carries no patches to the code for
handling paths to pam config files.

pam-auth-update is also not a "downstream change" to pam, it's integration
with the OS that has been done in /etc.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: /usr/-only image

[Luca Boccassi]

On Mon, 11 Sept 2023 at 15:09, Simon McVittie  wrote:
>
> On Mon, 11 Sep 2023 at 08:58:09 +0200, Gioele Barabucci wrote:
> > An even bigger prerequisite is that many upstream projects does not yet
> > support working without /etc or (orthogonally) reading their default
> > configuration from /usr.
>
> I agree that an "upstream first" approach is good here - even if we might
> want Debian maintainers to backport upstream changes that haven't made it
> into a release yet. Loading defaults from /usr is a behaviour change, and
> many downstream maintainers are (IMO correctly) reluctant to make
> Debian-specific behaviour changes for things that would ideally go
> upstream but have not yet been accepted upstream, because that carries a
> risk of the change never happening upstream and the Debian delta therefore
> becoming permanent.
>
> There are a couple of inter-related things here, which might make sense
> to treat as separate topics.
>
> Integration hooks
> =
>
> Quite a lot of packages have some sort of integration hook, where package A
> installs a file in a location "owned" by package B, to make the two
> packages integrate nicely together. It's best if we can change package B
> to load these integration hooks from /usr, and then systematically change
> all examples of package A to install there.
>
> Because this involves a large number of often trivially small files,
> working on this will probably have the largest impact on the number of
> files in /etc that would need special management for anyone who wants to
> use a /usr-only filesystem image.
>
> For instance, dbus-daemon (and other D-Bus implementations like
> dbus-broker) used to require system services to install a snippet into
> /etc/dbus-1/system.d. It's now deprecated for packages to install files
> into that directory, and instead, they should install into
> /usr/share/dbus-1/system.d, reserving /etc/dbus-1/system.d for sysadmin
> overrides.
>
> The steps to take here are:
>
> 1. Change package B to load from /usr in addition to /etc. This should
>usually be done upstream, because it's a behaviour change.
>(For instance where B = dbus-daemon, we did this in 2015.)
>If multiple packages all load from the same location (like dbus-daemon
>and dbus-broker) then all of them need this change.
>
> 2. For each package A, move the integration/configuration snippet from
>/etc to /usr. This should ideally be done upstream too, but it's usually
>straightforward (low maintenance cost) to do this in our packaging,
>and it doesn't necessarily *need* to happen upstream.
>
> Configuration defaults
> ==
>
> Some packages rely on their own configuration existing in /etc. For these
> packages, ideally they'd try loading from /etc as highest priority, but
> fall back to /usr as a lower-priority location. This is a
> package-by-package change, and probably best achieved upstream.

Actually this upstream work has already been happening, for years,
spearheaded by SUSE, who are the authors of libeconf
(https://github.com/openSUSE/libeconf) which provides a turn key
solution to implement this pattern of config loading.

In fact, Marco yesterday told me the only blocker to boot a minimal
Debian image with only /usr is PAM, and that's exclusively because of
downstream-specific changes - upstream not only has supported the
hermetic-usr config model for years, but the upstream maintainer is
one of the main drivers of the generic effort at SUSE.
Given this, it seems to me the first order of business should be to
get our downstream PAM sorted first.

In the meanwhile, we discussed this issue both at this week's
Image-Based Linux Summit and ASG2023, and I've got AIs on me to make
the hermetic-usr/libeconf format into a standalone spec under
uapi-group.org, and to see if we can get a cross-distro tracker for
the common upstreams (required for minimal/reduced images, it will
never be a 100% coverage affair as the problem space is too vast) that
still need changes. I'll try to get to those next week.

Re: /usr/-only image

[Russ Allbery]

Marc Haber  writes:

> I'd go so far that the systemd/udev way is a strategy to cope with
> nearly non-existent conffile handling on non-Debian distributions. We
> didn't do ourselves a favor by blindly adopting this scheme, while
> we're having a vastly superior package managed that handles conffiles
> and conffile changes so nicely.

> Please considernot throwing this advantage away for the rest of our
> distribution.

I've been using Debian for a lot of years now, and while describing our
dconfiguration handling as vastly superior is possibly warranted (it's
been a long time since I've tested the competition so I don't know from
first-hand experience), saying that changes are handled nicely doesn't fit
my experience.

I have spent hours resolving configuration changes on Debian systems that
turn out to be changes in comments or settings that I never changed, and
even more hours maintaining absurdly complicated code that tries to handle
in-place updates of all-in-one configuration files, extract information
from them that needs to be used by maintainer scripts, or juggle the
complicated interaction between debconf and the state machine of possible
user changes to the file outside of debconf.  This is certainly something
that we put a lot of effort into, and those of us who have used Debian for
a long time are used to it, but I wouldn't describe it as nice.

Most of this problem is not of our creation.  Managing configuration files
in an unbounded set of possible syntaxes, many of which are ad hoc and
have no standard parser and often do not support fragments in directories,
is an inherently impossible problem, and we try very hard to carve out
pieces of it that we can handle.  But there are many packages for which a
split configuration with a proper directory of overrides and a standard
configuration syntax would be a *drastic* improvement over our complex
single-file configuration management tools such as ucf, let alone over
basic dpkg configuration file management.

-- 
Russ Allbery (r...@debian.org)  

Re: /usr/-only image

[Marc Haber]

On Tue, 12 Sep 2023 14:37:10 +0900, Simon Richter 
wrote:
>The problem isn't so much the location of the configuration file, but 
>the method used to merge default, distro-provided and system-specific 
>configuration, and how much deviation from the default configuration is 
>expected.
>
>I'd argue that udev and systemd are kind of special here in that they 
>mostly provide a registry for other components to hook into, and that 
>the majority of users stick with the default configuration.

I'd go so far that the systemd/udev way is a strategy to cope with
nearly non-existent conffile handling on non-Debian distributions. We
didn't do ourselves a favor by blindly adopting this scheme, while
we're having a vastly superior package managed that handles conffiles
and conffile changes so nicely.

Please considernot throwing this advantage away for the rest of our
distribution.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: /usr/-only image

[Russ Allbery]

Simon Richter  writes:

> This would not work for a package like postfix, which absolutely
> requires system-specific configuration, and we'd have to be careful with
> packages like postgresql where there is a default configuration that
> works fine for hobbyists that we do not make life too difficult for
> professional users.

I don't think there's any desire to avoid system-specific configuration.
The model instead is that the package comes with a set of defaults, and if
you don't set something in the local configuration in /etc, the default is
used.  I think this is exactly the model used by Postfix for main.cf.
There are a few mandatory settings, but for the most part you can omit any
setting and the default is used.  The defaults are just hard-coded (at
least so far as I know) rather than stored in separate configuration files
in /usr, which doesn't make a fundamental difference.

The problem configuration files are ones like Postfix's master.cf, where a
whole ton of stuff almost no one ever changes is mixed into the same file
that you're supposed to change for local configuration and there's no
merger process.  And honestly I have always hated the Postfix master.cf
file, dating back to before systemd even existed.  I think it's a bad
configuration design.

That of course is just my opinion and doesn't get us anywhere closer to
using a defaults plus overrides syntax for master.cf, even assuming that
upstream would consider it.

There are a ton of packages with configuration syntaxes that were created
a very long time ago or have accumulated over time.  I maintain one of
them upstream, INN, and I'll be the first to say that the INN
configuration syntax is *awful*, and I have actively contributed to making
it what it is.  There are dozens of files, they use about fourteen
completely separate and incompatible syntaxes, there's boilerplate in some
places and defaults in other places, and learning all the ins and outs of
the configuration is a full-time job.  It's nonsense, and it's badly
designed, and if I were writing it from scratch I'd replace the whole
thing with simplified YAML or some similar well-known syntax with a schema
and good editor support and a data model that supports configuration
merging.

And the chances of any of that happening when I have more free software
projects lying on the floor in pieces than I have ones I'm managing to
keep in the air is... low, even though I do have a much more active
comaintainer.

-- 
Russ Allbery (r...@debian.org)  

Re: /usr/-only image

[Simon Richter]

Hi,

On 9/11/23 23:08, Simon McVittie wrote:


Some packages rely on their own configuration existing in /etc. For these
packages, ideally they'd try loading from /etc as highest priority, but
fall back to /usr as a lower-priority location. This is a
package-by-package change, and probably best achieved upstream.


The problem isn't so much the location of the configuration file, but 
the method used to merge default, distro-provided and system-specific 
configuration, and how much deviation from the default configuration is 
expected.


I'd argue that udev and systemd are kind of special here in that they 
mostly provide a registry for other components to hook into, and that 
the majority of users stick with the default configuration.


This would not work for a package like postfix, which absolutely 
requires system-specific configuration, and we'd have to be careful with 
packages like postgresql where there is a default configuration that 
works fine for hobbyists that we do not make life too difficult for 
professional users.


   Simon

Re: /usr/-only image

[Simon McVittie]

On Mon, 11 Sep 2023 at 08:58:09 +0200, Gioele Barabucci wrote:
> An even bigger prerequisite is that many upstream projects does not yet
> support working without /etc or (orthogonally) reading their default
> configuration from /usr.

I agree that an "upstream first" approach is good here - even if we might
want Debian maintainers to backport upstream changes that haven't made it
into a release yet. Loading defaults from /usr is a behaviour change, and
many downstream maintainers are (IMO correctly) reluctant to make
Debian-specific behaviour changes for things that would ideally go
upstream but have not yet been accepted upstream, because that carries a
risk of the change never happening upstream and the Debian delta therefore
becoming permanent.

There are a couple of inter-related things here, which might make sense
to treat as separate topics.

Integration hooks
=

Quite a lot of packages have some sort of integration hook, where package A
installs a file in a location "owned" by package B, to make the two
packages integrate nicely together. It's best if we can change package B
to load these integration hooks from /usr, and then systematically change
all examples of package A to install there.

Because this involves a large number of often trivially small files,
working on this will probably have the largest impact on the number of
files in /etc that would need special management for anyone who wants to
use a /usr-only filesystem image.

For instance, dbus-daemon (and other D-Bus implementations like
dbus-broker) used to require system services to install a snippet into
/etc/dbus-1/system.d. It's now deprecated for packages to install files
into that directory, and instead, they should install into
/usr/share/dbus-1/system.d, reserving /etc/dbus-1/system.d for sysadmin
overrides.

The steps to take here are:

1. Change package B to load from /usr in addition to /etc. This should
   usually be done upstream, because it's a behaviour change.
   (For instance where B = dbus-daemon, we did this in 2015.)
   If multiple packages all load from the same location (like dbus-daemon
   and dbus-broker) then all of them need this change.

2. For each package A, move the integration/configuration snippet from
   /etc to /usr. This should ideally be done upstream too, but it's usually
   straightforward (low maintenance cost) to do this in our packaging,
   and it doesn't necessarily *need* to happen upstream.

Configuration defaults
==

Some packages rely on their own configuration existing in /etc. For these
packages, ideally they'd try loading from /etc as highest priority, but
fall back to /usr as a lower-priority location. This is a
package-by-package change, and probably best achieved upstream.

smcv

Re: /usr/-only image

[Philipp Kern]

On 2023-09-10 22:42, Russ Allbery wrote:
So far as I know, no one has ever made a detailed, concrete proposal 
for

what the implications of this would be for Debian, what the transition
plan would look like, and how to address the various issues that will
arise.  Moving configuration files out of /etc, in particular, is
something I feel confident saying that we do not have any sort of 
project

consensus on, and is not something Debian as a project (as opposed to
individuals within the project) is currently planning on working on.


I don't feel like not having /etc is going to be feasible for a while. 
However another interesting question is what a minimal /etc looks like 
and what could be generated on-demand or regenerated from some staging 
ground in /usr and /var (e.g. the debconf database).


If you squint, that's what NixOS is doing. But we have a pretty messy 
way with multiple conflicting systems to put configuration in and how we 
merge the files when updates are installed. There would need to be some 
deeper primitives to make this happen.


Kind regards
Philipp Kern

Re: /usr/-only image

[Marco d'Itri]

On Sep 10, Russ Allbery  wrote:

> So far as I know, no one has ever made a detailed, concrete proposal for
> what the implications of this would be for Debian, what the transition
> plan would look like, and how to address the various issues that will
> arise.  Moving configuration files out of /etc, in particular, is
> something I feel confident saying that we do not have any sort of project
> consensus on, and is not something Debian as a project (as opposed to
> individuals within the project) is currently planning on working on.
Indeed, at this point we are just experimenting and trying to figure out 
what we can and cannot do.
I am not even sure that general distribution-wide support for this is 
a reasonable goal, but I expect that it will not be hard to have support
for the base system and cooperating packages.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: /usr/-only image

[Gioele Barabucci]

On 10/09/23 22:42, Russ Allbery wrote:

Luca Boccassi  writes:

It is being slowly worked towards, but we are still at the prerequisites
at this time. Hopefully we'll have some usable experiments for the
Trixie timeline, but nothing definite yet.


Just to make this explicit, one of the prerequisites that has not yet
happened is for Debian to agree that this is even something that we intend
to do.


An even bigger prerequisite is that many upstream projects does not yet 
support working without /etc or (orthogonally) reading their default 
configuration from /usr.


In my personal experience I found that working directly with upstream 
projects on supporting stateless installs (no /etc, no /var) and 
factory-resets (the system keeps working after `rm -Rf /etc /var`) has a 
better return in investment compared to trying to integrate Debian- or 
distro-specific patches.


Perhaps a more approachable goal for Debian towards statelessness is to 
reach the point where no packages ship files in /var. That is often a 
matter of passing the right parameters to `dh_auto_configure`.


Regards,

--
Gioele Barabucci

Re: /usr/-only image

[Luca Boccassi]

On Sun, 10 Sept 2023 at 21:43, Russ Allbery  wrote:
> (This does not rule out the possibility that certain carefully-crafted
> configurations with a subset of packages may work in this mode, of
> course.)

Yes, this is pretty much what we are talking about in these cases -
targeted experiments, to see what it takes to have a minimal working
bootable image, what are the biggest blockers, and so on. IE,
exploratory development work, not officially supported images or
things like that.

Re: /usr/-only image

[Russ Allbery]

Luca Boccassi  writes:
> On Sun, 10 Sept 2023 at 18:55, Nils Kattenbeck  wrote:

>> I am looking to generate a Debian image with only a /usr and /var
>> partition as per discoverable partition specification. However, it
>> seems to me like the omission of /etc leads to several issues in core
>> packages and logging in becomes impossible.

>> Is this an unsupported use case and if yes, is there ongoing work to
>> eventually support this?

>> Many packages in Fedora for example are already configured to support
>> this using systemd-sysuser, systemd-tmpfiles, and other declarative
>> means stored in /usr/ to create any required files upon boot.

> It is being slowly worked towards, but we are still at the prerequisites
> at this time. Hopefully we'll have some usable experiments for the
> Trixie timeline, but nothing definite yet.

Just to make this explicit, one of the prerequisites that has not yet
happened is for Debian to agree that this is even something that we intend
to do.

So far as I know, no one has ever made a detailed, concrete proposal for
what the implications of this would be for Debian, what the transition
plan would look like, and how to address the various issues that will
arise.  Moving configuration files out of /etc, in particular, is
something I feel confident saying that we do not have any sort of project
consensus on, and is not something Debian as a project (as opposed to
individuals within the project) is currently planning on working on.

That doesn't mean we won't eventually do this, or that people aren't
working on other prerequisites, or that it's not something that we're
considering.  But I just want to make clear that we are so early in this
process that it is not at all clear that we are even going to do this at
all, and there is a substantial discussion that would need to happen and
detailed design proposal that would need to be written before there is any
chance whatsoever that Debian will officially support this configuration.
(This does not rule out the possibility that certain carefully-crafted
configurations with a subset of packages may work in this mode, of
course.)

-- 
Russ Allbery (r...@debian.org)  

Re: /usr/-only image

[Luca Boccassi]

On Sun, 10 Sept 2023 at 18:55, Nils Kattenbeck  wrote:
>
> Hello,
>
> I am looking to generate a Debian image with only a /usr and /var
> partition as per discoverable partition specification. However, it
> seems to me like the omission of /etc leads to several issues in core
> packages and logging in becomes impossible.
> Is this an unsupported use case and if yes, is there ongoing work to
> eventually support this?
> Many packages in Fedora for example are already configured to support
> this using systemd-sysuser, systemd-tmpfiles, and other declarative
> means stored in /usr/ to create any required files upon boot.

It is being slowly worked towards, but we are still at the
prerequisites at this time. Hopefully we'll have some usable
experiments for the Trixie timeline, but nothing definite yet.

Re: /usr/-only image

[Marco d'Itri]

On Sep 10, Nils Kattenbeck  wrote:

> I am looking to generate a Debian image with only a /usr and /var
> partition as per discoverable partition specification. However, it
> seems to me like the omission of /etc leads to several issues in core
> packages and logging in becomes impossible.
> Is this an unsupported use case and if yes, is there ongoing work to
> eventually support this?
There are some people generally working in this area, but I fear that 
we are still far from being able to support an empty /etc/.
Still, it would be very good to start documenting what needs to be 
fixed.

I suggest that you join #debian-systemd.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

/usr/-only image

[Nils Kattenbeck]

Hello,

I am looking to generate a Debian image with only a /usr and /var
partition as per discoverable partition specification. However, it
seems to me like the omission of /etc leads to several issues in core
packages and logging in becomes impossible.
Is this an unsupported use case and if yes, is there ongoing work to
eventually support this?
Many packages in Fedora for example are already configured to support
this using systemd-sysuser, systemd-tmpfiles, and other declarative
means stored in /usr/ to create any required files upon boot.

Kind regards, Nils