Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Wed, May 03, 2017 at 04:51:27AM -0400, Tom H wrote:
> On Sun, Apr 30, 2017 at 2:22 PM, Wouter Verhelst  wrote:
> > On Sat, Apr 29, 2017 at 03:48:09PM -0400, Tom H wrote:
> >> On Thu, Apr 27, 2017 at 3:18 AM, Wouter Verhelst  wrote:
> >>>
> >>> I didn't say RPM *doesn't* deal with changed files; I said ours
> >>> deals with it better. I stand by that.
> >>
> >> Sure; and an rpm or emerge user'll tell you that dpkg is inferior
> >> because an interactive upgrade's a crazy thing to do.
> >
> > Yes, sure. This discussion is getting increasingly side-tracked though.
> >
> > The original question was "should I install defaults in /etc or /usr?"
> > to which I replied that in Debian, we've traditionally done the former
> > rather than the latter, and that the latter feels like a result of an
> > ecosystem (other than ours) where dealing with conflicting changes to
> > configuration files is frowned upon. I think our way is better, but
> > I'm sure others disagree.
> 
> If Debian decides to drop into "/etc" files that are dropped into
> "/usr/lib" (or "/lib") upstream because rpm and others can't handle
> config file upgrades, it would be a decision not based on facts.

You completely misunderstood what I said. EOT for me.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Sun, Apr 30, 2017 at 2:22 PM, Wouter Verhelst  wrote:
> On Sat, Apr 29, 2017 at 03:48:09PM -0400, Tom H wrote:
>> On Thu, Apr 27, 2017 at 3:18 AM, Wouter Verhelst  wrote:
>>>
>>> I didn't say RPM *doesn't* deal with changed files; I said ours
>>> deals with it better. I stand by that.
>>
>> Sure; and an rpm or emerge user'll tell you that dpkg is inferior
>> because an interactive upgrade's a crazy thing to do.
>
> Yes, sure. This discussion is getting increasingly side-tracked though.
>
> The original question was "should I install defaults in /etc or /usr?"
> to which I replied that in Debian, we've traditionally done the former
> rather than the latter, and that the latter feels like a result of an
> ecosystem (other than ours) where dealing with conflicting changes to
> configuration files is frowned upon. I think our way is better, but
> I'm sure others disagree.

If Debian decides to drop into "/etc" files that are dropped into
"/usr/lib" (or "/lib") upstream because rpm and others can't handle
config file upgrades, it would be a decision not based on facts.

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Mon, 1 May 2017 21:16:33 +1000, Scott Leggett 
wrote:
>On 2017-05-01.13:02, Marc Haber wrote:
>> On Mon, 1 May 2017 11:09:26 +0200, Christian Seiler
>>  wrote:
>> >And as I said in other places in this thread: I personally
>> >think that the separate /usr <-> /etc scheme is much better
>> >than just storing everything in /etc, so I would really
>> >prefer if as much software as possible would switch to that,
>> 
>> How would you expect the case "local admin has copied over the file
>> from /usr to /etc to apply local changes, and a new package version
>> changes its defaults so that the service wouldn't run with an
>> unadapted configuration any more" to be handled?
>
>Via a NEWS entry? I'd expect _any_ significant change in default
>behaviour in a package to be announced in a NEWS entry.

While that would be ideal, being pointed to it by dpkg is automatic.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Holger Levsen]

On Mon, May 01, 2017 at 11:09:26AM +0200, Christian Seiler wrote:
> My point is just the following: this is really nothing new.
> Some packages have been doing things like this for over a
> decade.

Thanks for coming up with these facts. 

(Much more interesting than developers of "system A" discussing how
"system A" is better than "system B"…)

> And as I said in other places in this thread: I personally
> think that the separate /usr <-> /etc scheme is much better
> than just storing everything in /etc, so I would really
> prefer if as much software as possible would switch to that,
> but do it consistently, [...]
 
agreed.


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: policy for shipping sysctl.d snippets in packages?

[Christian Seiler]

On 05/01/2017 01:02 PM, Marc Haber wrote:
> On Mon, 1 May 2017 11:09:26 +0200, Christian Seiler
>  wrote:
>> And as I said in other places in this thread: I personally
>> think that the separate /usr <-> /etc scheme is much better
>> than just storing everything in /etc, so I would really
>> prefer if as much software as possible would switch to that,
> 
> How would you expect the case "local admin has copied over the file
> from /usr to /etc to apply local changes, and a new package version
> changes its defaults so that the service wouldn't run with an
> unadapted configuration any more" to be handled?
> 
> The Debian way will handle this through conffile managent or ucf,
> making the local admin immediately aware of the changes, while the way
> you advocate would make the admin end up with a (probably silently)
> failed service, broken by a simple package upgrade.

That's not necessarily true. This may be true if you have a
single configuration file, but several programs come with
multiple configuration files - and sometimes certain settings
depend on each other. Heck, I've seen the following scheme
multiple times:

/etc/$package/$main_config:

 include /etc/$package/local

Where it's encouraged to leave the main config file alone
and just configure stuff in the local configuration file.
There you won't get any feedback due to dpkg/ucf either.

Or, let's say a default configuration option changes for
security reasons, but that configuration option makes another
option you're using (and added elsewhere in the file) not work
anymore and the daemon doesn't start either. If you are doing
an automatic 3-way-merge, you'll run into this problem. And
even if you merge manually, you might not realize that the
options are incompatible just from looking at the config file
at that point.

Or, to put it differently: I've experienced the problems
you've been describing very often already with the traditional
"config is in /etc" scheme. It's true that the separate /usr, 
/etc scheme has some disadvantages in this specific area, as
there are always trade-offs when it comes to solutions. But
from personal experience I simply don't agree with the case
made here that having everything in conffiles in /etc makes it
possible or even easy to track these kinds of things
consistently enough that it makes up for all of the pain I
experience when it comes to handling configuration files on
updates.

Furthermore, I don't think the "gold standard" for a /usr and
/etc split should be to copy the entire file from /usr into
/etc and just change a small setting. That might be necessary
in some cases (due to limitation of the configuration file
language in question), but I personally think the best solution
here to have .d/-style directories (see e.g.
/etc/apt/preferences.d/) where you can drop in small files that
change only the settings you want to change.

> And I am not yet bringing the case where the package maintainer fixes
> a security issue by a configuration change in the game.

I would expect the DSA announcement to contain information on
this topic, plus potentially even a NEWS file.

>> [1] This is _not_ systemd. systemd introduced an own scheme via
>>/usr/lib/binfmt.d + /etc/binfmt.d, which no package in Debian
>>sid or stretch currently uses.
>>
>>And note that binfmt-support has always worked that way, for
>>at least 15 years, way before systemd came around.
> 
> Having configuration in /usr without a possibility to override it via
> /etc is a policy violation, isn't it`

Maybe. (I've been burnt in the past by assuming that certain
things are in policy when they in fact weren't. I'm much more
careful with such statements now.) It certainly is something
that's not desirable. OTOH, one could argue that binfmt is
more like "plugins for the kernel" instead of configuration
(similar to how plugin .so files can be installed into
directories under /usr/lib). Since this has been this way since
forever, I don't think it makes sense to drastically change it
in Stretch so close to the release - but we should probably
revisit this at the beginning of the Buster release cycle.
(There's also the question of whether we'd not rather want to
adopt the binfmt.d scheme on all init systems, as that does
exactly the same with a slightly different syntax, and other
distros are now using just that, and slowly phasing out the
/usr/share/binfmts directory, while still supporting it for the
foreseeable future for compatibility reasons.)

Regards,
Christian

[1] I really wouldn't want to call it the "Debian way". Having
configuration files completely in /etc is something that
predates Debian by a long time.

Re: policy for shipping sysctl.d snippets in packages?

[Scott Leggett]

On 2017-05-01.13:02, Marc Haber wrote:
> On Mon, 1 May 2017 11:09:26 +0200, Christian Seiler
>  wrote:
> >And as I said in other places in this thread: I personally
> >think that the separate /usr <-> /etc scheme is much better
> >than just storing everything in /etc, so I would really
> >prefer if as much software as possible would switch to that,
> 
> How would you expect the case "local admin has copied over the file
> from /usr to /etc to apply local changes, and a new package version
> changes its defaults so that the service wouldn't run with an
> unadapted configuration any more" to be handled?

Via a NEWS entry? I'd expect _any_ significant change in default
behaviour in a package to be announced in a NEWS entry.

-- 
Regards,
Scott.


signature.asc
Description: PGP signature

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Mon, 1 May 2017 11:09:26 +0200, Christian Seiler
 wrote:
>And as I said in other places in this thread: I personally
>think that the separate /usr <-> /etc scheme is much better
>than just storing everything in /etc, so I would really
>prefer if as much software as possible would switch to that,

How would you expect the case "local admin has copied over the file
from /usr to /etc to apply local changes, and a new package version
changes its defaults so that the service wouldn't run with an
unadapted configuration any more" to be handled?

The Debian way will handle this through conffile managent or ucf,
making the local admin immediately aware of the changes, while the way
you advocate would make the admin end up with a (probably silently)
failed service, broken by a simple package upgrade.

And I am not yet bringing the case where the package maintainer fixes
a security issue by a configuration change in the game.

>[1] This is _not_ systemd. systemd introduced an own scheme via
>/usr/lib/binfmt.d + /etc/binfmt.d, which no package in Debian
>sid or stretch currently uses.
>
>And note that binfmt-support has always worked that way, for
>at least 15 years, way before systemd came around.

Having configuration in /usr without a possibility to override it via
/etc is a policy violation, isn't it`

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Christian Seiler]

On 05/01/2017 09:13 AM, Marc Haber wrote:
> I find it already disturbing that we have diverged from "our" way in
> systemd, which is probably the first package a local admin will be
> exposed to.

This is nothing new though. For example, DBus has had the /usr and
/etc split since as far back as I can remember. (Granted, DBus isn't
a great example, because I consider its configuration to be quite
horrible for an end-user, especially because it's XML.)

Another example: Firefox stores its default preferences in
/usr/share/firefox-esr/browser/defaults/preferences/firefox.js,
and while there is /etc/firefox-esr/firefox.js, that is applied as
a delta. (Plus then the user's local configuration is applied as
a delta on top of that.)

binfmt-support, the Debian-specific way of adding binfmts to the
kernel [1], is _only_ configured in /usr/share/binfmts. There is
no way for users to override this other than by using dpkg-divert.

The entire XDG menu and MIME specifications are built around the
defaults being in /usr and /usr/local.

LXC's configuration system works via includes, and containers
created by the default templates just add

   include /usr/share/lxc/config/$TYPE.common.conf

to the container's specific configuration file.

The current X server supports split-snippets in
/usr/share/X11/xorg.conf.d and /etc/X11/xorg.conf.d - and all
Debian packages that install snippets put them in /usr/share.
(Granted, most are part of the X11 project.)

dput-ng searches for its configuration in /usr/share/dput-ng,
/etc/dput.d and ~/.dput.d. (Plus /etc/dput.cf and ~/.dput.cf
for compatibility reasons.) You can't get more Debian-specific
than that. And support for dput.d is 5 years old already.

The nano editor's default configuration has
include "/usr/share/nano/*.nanorc"
in /etc/nanorc.

AppArmor has
Include /usr/share/apparmor
in its default configuration.

There are likely quite a few more examples.

My point is just the following: this is really nothing new.
Some packages have been doing things like this for over a
decade. Of course, since systemd is quite a central component
to a system, this has become more prominent, but the systemd
authors were not the first to implement a scheme like this.
And if you look at the subtle differences between all of the
examples I've provided in how they read stuff from /usr, I'm
pretty sure that several of them came up with that type of
scheme independently of each other. Which is why the list of
examples I've provided is not necessarily a list of "good"
examples in the sense that "yes, they're doing it right",
but rather a list of examples that demonstrate that this
scheme scratches a very real itch.

And as I said in other places in this thread: I personally
think that the separate /usr <-> /etc scheme is much better
than just storing everything in /etc, so I would really
prefer if as much software as possible would switch to that,
but do it consistently, following e.g. dput-ng or systemd,
but not for example like binfmt-support.

Regards,
Christian

[1] This is _not_ systemd. systemd introduced an own scheme via
/usr/lib/binfmt.d + /etc/binfmt.d, which no package in Debian
sid or stretch currently uses.

And note that binfmt-support has always worked that way, for
at least 15 years, way before systemd came around.

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Sun, 30 Apr 2017 20:22:45 +0200, Wouter Verhelst
 wrote:
>The original question was "should I install defaults in /etc or /usr?"
>to which I replied that in Debian, we've traditionally done the former
>rather than the latter, and that the latter feels like a result of an
>ecosystem (other than ours) where dealing with conflicting changes to
>configuration files is frowned upon. I think our way is better, but I'm
>sure others disagree.

Just for the record: I agree.

>Regardless, traditionally we've been doing it "our" way, and therefore
>it seems reasonable to believe that our users will expect things to
>continue working "our" way. Whether that is better than the "other" way
>is besides the point that I was trying to make (even if I believe that
>to be true).

What Wouter says.

I find it already disturbing that we have diverged from "our" way in
systemd, which is probably the first package a local admin will be
exposed to.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Sat, Apr 29, 2017 at 03:48:09PM -0400, Tom H wrote:
> On Thu, Apr 27, 2017 at 3:18 AM, Wouter Verhelst  wrote:
> > I didn't say RPM *doesn't* deal with changed files; I said ours deals
> > with it better. I stand by that.
> 
> Sure; and an rpm or emerge user'll tell you that dpkg is inferior
> because an interactive upgrade's a crazy thing to do.

Yes, sure. This discussion is getting increasingly side-tracked though.

The original question was "should I install defaults in /etc or /usr?"
to which I replied that in Debian, we've traditionally done the former
rather than the latter, and that the latter feels like a result of an
ecosystem (other than ours) where dealing with conflicting changes to
configuration files is frowned upon. I think our way is better, but I'm
sure others disagree.

Regardless, traditionally we've been doing it "our" way, and therefore
it seems reasonable to believe that our users will expect things to
continue working "our" way. Whether that is better than the "other" way
is besides the point that I was trying to make (even if I believe that
to be true).

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Sun, 30 Apr 2017 15:49:52 +0200, Adam Borowski
 wrote:
>The only "benefit" I see is that the RPM way is less work for the
>maintainer.

Amen.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Adam Borowski]

On Sun, Apr 30, 2017 at 12:16:31PM +0200, Marc Haber wrote:
> On Sat, 29 Apr 2017 15:48:09 -0400, Tom H  wrote:
> >On Thu, Apr 27, 2017 at 3:18 AM, Wouter Verhelst  wrote:
> >> I didn't say RPM *doesn't* deal with changed files; I said ours deals
> >> with it better. I stand by that.
> >
> >Sure; and an rpm or emerge user'll tell you that dpkg is inferior
> >because an interactive upgrade's a crazy thing to do.
> 
> Debian is about choice. You can choose to reinstall or to upgrade. You
> don't have that choice in the RHEL world.

In real world, this means you end up with RHEL servers all over the place
that lost their security support a decade ago.  I on the other hand have a
potato install (currently in a lxc container but still doing its work) and a
testing/sarge install still in mostly-original machine, already on stretch.

Reinstalling everything on a server is costly, thus it's no surprise it's
postponed indefinitely while that server appears to work fine.

No conffile handling is a smaller version of this: you either leave the old,
wrong and likely insecure config untouched, or need to rewrite it from
scratch.  That's hardly better than reinstalling.

The only "benefit" I see is that the RPM way is less work for the
maintainer.


Meow!
-- 
Don't be racist.  White, amber or black, all beers should be judged based
solely on their merits.  Heck, even if occasionally a cider applies for a
beer's job, why not?
On the other hand, corpo lager is not a race.

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Sat, 29 Apr 2017 15:48:09 -0400, Tom H  wrote:
>On Thu, Apr 27, 2017 at 3:18 AM, Wouter Verhelst  wrote:
>> I didn't say RPM *doesn't* deal with changed files; I said ours deals
>> with it better. I stand by that.
>
>Sure; and an rpm or emerge user'll tell you that dpkg is inferior
>because an interactive upgrade's a crazy thing to do.

Debian is about choice. You can choose to reinstall or to upgrade. You
don't have that choice in the RHEL world.

-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Fri, Apr 28, 2017 at 12:08 PM, Wouter Verhelst  wrote:
> On Fri, Apr 28, 2017 at 04:21:17AM -0400, Tom H wrote:
>>
>> Did Linux development move as quickly as it does now?
>> Did users experience more problems or failures when running those 
>> dist-upgrades?
>
> RedHat also did not support upgrades back when they did not wait four
> years to do finish a new release. They do not support them, because
> they *choose* not to support them, instead telling people to reinstall.
>
> Yes, that makes it easier for them to wait four years between releases.
> But I think you have cause and effect swapped around.

I've never even tried to dist-upgrade RHEL but I have dist-upgraded
RHL and Fedora. It was/is perfectly do-able but they both had/have
shorter release cycles.

IF Red Hat had felt that there was a market for dist-upgrade support,
it could have offered it for a extra fee, just like it offers an
extended support subscription for companies that want to stick to want
to stick to a release beyond the standard EOL or to a minor release.
It does offer an *unsupported* upgrade via anaconda.

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Thu, Apr 27, 2017 at 3:18 AM, Wouter Verhelst  wrote:
> On Wed, Apr 26, 2017 at 07:53:57AM -0400, Tom H wrote:
>> On Sun, Apr 23, 2017 at 3:00 PM, Wouter Verhelst  wrote:
>>>
>>> The "packages drop files in /usr/*, sysadmins override in /etc" way of
>>> doing things is prevalent in the RPM world; in Debian, however, we
>>> traditionally have packages drop files in /etc, and let the maintainer
>>> change them in place. This is possible, because our package management
>>> system deals better with changed files than does RPM (which must work
>>> silently, rather than confirming things with the user).
>>
>> s/package management system deals better/package management system
>> deals differently/
>>
>> rpm doesn't have a problem with config file handling and deals with
>> config files in a similar way that dpkg uses the "conffile" attribute
>> to deal with them. rpm spec files use two (one-and-a-half?) macros:
>>
>> - "%config": "foo.conf" is replaced in an upgrade and saved as
>> "foo.conf.rpmsave";
>>
>> - "%config(noreplace)": "foo.conf" isn't replaced in an upgrade and
>> the new "foo.conf" is installed as "foo.conf.rpmnew".
>
> Yes, I am aware of that (many of my customers use RedHat systems).
> However, you will notice the complete absense of a "merge" option in the
> above. This means that new configuration files are dropped on the
> system, *without* any active notification to the user, so it's up to you
> to figure out that this has happened and that you may have work to do.
>
> I didn't say RPM *doesn't* deal with changed files; I said ours deals
> with it better. I stand by that.

Sure; and an rpm or emerge user'll tell you that dpkg is inferior
because an interactive upgrade's a crazy thing to do.

Re: policy for shipping sysctl.d snippets in packages?

[Russ Allbery]

Russ Allbery  writes:

> Yup, this.  It works because we support it, test it, treat bugs in the
> upgrade process as critical, and take it into account in our release
> engineering.  It's a lot of work.

> Red Hat has chosen not to do that work, so they don't support it.  It's
> a tradeoff decision.

Just for the Google record, someone mailed me privately (apparently
something went wrong with getting mail sent to the list) to point out that
this lack of an upgrade path between RHEL versions was a one-time thing
for one specific RHEL version transition, and RHEL does indeed support
in-place upgrades for support customers for subsequent releases (and has
tools and whatnot to help determine if anything will break).

-- 
Russ Allbery (r...@debian.org)   

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Fri, Apr 28, 2017 at 04:21:17AM -0400, Tom H wrote:
> Did Linux development move as quickly as it does now?
> Did users experience more problems or failures when running those 
> dist-upgrades?

RedHat also did not support upgrades back when they did not wait four
years to do finish a new release. They do not support them, because they
*choose* not to support them, instead telling people to reinstall.

Yes, that makes it easier for them to wait four years between releases.
But I think you have cause and effect swapped around.

[...]

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Thu, Apr 27, 2017 at 2:34 AM, Brian May  wrote:
> On 2017-04-27 16:19, Andrey Rahmatullin wrote:
>>
>> It seems you've missed the point (which was about 4 years between RHEL
>> releases).
>
> There was almost three years between Woody (July 19th 2002) and Sarge (June
> 6th 2005), yet we still allowed upgrades from Woody to Sarge.
>
> The time duration is irrelevant. It is the policy we have that we support
> and test upgrades that matters. It is much easier to ignore upgrades and
> recommend to reinstall from scratch, that means we don't need to test and
> debug why upgrades break under various corner cases. Not so good for our
> users however.

Did Linux development move as quickly as it does now?

Did users experience more problems or failures when running those dist-upgrades?

Of course duration matters. It's not the same use-case as a Debian
dist-upgrade but feel free to look up gentoo-user@ threads where a
user kicks them off with "I haven't upgraded for 6 months, 1 year, 3
years." The longer the period, the more problems.

Simply because Debian supports dist-upgrades doesn't make them easy or
doesn't make the duration between them irrelevant. We're on a more or
less two-year cycle and it makes dist-upgrades easier that if we were
on a 4-year cycle; I don't see what can possibly be debatable about
this.

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Wed, Apr 26, 2017 at 8:12 PM, Luca Capello  wrote:
> On Wed, 26 Apr 2017 08:05:10 -0400, Tom H wrote:
>>
>> You can't dist-upgrade RHEL from 6 to 7 and you can't dist-upgrade
>> Debian from 6 to 8 in one leap.
>
> Debian *does* support dist-upgrading between *following* releases only,
> which is what we are talking about.
>
> And, in any case, for your example on Debian you will do 6 -> 7 -> 8.

"in one leap"

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Wed, Apr 26, 2017 at 2:55 PM, Ben Hutchings  wrote:
> On Wed, 2017-04-26 at 07:53 -0400, Tom H wrote:
>>> On Sun, Apr 23, 2017 at 3:00 PM, Wouter Verhelst  wrote:


>>> The "packages drop files in /usr/*, sysadmins override in /etc" way of
>>> doing things is prevalent in the RPM world; in Debian, however, we
>>> traditionally have packages drop files in /etc, and let the maintainer
>>> change them in place. This is possible, because our package management
>>> system deals better with changed files than does RPM (which must work
>>> silently, rather than confirming things with the user).
>>
>> s/package management system deals better/package management system
>> deals differently/
>>
>> rpm doesn't have a problem with config file handling and deals with
>> config files in a similar way that dpkg uses the "conffile" attribute
>> to deal with them. rpm spec files use two (one-and-a-half?) macros:
>>
>> - "%config": "foo.conf" is replaced in an upgrade and saved as
>> "foo.conf.rpmsave";
>>
>> - "%config(noreplace)": "foo.conf" isn't replaced in an upgrade and
>> the new "foo.conf" is installed as "foo.conf.rpmnew".
>
> I didn't know about this, and I'm pleased to see that this is (now)
> possible. Is this documented somewhere? (I've never been able to find
> documentation of RPM macros that isn't very old and incomplete.)

I first used it with RHEL 3 for some company-developed apps but it
might predate that.
But Google says that it was available since at least RHL 9:

http://people.ds.cam.ac.uk/jw35/docs/rpm_config.html

Fedora has an rpm draft doc:

https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/index.html


>> So rpm isn't a factor;
>
> It is, because rpm is non-interactive and the above choice has to be
> made by the packager and not the adminsitrator.

It must've slipped my mind because I disable the interactivity.

Indeed, dpkg has the advantage that an administrator can change the
behavior at run-time, for the current transaction or for all, but that
flexibility doesn't give dpkg an advantage over rpm for dropping
config files in "/etc" rather than in "/usr/lib".


>> upstreams drop files into "/usr/lib" because
>> Red Hat is pushing the concept of all/almost-all distro-provided files
>> in "/usr".

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Fri, Apr 28, 2017 at 08:03:13AM +0200, Marc Haber wrote:
> On Thu, 27 Apr 2017 09:18:54 +0200, Wouter Verhelst
>  wrote:
> >I didn't say RPM *doesn't* deal with changed files; I said ours deals
> >with it better. I stand by that.
> 
> Given that dpkg's conffile handling still has much headroom for
> improvement, this is a rather sad statement for the rpm half of the
> world.

Well, yes. I know why I'm involved with Debian, not Fedora ;-)

(to be fair, there's loads of other reasons besides just "better
conffile handling", but that's a different matter)

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[Evgeni Golov]

On Sun, Apr 23, 2017 at 09:00:41PM +0200, Wouter Verhelst wrote:
> On Sun, Apr 23, 2017 at 12:16:58PM +0200, Evgeni Golov wrote:
> > Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
> > one package (systemd-coredump) uses it, all others drop files in
> > /etc/sysctl.d.
> 
> The "packages drop files in /usr/*, sysadmins override in /etc" way of
> doing things is prevalent in the RPM world;

Is it? All my Fedora, CentOS and RHEL systems have quite a full /etc ;)

> in Debian, however, we
> traditionally have packages drop files in /etc, and let the maintainer
> change them in place. This is possible, because our package management
> system deals better with changed files than does RPM (which must work
> silently, rather than confirming things with the user).

While I would agree that RPM sometimes has its difficulties to keep up
with dpkg, I welcome the idea that neither the package manager nor the
package itself tries to be "too smart". But that a whole different
discussion than /etc vs /usr/lib.

> The reason both procps and systemd support /usr/* files is presumably
> because they're installed and shipped in both worlds, and it makes
> little sense to patch software to *remove* a feature, even if we end up
> not using it. However, that doesn't mean we should necessarily drop
> files in /usr if we can avoid it.

But there is no policy for either.
And that is what I am looking for.

> There are things to be said to have the whole default configuration live
> in /etc; IMO, it makes it easier for a system administrator to figure
> out what the current configuration is, rather than having to mentally
> merge configuration files from several locations. Additionally, when a
> configuration file that had been edited by a user is now also edited by
> the package maintainer, on Debian the system will ask how to handle it,
> rather than changing the defaults and not telling people (which can
> break in some circumstances). In contrast, on an RPM system the system
> will just create the new configuration file with a .rpmnew extension,
> but will have it not active.

To keep the rpm vs dpkg:
RPM supports both: keeping the old config and storing the new one as .rpmnew
and replacing the old config and saving the old one as .rpmsave.
Super handy when the config actually changes so much that you don't want
to keep the old one.

And no, I don't think dpkg is always helpful when it comes to conffiles:

% docker run -ti debian:wheezy /bin/bash
# apt-get update
# apt-get install apache2
# echo "# a comment" >> /etc/apache2/apache2.conf
# sed -i 's/wheezy/jessie/g' /etc/apt/sources.list
# apt-get update
# apt-get install apache2
# diff -Nrwu /etc/apache2/apache2.conf /etc/apache2/apache2.conf.dpkg-dist 
|diffstat
 apache2.conf |  220 +++
 1 file changed, 86 insertions(+), 134 deletions(-)

bonus points: apache does not start anymore until I resolve the diff.

Yes, this is an extreme example.
Yes, there is /etc/apache2/conf-enabled/ (which changed from
/etc/apache2/conf.d in wheezy).

In the end, on both systems some config management will fix it for me.

Cheers
Evgeni

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Thu, 27 Apr 2017 09:18:54 +0200, Wouter Verhelst
 wrote:
>I didn't say RPM *doesn't* deal with changed files; I said ours deals
>with it better. I stand by that.

Given that dpkg's conffile handling still has much headroom for
improvement, this is a rather sad statement for the rpm half of the
world.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Russ Allbery]

Brian May  writes:
> On 2017-04-27 16:19, Andrey Rahmatullin wrote:

>> It seems you've missed the point (which was about 4 years between RHEL
>> releases).

> There was almost three years between Woody (July 19th 2002) and Sarge
> (June 6th 2005), yet we still allowed upgrades from Woody to Sarge.

> The time duration is irrelevant. It is the policy we have that we support
> and test upgrades that matters. It is much easier to ignore upgrades and
> recommend to reinstall from scratch, that means we don't need to test and
> debug why upgrades break under various corner cases. Not so good for our
> users however.

Yup, this.  It works because we support it, test it, treat bugs in the
upgrade process as critical, and take it into account in our release
engineering.  It's a lot of work.

Red Hat has chosen not to do that work, so they don't support it.  It's a
tradeoff decision.

-- 
Russ Allbery (r...@debian.org)   

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Wed, Apr 26, 2017 at 07:53:57AM -0400, Tom H wrote:
> On Sun, Apr 23, 2017 at 3:00 PM, Wouter Verhelst  wrote:
> >
> > The "packages drop files in /usr/*, sysadmins override in /etc" way of
> > doing things is prevalent in the RPM world; in Debian, however, we
> > traditionally have packages drop files in /etc, and let the maintainer
> > change them in place. This is possible, because our package management
> > system deals better with changed files than does RPM (which must work
> > silently, rather than confirming things with the user).
> 
> s/package management system deals better/package management system
> deals differently/
> 
> rpm doesn't have a problem with config file handling and deals with
> config files in a similar way that dpkg uses the "conffile" attribute
> to deal with them. rpm spec files use two (one-and-a-half?) macros:
> 
> - "%config": "foo.conf" is replaced in an upgrade and saved as
> "foo.conf.rpmsave";
> 
> - "%config(noreplace)": "foo.conf" isn't replaced in an upgrade and
> the new "foo.conf" is installed as "foo.conf.rpmnew".

Yes, I am aware of that (many of my customers use RedHat systems).
However, you will notice the complete absense of a "merge" option in the
above. This means that new configuration files are dropped on the
system, *without* any active notification to the user, so it's up to you
to figure out that this has happened and that you may have work to do.

I didn't say RPM *doesn't* deal with changed files; I said ours deals
with it better. I stand by that.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Wed, Apr 26, 2017 at 08:05:10AM -0400, Tom H wrote:
> On Mon, Apr 24, 2017 at 9:10 AM, Adam Borowski  wrote:
> >
> > All of this is caused by Red Hat having no support for upgrades:
> >
> > https://access.redhat.com/solutions/21964
> >
> > # Red Hat does not support in-place upgrades between major versions 4, 5 and
> > # 6 of Red Hat Enterprise Linux. (A major version is denoted by a whole
> > # number version change. For example, Red Hat Enterprise Linux 5 and Red
> > # Hat Enterprise Linux 6 are both major versions of Red Hat Enterprise
> > # Linux).
> > #
> > # In-place upgrades across major releases do not preserve all system
> > # settings, services or custom configurations. Consequently, Red Hat
> > # strongly recommends fresh installations when upgrading from one major
> > # version to another.
> >
> > # Red Hat currently supports only upgrades from Red Hat Enterprise Linux 6
> > # to Red Hat Enterprise Linux 7 for specific/targeted use cases only.
> >
> > On the other hand, being able to effortlessly dist-upgrade is one of biggest
> > reasons many of us have chosen Debian.
> 
> The reason that you can't dist-upgrade RHEL is that there's too large
> a gap between releases.

No, that's not true. RedHat has never supported upgrades like that, even
before they switched to their "Enterprise Linux" stuff and took years
between releases.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[frank paulsen]

Andrey Rahmatullin  writes:

> On Thu, Apr 27, 2017 at 02:12:25AM +0200, Luca Capello wrote:
>> 
>> And, in any case, for your example on Debian you will do 6 -> 7 -> 8.
> It seems you've missed the point (which was about 4 years between RHEL
> releases).

there were three years between Woody and Sarge.  what is the critical
point for dist-upgrades, three years and a half?

-- 
frobnicate foo

Re: policy for shipping sysctl.d snippets in packages?

[Brian May]

On 2017-04-27 16:19, Andrey Rahmatullin wrote:


It seems you've missed the point (which was about 4 years between RHEL
releases).


There was almost three years between Woody (July 19th 2002) and Sarge 
(June 6th 2005), yet we still allowed upgrades from Woody to Sarge.


The time duration is irrelevant. It is the policy we have that we 
support and test upgrades that matters. It is much easier to ignore 
upgrades and recommend to reinstall from scratch, that means we don't 
need to test and debug why upgrades break under various corner cases. 
Not so good for our users however.

Re: policy for shipping sysctl.d snippets in packages?

[Andrey Rahmatullin]

On Thu, Apr 27, 2017 at 02:12:25AM +0200, Luca Capello wrote:
> Hi there,
> 
> On Wed, 26 Apr 2017 08:05:10 -0400, Tom H wrote:
> > The reason that you can't dist-upgrade RHEL is that there's too large
> > a gap between releases.
> [...]
> > You can't dist-upgrade RHEL from 6 to 7 and you can't dist-upgrade
> > Debian from 6 to 8 in one leap.
> 
> Debian *does* support dist-upgrading between *following* releases only,
> which is what we are talking about.
> 
> And, in any case, for your example on Debian you will do 6 -> 7 -> 8.
It seems you've missed the point (which was about 4 years between RHEL
releases).


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: policy for shipping sysctl.d snippets in packages?

[Luca Capello]

Hi there,

On Wed, 26 Apr 2017 08:05:10 -0400, Tom H wrote:
> The reason that you can't dist-upgrade RHEL is that there's too large
> a gap between releases.
[...]
> You can't dist-upgrade RHEL from 6 to 7 and you can't dist-upgrade
> Debian from 6 to 8 in one leap.

Debian *does* support dist-upgrading between *following* releases only,
which is what we are talking about.

And, in any case, for your example on Debian you will do 6 -> 7 -> 8.

Thx, bye,
Gismo / Luca


signature.asc
Description: Digital signature

Re: policy for shipping sysctl.d snippets in packages?

[Andrey Rahmatullin]

On Wed, Apr 26, 2017 at 07:55:48PM +0100, Ben Hutchings wrote:
> > rpm doesn't have a problem with config file handling and deals with
> > config files in a similar way that dpkg uses the "conffile" attribute
> > to deal with them. rpm spec files use two (one-and-a-half?) macros:
> > 
> > - "%config": "foo.conf" is replaced in an upgrade and saved as
> > "foo.conf.rpmsave";
> > 
> > - "%config(noreplace)": "foo.conf" isn't replaced in an upgrade and
> > the new "foo.conf" is installed as "foo.conf.rpmnew".
> 
> I didn't know about this, and I'm pleased to see that this is (now)
> possible. 
... for at least 15 years (I just don't have earlier experience)

> Is this documented somewhere?  (I've never been able to find
> documentation of RPM macros that isn't very old and incomplete.)
https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch09s05s03.html

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: policy for shipping sysctl.d snippets in packages?

[Ben Hutchings]

On Wed, 2017-04-26 at 07:53 -0400, Tom H wrote:
> > On Sun, Apr 23, 2017 at 3:00 PM, Wouter Verhelst  wrote:
> > 
> > The "packages drop files in /usr/*, sysadmins override in /etc" way of
> > doing things is prevalent in the RPM world; in Debian, however, we
> > traditionally have packages drop files in /etc, and let the maintainer
> > change them in place. This is possible, because our package management
> > system deals better with changed files than does RPM (which must work
> > silently, rather than confirming things with the user).
> 
> s/package management system deals better/package management system
> deals differently/
> 
> rpm doesn't have a problem with config file handling and deals with
> config files in a similar way that dpkg uses the "conffile" attribute
> to deal with them. rpm spec files use two (one-and-a-half?) macros:
> 
> - "%config": "foo.conf" is replaced in an upgrade and saved as
> "foo.conf.rpmsave";
> 
> - "%config(noreplace)": "foo.conf" isn't replaced in an upgrade and
> the new "foo.conf" is installed as "foo.conf.rpmnew".

I didn't know about this, and I'm pleased to see that this is (now)
possible.  Is this documented somewhere?  (I've never been able to find
documentation of RPM macros that isn't very old and incomplete.)

> So rpm isn't a factor;

It is, because rpm is non-interactive and the above choice has to be
made by the packager and not the adminsitrator.

Ben.

> upstreams drop files into "/usr/lib" because
> Red Hat is pushing the concept of all/almost-all distro-provided files
> in "/usr".
> 
> [OT: If I've *had* a complaint about
> "/usr/lib/{modules-load.d,sysctl.d,tmpfiles.d}", it's that, when I
> first looked for them on Debian, I expected them to be under "/lib"
> and not "/usr/lib" given that systemd installs its boot-time files
> under "/lib/systemd".]

-- 
Ben Hutchings
All extremists should be taken out and shot.



signature.asc
Description: This is a digitally signed message part

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Mon, Apr 24, 2017 at 9:10 AM, Adam Borowski  wrote:
>
> All of this is caused by Red Hat having no support for upgrades:
>
> https://access.redhat.com/solutions/21964
>
> # Red Hat does not support in-place upgrades between major versions 4, 5 and
> # 6 of Red Hat Enterprise Linux. (A major version is denoted by a whole
> # number version change. For example, Red Hat Enterprise Linux 5 and Red
> # Hat Enterprise Linux 6 are both major versions of Red Hat Enterprise
> # Linux).
> #
> # In-place upgrades across major releases do not preserve all system
> # settings, services or custom configurations. Consequently, Red Hat
> # strongly recommends fresh installations when upgrading from one major
> # version to another.
>
> # Red Hat currently supports only upgrades from Red Hat Enterprise Linux 6
> # to Red Hat Enterprise Linux 7 for specific/targeted use cases only.
>
> On the other hand, being able to effortlessly dist-upgrade is one of biggest
> reasons many of us have chosen Debian.

The reason that you can't dist-upgrade RHEL is that there's too large
a gap between releases.

Let's look at the release dates and compare like with like.

RHEL 6: November 2010
RHEL 7: June 2014

Debian 6 (squeeze): February 2011
Debian 7 (wheezy): May 2013
Debian 8 (jessie): April 2015

You can't dist-upgrade RHEL from 6 to 7 and you can't dist-upgrade
Debian from 6 to 8 in one leap.

Re: policy for shipping sysctl.d snippets in packages?

[Tom H]

On Sun, Apr 23, 2017 at 3:00 PM, Wouter Verhelst  wrote:
>
> The "packages drop files in /usr/*, sysadmins override in /etc" way of
> doing things is prevalent in the RPM world; in Debian, however, we
> traditionally have packages drop files in /etc, and let the maintainer
> change them in place. This is possible, because our package management
> system deals better with changed files than does RPM (which must work
> silently, rather than confirming things with the user).

s/package management system deals better/package management system
deals differently/

rpm doesn't have a problem with config file handling and deals with
config files in a similar way that dpkg uses the "conffile" attribute
to deal with them. rpm spec files use two (one-and-a-half?) macros:

- "%config": "foo.conf" is replaced in an upgrade and saved as
"foo.conf.rpmsave";

- "%config(noreplace)": "foo.conf" isn't replaced in an upgrade and
the new "foo.conf" is installed as "foo.conf.rpmnew".

So rpm isn't a factor; upstreams drop files into "/usr/lib" because
Red Hat is pushing the concept of all/almost-all distro-provided files
in "/usr".

[OT: If I've *had* a complaint about
"/usr/lib/{modules-load.d,sysctl.d,tmpfiles.d}", it's that, when I
first looked for them on Debian, I expected them to be under "/lib"
and not "/usr/lib" given that systemd installs its boot-time files
under "/lib/systemd".]

Re: policy for shipping sysctl.d snippets in packages?

[Paul Wise]

On Wed, Apr 26, 2017 at 2:52 AM, Christian Seiler wrote:

> But this is a much more general problem. A lot of software in Debian
> ships configuration files in /etc that look like this:
>
> #
> # Setting ABC
> #
> #ABC = 123

I always thought that Debian shipping what is essentially
documentation in /etc is weird. This stuff belongs in the manual pages
or other documentation.

> If in any of these cases the default in the code changes, the behavior
> of the software changes.

Indeed, most default configuration is in one of /usr/*bin /usr/lib
/usr/share and isn't in a form that can be version controlled or
compared between versions.

> In my experience, having to merge configuration files on updates
> has cost me a _lot_ of time.

Especially since most packages don't use ucf, which has 3-way merge.

> Personally, from a user point of view  I really like the "vendor
> defaults in /usr, user configuration in /etc" scheme.

Likewise.

> I consider dpkg's default behavior to be horrible (no copy
> of the original is saved anywhere, so no 3-way merge is
> possible [2]), I never completely grokked ucf as a user (I stumbled
> upon ucf prompts on updates of some packages that used it, and only
> once it actually did manage to do what I wanted automatically, the
> other times I found the way you could manually intervene in the
> merging process there to be highly unintuitive), and I gave up on
> trying to understand ucf from a package maintainer's perspective a
> long time ago.

I've always found ucf's 3-way merge to do exactly what I want.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: policy for shipping sysctl.d snippets in packages?

[Russ Allbery]

Christian Seiler  writes:

> In my experience, having to merge configuration files on updates has
> cost me a _lot_ of time. At the same time I can't recall a single
> instance in the last couple of years where the default behavior of an
> application I used changed and I really wanted to revert that via
> configuration. [1] I'm sure it has happened occasionally, but nothing
> that stuck to my mind.

Yes.  This.

-- 
Russ Allbery (r...@debian.org)   

Re: policy for shipping sysctl.d snippets in packages?

[Christian Seiler]

On 04/25/2017 07:59 PM, Wouter Verhelst wrote:
> In my experience as a user of some of those packages, it is intensely
> annoying that a package upgrade suddenly changes some behaviour in ways
> that I dislike and it's difficult to figure out what changed, because
> the configuration is not in /etc.

But this is a much more general problem. A lot of software in Debian
ships configuration files in /etc that look like this:

#
# Setting ABC
#
#ABC = 123

Where 123 is the default value (in the code of the application), and
in the version of the configuration file there isn't even a single line
that isn't a comment or blank.

Or they install a config file that only contains 3 settings or so, and
the user would have to manually add all the other 60 settings the
software supports if they wanted to change them.

If in any of these cases the default in the code changes, the behavior
of the software changes. Granted, in the first case you might be able
to figure that out if the developers of the software thought to also
update the comments in the config file - but if the software ships
multiple configuration files and the change was in a file you didn't
touch, dpkg will upgrade that file anyway.

Furthermore there are also behavioral changes in software that are not
encoded in configuration, but rather just in code.

And finally, sometimes you don't even configure things, because you're
happy with the defaults, and _then_ a default changes.

So in practice I don't see that as being that big of a benefit here.

In my experience, having to merge configuration files on updates
has cost me a _lot_ of time. At the same time I can't recall a single
instance in the last couple of years where the default behavior of an
application I used changed and I really wanted to revert that via
configuration. [1] I'm sure it has happened occasionally, but nothing
that stuck to my mind.

Personally, from a user point of view  I really like the "vendor
defaults in /usr, user configuration in /etc" scheme. If I look at
current /etc, 99% of the stuff in there is stuff I've never ever
configured or even looked at. I rather like the vision of /etc being
a directory where just my local changes to the system configuration
are stored. Maybe that's influenced by the way I see conffile
handling: I consider dpkg's default behavior to be horrible (no copy
of the original is saved anywhere, so no 3-way merge is
possible [2]), I never completely grokked ucf as a user (I stumbled
upon ucf prompts on updates of some packages that used it, and only
once it actually did manage to do what I wanted automatically, the
other times I found the way you could manually intervene in the
merging process there to be highly unintuitive), and I gave up on
trying to understand ucf from a package maintainer's perspective a
long time ago.

Regards,
Christian

[1] I do recall behavior changing for the worse without a
configuration possibility to revert that though. There's a
reason I packaged gtk3-nocsd. ;-)

[2] Don't get me wrong: I consider RPM's behavior to be worse than
dpkg's, but that doesn't mean I have to like either one.

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Mon, Apr 24, 2017 at 11:17:48AM +0200, Marco d'Itri wrote:
> On Apr 23, Wouter Verhelst  wrote:
> 
> > The "packages drop files in /usr/*, sysadmins override in /etc" way of
> > doing things is prevalent in the RPM world; in Debian, however, we
> > traditionally have packages drop files in /etc, and let the maintainer
>
> While this scheme was probably instigated by limitations in RPM, at this 
> point we have had multiple packages (kmod, systemd, udev for a start) 
> using it for years.

True, never contested that. I merely pointed out that *traditionally*, we've
been doing it differently.

> Moving the sysctl.d default settings to /etc would be:
> - a waste of developers time

There are many other things that are a waste of developers time (dropping
minified javascript, for example) which we still expect our developers to do.
By itself, that is not a good argument.

> - a gratuitous incompatibility with other distributions
> - inconsistent with the documentation both inside and outside Debian

These are good arguments, on the other hand. If upstream implements one
way of doing things, and it is publically documented to be so, then it
is not necessarily a good idea to divert from that just because Debian
traditionally does things in a different way.

However, this question was not about what upstream should do; the
original question was about what package maintainers should do when they
want to configure something not directly related to their own packages
(in casu, a sysctl setting). In that case, it makes sense to look at
what Debian would traditionally do, which here is to ship a file in
/etc, not in /usr.

> - inconsistent with other configuration files implementing this scheme

That is an argument which can equally well be turned around: shipping
configuration in /usr is equally inconsistent with other configuration
files that ship all of their configuration in /etc.

> > There are things to be said to have the whole default configuration live
> > in /etc; IMO, it makes it easier for a system administrator to figure
> > out what the current configuration is, rather than having to mentally
> > merge configuration files from several locations. Additionally, when a
>
> There are also good arguments for having the whole default configuration 
> live in /usr and only local changes in /etc: e.g. this allows supporting 
> systems with an empty /etc, which greatly reduces the administrative 
> time needed to manage a large number of servers/containers.

I disagree that this is a major benefit, but that isn't my point.

> > configuration file that had been edited by a user is now also edited by
> > the package maintainer, on Debian the system will ask how to handle it,
> > rather than changing the defaults and not telling people (which can
> > break in some circumstances).
>
> In my experience as the maintainer of the packages which introduced this 
> scheme this has not been a noticeable problem.

Unfortunately for that argument, SC4 does not state that our priority is
Package Maintainers and Free Software ;-)

In my experience as a user of some of those packages, it is intensely
annoying that a package upgrade suddenly changes some behaviour in ways
that I dislike and it's difficult to figure out what changed, because
the configuration is not in /etc.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: default config not in /etc [was: policy for shipping sysctl.d snippets in packages?]

[Josh Triplett]

Vincent Danjean wrote:
> Perhaps, Debian can try to standardize this (for future releases), for
> example asking to put the default config files in a central place (with
> symlinks if required), for example /etc/default-config or even
> /lib/default-config and/or /usr/lib/default-config.

We cannot unilaterally standardize something like that.  If we want to
push for a standard for package defaults (and I do like the idea of
having a standardized directory under /usr for those rather than just
putting them under /usr/lib/package or similar), we need to work with
upstreams to do so.  /usr/lib/defaults/ does seem plausible.

Re: policy for shipping sysctl.d snippets in packages?

[Josh Triplett]

Adam Borowski wrote:
> On Mon, Apr 24, 2017 at 11:17:48AM +0200, Marco d'Itri wrote:
> > While this scheme was probably instigated by limitations in RPM, at this 
> > point we have had multiple packages (kmod, systemd, udev for a start) 
> > using it for years.
> > 
> > Moving the sysctl.d default settings to /etc would be:
> > - a waste of developers time
> > - a gratuitous incompatibility with other distributions
> > - inconsistent with the documentation both inside and outside Debian
> > - inconsistent with other configuration files implementing this scheme
> 
> And:
> - inconsistent with every package in Debian not in that particular stack

Untrue.  A quick search confirmed a dozen other packages that use the
same approach, most of them in the default desktop install.

> - hard for admins to make edits (one needs to guess this particular scheme
>   is in place and find the file to modify)

Speaking as a sysadmin, I find this approach far easier to work with.
It allows me to *package* my configuration, including both standalone
configuration and overrides of other packages.  Then I simply install
the configuration packages on various systems.

As for "guessing this particular scheme is in place", that's typically
documented directly in the manpage for the configuration file.

> - fails whenever the files in /usr are rearranged

A package should ideally have only one such file, named after the
package.  But in any case, such changes would typically get mentioned in
NEWS.Debian.

> - can't be managed sanely with usual configuration management systems
>   (including raw git)

All of the actual configuration can easily be managed with a
configuration management system, including raw git or etckeeper.  What's
in /usr isn't configuration, it's package defaults.  All configuration
(potentially none) lives in /etc.

> - makes the history of changes done by you vs the package (on a
>   non-overridden file) hard to audit

On a non-overridden file, there are no changes by you, and all of them
come from the package.  And it's *easy* to audit the changes made by
you versus by the package: your changes are in /etc, and the package's
are in /usr.

> All of this is caused by Red Hat having no support for upgrades:

No, this has nothing to do with limitations of Red Hat.  This exists to
make sysadmin's lives easier, and to support new use cases.

> Thus, what about we stop digging the hole and at least forbid this scheme
> for new packages?

No, we should not patch *out* all the support that people have added
upstream; if anything, we should improve more upstreams by adding
support for this mechanism, so that /etc can contain nothing but
sysadmin configuration.  Stateless systems are much easier to work with.

> > There are also good arguments for having the whole default configuration 
> > live in /usr and only local changes in /etc: e.g. this allows supporting 
> > systems with an empty /etc, which greatly reduces the administrative 
> > time needed to manage a large number of servers/containers.
> 
> If all your boxes are identical and get their snowflake number from the net,
> /etc can come from the same place /var and /usr does.  You do need to
> ensure the rest is unmodified anyway.  Shoving things under the carpet
> doesn't help.

What you're referring to here is not the same thing.

> But beside the maintainer hat, do you wear a sysadmin hat sometimes?

That was entirely uncalled for.

Re: policy for shipping sysctl.d snippets in packages?

[Philipp Kern]

On 04/24/2017 08:20 PM, Marc Haber wrote:
> Please note that the current sysctl interface doesn't play well with
> network interfaces that get created on the fly, such as bonding, VLAN
> interfaces or bridges. One needs to first initialize the network, then
> do the sysctl business to catch those interfaces, to catch even the
> dynamically created interfaces.

Hear, hear. Which then means contortions like this:

ACTION=="add", SUBSYSTEM=="net", KERNEL=="wlp*", RUN+="/bin/bash -c
'echo 0 > /proc/sys/net/ipv6/conf/%k/accept_dad'"

I suppose it'd be useful to have a generalized solution to this. :/

Kind regards
Philipp Kern

Re: policy for shipping sysctl.d snippets in packages?

[Christian Seiler]

On 04/24/2017 08:20 PM, Marc Haber wrote:
> Or it was the other way round. I remember going through bizarre
> contortions to set IPv6 ip_forwarding on jessie without
> systemd-networkd supporting this "exotic" use case.

Note that it took a _ton_ of iterations for systemd-networkd to
converge on a behavior that at least most people would consider
sane in the case of IPv6 forwarding. And part of the problem is
that the kernel's knob for IPv4 has very different semantics
from the same knob for IPv6.

In general I believe that the networking part of sysctl kernel
interface is not really that well designed. At lest with Jessie's
kernel (I haven't checked in a while) settings aren't inherited
by a newly created namespace, but instead are the kernel's
default settings - just to give one other example I've stumbled
upon.

Regards,
Christian

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Sun, 23 Apr 2017 21:00:41 +0200, Wouter Verhelst
 wrote:
>The "packages drop files in /usr/*, sysadmins override in /etc" way of
>doing things is prevalent in the RPM world; in Debian, however, we
>traditionally have packages drop files in /etc, and let the maintainer
>change them in place. This is possible, because our package management
>system deals better with changed files than does RPM (which must work
>silently, rather than confirming things with the user). 
>
>The reason both procps and systemd support /usr/* files is presumably
>because they're installed and shipped in both worlds, and it makes
>little sense to patch software to *remove* a feature, even if we end up
>not using it. However, that doesn't mean we should necessarily drop
>files in /usr if we can avoid it.

With the notable exception of systemd, where "doing things like
upstream wants" is feasible because of the good tooling support
(systemctl edit) and the hostile upstream being likely to breath fire
at us for "doing it wrong".

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Marc Haber]

On Sun, 23 Apr 2017 11:11:37 -0700, Josh Triplett
 wrote:
>Henrique de Moraes Holschuh wrote:
>> 1. read current levels (using sysctl, not directly).
>> 
>> 2. if they are above the default, don't change the state of the system:
>>if your config file is there, let ucf handle its update normally.  if
>>your config file is *NOT* there, assume deleted and help ucf a little
>>(ucf can do this by itself most of the time: we have always handled
>>deletion of config files in /etc as an action to be preserved, but
>>*not* at first install)
>> 
>> 3. if they are at a dangerous level, install your config file to /etc
>>normally, using ucf.  And document that the user needs to reboot
>>somewhere.
>
>This seems like a recipe for a very confused sysadmin, wondering what
>non-standard mechanism is messing with sysctls out-of-band and making
>configuration file editing/preservation decisions based on sysctl values
>rather than just the files themselves.
>
>I would suggest, instead, working with sysctl upstream to add an
>"increase if not at least this value, but don't decrease if above this
>value" mechanism.  That'll take a while to propagate widely enough, but
>once it does, you could use such a mechanism to ensure you have a large
>enough value to function without overriding a larger value set in
>another file.
>
>Also, you don't generally need to reboot to apply changed sysctls.

Please note that the current sysctl interface doesn't play well with
network interfaces that get created on the fly, such as bonding, VLAN
interfaces or bridges. One needs to first initialize the network, then
do the sysctl business to catch those interfaces, to catch even the
dynamically created interfaces.


Or it was the other way round. I remember going through bizarre
contortions to set IPv6 ip_forwarding on jessie without
systemd-networkd supporting this "exotic" use case.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: policy for shipping sysctl.d snippets in packages?

[Adam Borowski]

On Mon, Apr 24, 2017 at 11:17:48AM +0200, Marco d'Itri wrote:
> On Apr 23, Wouter Verhelst  wrote:
> 
> > The "packages drop files in /usr/*, sysadmins override in /etc" way of
> > doing things is prevalent in the RPM world; in Debian, however, we
> > traditionally have packages drop files in /etc, and let the maintainer
> While this scheme was probably instigated by limitations in RPM, at this 
> point we have had multiple packages (kmod, systemd, udev for a start) 
> using it for years.
> 
> Moving the sysctl.d default settings to /etc would be:
> - a waste of developers time
> - a gratuitous incompatibility with other distributions
> - inconsistent with the documentation both inside and outside Debian
> - inconsistent with other configuration files implementing this scheme

And:
- inconsistent with every package in Debian not in that particular stack
- hard for admins to make edits (one needs to guess this particular scheme
  is in place and find the file to modify)
- failure to do configuration upgrades; in some cases this can result in
  serious security issues
- fails whenever the files in /usr are rearranged
- can't be managed sanely with usual configuration management systems
  (including raw git)
- makes the history of changes done by you vs the package (on a
  non-overridden file) hard to audit

All of this is caused by Red Hat having no support for upgrades:

https://access.redhat.com/solutions/21964
# Red Hat does not support in-place upgrades between major versions 4, 5 and
# 6 of Red Hat Enterprise Linux.  (A major version is denoted by a whole
# number version change.  For example, Red Hat Enterprise Linux 5 and Red
# Hat Enterprise Linux 6 are both major versions of Red Hat Enterprise
# Linux).
#
# In-place upgrades across major releases do not preserve all system
# settings, services or custom configurations.  Consequently, Red Hat
# strongly recommends fresh installations when upgrading from one major
# version to another.

# Red Hat currently supports only upgrades from Red Hat Enterprise Linux 6
# to Red Hat Enterprise Linux 7 for specific/targeted use cases only.

On the other hand, being able to effortlessly dist-upgrade is one of biggest
reasons many of us have chosen Debian.

Thus, what about we stop digging the hole and at least forbid this scheme
for new packages?
 
> There are also good arguments for having the whole default configuration 
> live in /usr and only local changes in /etc: e.g. this allows supporting 
> systems with an empty /etc, which greatly reduces the administrative 
> time needed to manage a large number of servers/containers.

If all your boxes are identical and get their snowflake number from the net,
/etc can come from the same place /var and /usr does.  You do need to
ensure the rest is unmodified anyway.  Shoving things under the carpet
doesn't help.

> > configuration file that had been edited by a user is now also edited by
> > the package maintainer, on Debian the system will ask how to handle it,
> > rather than changing the defaults and not telling people (which can
> > break in some circumstances).
> In my experience as the maintainer of the packages which introduced this 
> scheme this has not been a noticeable problem.

But beside the maintainer hat, do you wear a sysadmin hat sometimes?


ᛗᛖᛟᚹ.
-- 
⢀⣴⠾⠻⢶⣦⠀ Meow!
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Collisions shmolisions, let's see them find a collision or second
⠈⠳⣄ preimage for double rot13!

Re: policy for shipping sysctl.d snippets in packages?

[Marco d'Itri]

On Apr 23, Wouter Verhelst  wrote:

> The "packages drop files in /usr/*, sysadmins override in /etc" way of
> doing things is prevalent in the RPM world; in Debian, however, we
> traditionally have packages drop files in /etc, and let the maintainer
While this scheme was probably instigated by limitations in RPM, at this 
point we have had multiple packages (kmod, systemd, udev for a start) 
using it for years.

Moving the sysctl.d default settings to /etc would be:
- a waste of developers time
- a gratuitous incompatibility with other distributions
- inconsistent with the documentation both inside and outside Debian
- inconsistent with other configuration files implementing this scheme

> There are things to be said to have the whole default configuration live
> in /etc; IMO, it makes it easier for a system administrator to figure
> out what the current configuration is, rather than having to mentally
> merge configuration files from several locations. Additionally, when a
There are also good arguments for having the whole default configuration 
live in /usr and only local changes in /etc: e.g. this allows supporting 
systems with an empty /etc, which greatly reduces the administrative 
time needed to manage a large number of servers/containers.

> configuration file that had been edited by a user is now also edited by
> the package maintainer, on Debian the system will ask how to handle it,
> rather than changing the defaults and not telling people (which can
> break in some circumstances).
In my experience as the maintainer of the packages which introduced this 
scheme this has not been a noticeable problem.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: policy for shipping sysctl.d snippets in packages?

[Josh Triplett]

On Sun, Apr 23, 2017 at 09:01:13PM +0200, Evgeni Golov wrote:
> On Sun, Apr 23, 2017 at 11:40:34AM -0700, Josh Triplett wrote:
> > On Sun, Apr 23, 2017 at 08:37:59PM +0200, Evgeni Golov wrote:
> > > On Sun, Apr 23, 2017 at 10:48:33AM -0700, Josh Triplett wrote:
> > > > Evgeni Golov wrote:
> > > > > But this does not account for the fact that this specific tunable may 
> > > > > be
> > > > > already overriden in another sysctl.d file and the package would reset
> > > > > it to a lower value?
> > > > 
> > > > You might ask systemd upstream if they'd consider extending the syntax
> > > > to support "increase if below this value but don't decrease".  But in
> > > > the absence of that, I don't think you need to worry about that kind of
> > > > configuration conflict unless it comes up.  Ideally if multiple packages
> > > > need to change this limit, they'll coordinate and agree on the new
> > > > value, or perhaps even depend on a common configuration package.
> > > 
> > > I think such an extension would be quite tricky and probably not worth it.
> > 
> > Can't hurt to ask, given the use case.  Doesn't seem like that much of a
> > challenge to implement; the main challenge would be getting it
> > propagated widely enough to use.
> 
> That was my first thought, too. But then I realized you need operators for:
> - increasing, like above
> - decreasing like for vm.swappiness
> - handling "booleans" and tri-states as used by several security features
>   granted, this is mostly as increasing

I think it'd suffice to add the one thing you currently want, and worry
about other things in the future if they arise.  Increasing works for a
large number of cases.

default config not in /etc [was: policy for shipping sysctl.d snippets in packages?]

[Vincent Danjean]

Le 23/04/2017 à 19:48, Josh Triplett a écrit :
> Evgeni Golov wrote:
>> My gut feeling is that droping the file to /usr/lib and allowing the admin
>> to override it later via /etc. And then load it in postinst.
> 
> That seems like exactly the right approach, and yes, you should put it
> in /usr/lib/50-yourpackagename.conf , following the conventions
> explicitly established for that purpose.  That makes it easy for the
> sysadmin to override *either* by directly creating a file with the same
> name in /etc *or* by adding a file later in the sequence to tweak
> individual settings, both of which can easily be done in packages for
> sysadmins who want to package their configuration tweaks. 

  It is very easy to override, yes. But, once done, it is very difficult
to know that a modification is done in /usr/lib/* that must be adapted
to the overriding /etc/* file.
  For me, this is a major regression over the 'all config is in /etc'
that has been pushed by Debian for a long time.

  Moreover, my /etc is tracked by etckeeper (and this save me several
times). Files in /usr/lib/* with the default configuration are not
(and there is even no way to 'automatically know where they are).
  On several of my systems, I moved /lib/systemd to /etc/something and
created a symlink from /lib/systemd to /etc/something so that
etckeeper track the default configuration on my system. It is a pity
that the same thing should be done more and more now.

  Perhaps, Debian can try to standardize this (for future releases),
for example asking to put the default config files in a central
place (with symlinks if required), for example /etc/default-config
or even /lib/default-config and/or /usr/lib/default-config.
  So we can imagine programs (such as etckeeper) that will track this
(these) directory(ies) and dpkg/apt hooks that can tell the admin
that a default config file has been modified when a override file
as been installed in /etc.

  Regards,
Vincent

-- 
Vincent Danjean   GPG key ID 0xD17897FA vdanj...@debian.org
GPG key fingerprint: 621E 3509 654D D77C 43F5  CA4A F6AE F2AF D178 97FA
Unofficial pkgs: http://moais.imag.fr/membres/vincent.danjean/deb.html
APT repo:  deb http://people.debian.org/~vdanjean/debian unstable main

Re: policy for shipping sysctl.d snippets in packages?

[Evgeni Golov]

On Sun, Apr 23, 2017 at 11:40:34AM -0700, Josh Triplett wrote:
> On Sun, Apr 23, 2017 at 08:37:59PM +0200, Evgeni Golov wrote:
> > On Sun, Apr 23, 2017 at 10:48:33AM -0700, Josh Triplett wrote:
> > > Evgeni Golov wrote:
> > > > But this does not account for the fact that this specific tunable may be
> > > > already overriden in another sysctl.d file and the package would reset
> > > > it to a lower value?
> > > 
> > > You might ask systemd upstream if they'd consider extending the syntax
> > > to support "increase if below this value but don't decrease".  But in
> > > the absence of that, I don't think you need to worry about that kind of
> > > configuration conflict unless it comes up.  Ideally if multiple packages
> > > need to change this limit, they'll coordinate and agree on the new
> > > value, or perhaps even depend on a common configuration package.
> > 
> > I think such an extension would be quite tricky and probably not worth it.
> 
> Can't hurt to ask, given the use case.  Doesn't seem like that much of a
> challenge to implement; the main challenge would be getting it
> propagated widely enough to use.

That was my first thought, too. But then I realized you need operators for:
- increasing, like above
- decreasing like for vm.swappiness
- handling "booleans" and tri-states as used by several security features
  granted, this is mostly as increasing

And you obviously get into funny situations when different files mandate
different change directions.

Re: policy for shipping sysctl.d snippets in packages?

[Wouter Verhelst]

On Sun, Apr 23, 2017 at 12:16:58PM +0200, Evgeni Golov wrote:
> Ohai,
> 
> LXC recently got a bug (#860974) that is best fixed by bumping a certain
> sysctl limit above the default.
> 
> However I could not find any documented policy how to do this (if at all).
> 
> Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
> one package (systemd-coredump) uses it, all others drop files in
> /etc/sysctl.d.

The "packages drop files in /usr/*, sysadmins override in /etc" way of
doing things is prevalent in the RPM world; in Debian, however, we
traditionally have packages drop files in /etc, and let the maintainer
change them in place. This is possible, because our package management
system deals better with changed files than does RPM (which must work
silently, rather than confirming things with the user). 

The reason both procps and systemd support /usr/* files is presumably
because they're installed and shipped in both worlds, and it makes
little sense to patch software to *remove* a feature, even if we end up
not using it. However, that doesn't mean we should necessarily drop
files in /usr if we can avoid it.

There are things to be said to have the whole default configuration live
in /etc; IMO, it makes it easier for a system administrator to figure
out what the current configuration is, rather than having to mentally
merge configuration files from several locations. Additionally, when a
configuration file that had been edited by a user is now also edited by
the package maintainer, on Debian the system will ask how to handle it,
rather than changing the defaults and not telling people (which can
break in some circumstances). In contrast, on an RPM system the system
will just create the new configuration file with a .rpmnew extension,
but will have it not active.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: policy for shipping sysctl.d snippets in packages?

[Josh Triplett]

On Sun, Apr 23, 2017 at 08:37:59PM +0200, Evgeni Golov wrote:
> On Sun, Apr 23, 2017 at 10:48:33AM -0700, Josh Triplett wrote:
> > Evgeni Golov wrote:
> > > But this does not account for the fact that this specific tunable may be
> > > already overriden in another sysctl.d file and the package would reset
> > > it to a lower value?
> > 
> > You might ask systemd upstream if they'd consider extending the syntax
> > to support "increase if below this value but don't decrease".  But in
> > the absence of that, I don't think you need to worry about that kind of
> > configuration conflict unless it comes up.  Ideally if multiple packages
> > need to change this limit, they'll coordinate and agree on the new
> > value, or perhaps even depend on a common configuration package.
> 
> I think such an extension would be quite tricky and probably not worth it.

Can't hurt to ask, given the use case.  Doesn't seem like that much of a
challenge to implement; the main challenge would be getting it
propagated widely enough to use.

Re: policy for shipping sysctl.d snippets in packages?

[Evgeni Golov]

Hi Josh,

On Sun, Apr 23, 2017 at 10:48:33AM -0700, Josh Triplett wrote:
> Evgeni Golov wrote:
> > LXC recently got a bug (#860974) that is best fixed by bumping a certain
> > sysctl limit above the default.
> 
> Note that in addition to going this route, you might consider talking with the
> kernel maintainers about increasing the default limit, or potentially even
> getting the limit raised upstream.

This *specific* issue should go away as soon as we have user-ns aware inotify
counters [1], so it's thankfully just a temporary band-aid.

> > However I could not find any documented policy how to do this (if at all).
> > 
> > Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
> > one package (systemd-coredump) uses it, all others drop files in
> > /etc/sysctl.d.
> > 
> > Some packages also trigger "sysctl -q -p " in their postinst, but
> > most do not.
> > 
> > My gut feeling is that droping the file to /usr/lib and allowing the admin
> > to override it later via /etc. And then load it in postinst.
> 
> That seems like exactly the right approach, and yes, you should put it
> in /usr/lib/50-yourpackagename.conf , following the conventions
> explicitly established for that purpose.

Do you have a pointer to the conventions?
Neither /etc/sysctl.d/README.sysctl, nor sysctl.conf(5) or sysctl.d(5)
have anything on naming, besides "*.conf".

And the current in-archive packages are not telling much either:
% apt-file search sysctl.d
bit-babbler: /etc/sysctl.d/bit-babbler-sysctl.conf
corekeeper: /etc/sysctl.d/corekeeper.conf
dms-core: /etc/sysctl.d/30-dms-core-net.conf
dms-core: /etc/sysctl.d/30-dms-core-shm.conf
freedombox-setup: /etc/sysctl.d/freedombox.conf
linux-grsec-base: /etc/sysctl.d/grsec.conf
postgresql-common: /etc/sysctl.d/30-postgresql-shm.conf
systemd: /etc/sysctl.d/99-sysctl.conf
systemd-coredump: /usr/lib/sysctl.d/50-coredump.conf
tracker-miner-fs: /etc/sysctl.d/30-tracker.conf
uhd-host: /etc/sysctl.d/uhd-usrp2.conf

>  That makes it easy for the
> sysadmin to override *either* by directly creating a file with the same
> name in /etc *or* by adding a file later in the sequence to tweak
> individual settings, both of which can easily be done in packages for
> sysadmins who want to package their configuration tweaks.  (By contrast,
> a configuration package can't safely tweak or override existing files in
> /etc, only drop in a file later in the sequence.)  This also makes it
> easy to use tools to identify any local overrides, or to just look in
> /etc and see at a glance only what the sysadmin has modified, not large
> swathes of stock values.
> 
> Effectively, the bits installed in /usr/lib (or /lib) represent the
> stock state of the package (similar to compiled-in defaults, but more
> easily adjusted by the distribution without patching).
> 
> This is something that people have argued over in the past, and will
> likely continue to argue over; it's also highly correlated with general
> argument over systemd, since systemd is the most popular package
> following and encouraging this configuration pattern, though not the
> first such package.  However, this pattern is supported by upstream
> convention, convention across multiple distributions (meeting sysadmin
> expectations), and existing use by many other packages already in
> Debian.

Agreed.

> > But this does not account for the fact that this specific tunable may be
> > already overriden in another sysctl.d file and the package would reset
> > it to a lower value?
> 
> You might ask systemd upstream if they'd consider extending the syntax
> to support "increase if below this value but don't decrease".  But in
> the absence of that, I don't think you need to worry about that kind of
> configuration conflict unless it comes up.  Ideally if multiple packages
> need to change this limit, they'll coordinate and agree on the new
> value, or perhaps even depend on a common configuration package.

I think such an extension would be quite tricky and probably not worth it.

Thanks
Evgeni

[1] https://patchwork.kernel.org/patch/9370199/

Re: policy for shipping sysctl.d snippets in packages?

[Josh Triplett]

Henrique de Moraes Holschuh wrote:
> 1. read current levels (using sysctl, not directly).
> 
> 2. if they are above the default, don't change the state of the system:
>if your config file is there, let ucf handle its update normally.  if
>your config file is *NOT* there, assume deleted and help ucf a little
>(ucf can do this by itself most of the time: we have always handled
>deletion of config files in /etc as an action to be preserved, but
>*not* at first install)
> 
> 3. if they are at a dangerous level, install your config file to /etc
>normally, using ucf.  And document that the user needs to reboot
>somewhere.

This seems like a recipe for a very confused sysadmin, wondering what
non-standard mechanism is messing with sysctls out-of-band and making
configuration file editing/preservation decisions based on sysctl values
rather than just the files themselves.

I would suggest, instead, working with sysctl upstream to add an
"increase if not at least this value, but don't decrease if above this
value" mechanism.  That'll take a while to propagate widely enough, but
once it does, you could use such a mechanism to ensure you have a large
enough value to function without overriding a larger value set in
another file.

Also, you don't generally need to reboot to apply changed sysctls.

Re: policy for shipping sysctl.d snippets in packages?

[Josh Triplett]

Evgeni Golov wrote:
> LXC recently got a bug (#860974) that is best fixed by bumping a certain
> sysctl limit above the default.

Note that in addition to going this route, you might consider talking with the
kernel maintainers about increasing the default limit, or potentially even
getting the limit raised upstream.

> However I could not find any documented policy how to do this (if at all).
> 
> Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
> one package (systemd-coredump) uses it, all others drop files in
> /etc/sysctl.d.
> 
> Some packages also trigger "sysctl -q -p " in their postinst, but
> most do not.
> 
> My gut feeling is that droping the file to /usr/lib and allowing the admin
> to override it later via /etc. And then load it in postinst.

That seems like exactly the right approach, and yes, you should put it
in /usr/lib/50-yourpackagename.conf , following the conventions
explicitly established for that purpose.  That makes it easy for the
sysadmin to override *either* by directly creating a file with the same
name in /etc *or* by adding a file later in the sequence to tweak
individual settings, both of which can easily be done in packages for
sysadmins who want to package their configuration tweaks.  (By contrast,
a configuration package can't safely tweak or override existing files in
/etc, only drop in a file later in the sequence.)  This also makes it
easy to use tools to identify any local overrides, or to just look in
/etc and see at a glance only what the sysadmin has modified, not large
swathes of stock values.

Effectively, the bits installed in /usr/lib (or /lib) represent the
stock state of the package (similar to compiled-in defaults, but more
easily adjusted by the distribution without patching).

This is something that people have argued over in the past, and will
likely continue to argue over; it's also highly correlated with general
argument over systemd, since systemd is the most popular package
following and encouraging this configuration pattern, though not the
first such package.  However, this pattern is supported by upstream
convention, convention across multiple distributions (meeting sysadmin
expectations), and existing use by many other packages already in
Debian.

> But this does not account for the fact that this specific tunable may be
> already overriden in another sysctl.d file and the package would reset
> it to a lower value?

You might ask systemd upstream if they'd consider extending the syntax
to support "increase if below this value but don't decrease".  But in
the absence of that, I don't think you need to worry about that kind of
configuration conflict unless it comes up.  Ideally if multiple packages
need to change this limit, they'll coordinate and agree on the new
value, or perhaps even depend on a common configuration package.

- Josh Triplett

Re: policy for shipping sysctl.d snippets in packages?

[Evgeni Golov]

On Sun, Apr 23, 2017 at 09:08:45AM -0300, Henrique de Moraes Holschuh wrote:
> On Sun, 23 Apr 2017, Evgeni Golov wrote:
> > Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
> > one package (systemd-coredump) uses it, all others drop files in
> > /etc/sysctl.d.
> 
> Please drop it in /etc, debhelper/dh should mark it as conffile and
> everything will work.

It would, yeah.

> Alternatively, use ufc (refer to ucf(1) and its documentation if you are
> not used to ucf.  Help is also available at debian-mentors@l.d.o), and
> handle it as a configuration file in /etc managed through ucf and
> package maintainer scripts.
> 
> > Some packages also trigger "sysctl -q -p " in their postinst, but
> > most do not.
> 
> What to do here is decided on a case-by-case basis, I suppose.

Sure, but I was looking for some general purpose advise we could include in
our documentation. And IMHO the default should be apply-on-install?

> > My gut feeling is that droping the file to /usr/lib and allowing the admin
> > to override it later via /etc. And then load it in postinst.
> 
> Drop it in /etc where it belongs, and let the maintainer to modify or
> override (by deleting, even).

s/maintainer/admin/ I guess?

> Leave the /usr/lib overriden by /etc thing alone.

Why do we support it then? And document in sysctl.d(5)? Granted, the file is
from src:systemd, but procps also supports it since 5 years [1].

> > But this does not account for the fact that this specific tunable may be
> > already overriden in another sysctl.d file and the package would reset
> > it to a lower value?
> 
> Yes.  If you use ucf instead of the builtin dpkg conffile management,
> you can do something much better:
> 
> 
> 1. read current levels (using sysctl, not directly).
> 
> 2. if they are above the default, don't change the state of the system:
>if your config file is there, let ucf handle its update normally.  if
>your config file is *NOT* there, assume deleted and help ucf a little
>(ucf can do this by itself most of the time: we have always handled
>deletion of config files in /etc as an action to be preserved, but
>*not* at first install)
> 
> 3. if they are at a dangerous level, install your config file to /etc
>normally, using ucf.  And document that the user needs to reboot
>somewhere.
> 
> The above is a rough idea.  You are likely to also have to have
> different paths for initial install and upgrade/downgrade.  And if you
> actually activate the new sysctl, you might not be able to do (1) that
> way should it would break indepondence (and complexity would go up a
> great deal).

That sounds like a nice debhelper addon.
Do you know of any packages already doing this?

Cheers
Evgeni

[1] 
https://gitlab.com/procps-ng/procps/commit/100959274c9d9fa66a570099d4d4a3d6c257fb1f

Re: policy for shipping sysctl.d snippets in packages?

[Henrique de Moraes Holschuh]

On Sun, 23 Apr 2017, Evgeni Golov wrote:
> Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
> one package (systemd-coredump) uses it, all others drop files in
> /etc/sysctl.d.

Please drop it in /etc, debhelper/dh should mark it as conffile and
everything will work.

Alternatively, use ufc (refer to ucf(1) and its documentation if you are
not used to ucf.  Help is also available at debian-mentors@l.d.o), and
handle it as a configuration file in /etc managed through ucf and
package maintainer scripts.

> Some packages also trigger "sysctl -q -p " in their postinst, but
> most do not.

What to do here is decided on a case-by-case basis, I suppose.

> My gut feeling is that droping the file to /usr/lib and allowing the admin
> to override it later via /etc. And then load it in postinst.

Drop it in /etc where it belongs, and let the maintainer to modify or
override (by deleting, even).

Leave the /usr/lib overriden by /etc thing alone.

> But this does not account for the fact that this specific tunable may be
> already overriden in another sysctl.d file and the package would reset
> it to a lower value?

Yes.  If you use ucf instead of the builtin dpkg conffile management,
you can do something much better:


1. read current levels (using sysctl, not directly).

2. if they are above the default, don't change the state of the system:
   if your config file is there, let ucf handle its update normally.  if
   your config file is *NOT* there, assume deleted and help ucf a little
   (ucf can do this by itself most of the time: we have always handled
   deletion of config files in /etc as an action to be preserved, but
   *not* at first install)

3. if they are at a dangerous level, install your config file to /etc
   normally, using ucf.  And document that the user needs to reboot
   somewhere.

The above is a rough idea.  You are likely to also have to have
different paths for initial install and upgrade/downgrade.  And if you
actually activate the new sysctl, you might not be able to do (1) that
way should it would break indepondence (and complexity would go up a
great deal).

-- 
  Henrique Holschuh

policy for shipping sysctl.d snippets in packages?

[Evgeni Golov]

Ohai,

LXC recently got a bug (#860974) that is best fixed by bumping a certain
sysctl limit above the default.

However I could not find any documented policy how to do this (if at all).

Both, procps and systemd support (/usr)?/lib/sysctl.d/*.conf, however only
one package (systemd-coredump) uses it, all others drop files in
/etc/sysctl.d.

Some packages also trigger "sysctl -q -p " in their postinst, but
most do not.

My gut feeling is that droping the file to /usr/lib and allowing the admin
to override it later via /etc. And then load it in postinst.

But this does not account for the fact that this specific tunable may be
already overriden in another sysctl.d file and the package would reset
it to a lower value?

What do others think about that?

Cheers
Evgeni