Re: [Announce] s6.rc: a distribution-friendly init/rc framework (long, off-topic)

[Avery Payne]

>
>  I see that s6.rc comes with a lot of pre-written scripts, from acpid
> to wpa_supplicant. Like Avery's supervision-scripts package, this is
> something that I think goes above and beyond simple "policy": this is
> seriously the beginning of a distribution initiative. I have no wish
> at all to do this outside of a distribution, and am convinced that
> the software base must come first, and then service-specific
> scripts must be written in coordination with distributions that use
> it; that is what I plan to do for s6-frontend in coordination with
> Adélie or Alpine (which are the most likely to use it first). But there
> is a large gray area here: what is "reusable policy" (RP) and what is
> "distribution-specific" (DS)? For instance, look at the way the
> network is brought up - in s6.rc, in OpenRC, in sysvinit scripts,
> in systemd. Is bringing up the network RP or DS? If it involves
> choosing between several external software packages that provide
> equivalent functionality, is it okay to hardcode a dependency, or
> should we provide flexibility (with a complexity cost)?
>
>  This is very much the kind of discussion that I think is important
> to have, that I would like to have in the relatively near future, and
> since more and more people are getting experience with semi-packaging
> external software, and flirting with the line between software
> development and distro integration - be it Olivier, Avery, Casper,
> Jonathan, or others - I think we're now in a good position to have it.
>
>
I'm still thinking this over, especially the distribution-specific
dependencies.  The tl;dr version is, we are really dealing with the
intersection of settings specific to the supervisor, the distribution's
policy (in the form of naming-by-path, environment settings, file
locations, etc), and the options needed for the version of the daemon
used.  If you can account for all three, you should be able to get
consistent run scripts.

The launch of a simple longrun process is nearly (but not entirely)
universal.  What I typically see in > 90% of cases are:

1. designation of the scripting environment in the shebang, to enforce
consistency.
2. clearing and resetting the environment state.
3. if needed, capture STDERR for in-line logging.
4. if needed, running any pre-start programs to create conditions (example:
uuid generation prior to launching udev)
5. if needed, the creation of a run directory at the distribution-approved
location
7. if needed, permission changes to the run directory
6. if needed, ownership changes to the run directory
7. as needed, chain loading helper programs, with dependencies on path
8. as needed, chain loading environment variables
9. specification of the daemon to run, with dependencies on path
10. settings as appropriate for the version of daemon used, with
dependencies on path

The few processes that can't do this, typically have either a design flaw
or a very elaborate launch process.  Either of those require a "special"
run file anyways, so they are already exceptions.

The following issues arise from distribution causing policy to be needed:

* The type of logging used, which can vary quite a bit
* The various names of the supervisory programs
* The path of the daemon itself
* The path of the daemon's settings file and/or directory
* Naming conventions for devices, especially network devices
* How to deal with "instances" of a service
* Handling of failures in the finish file
* Changes in valid option flags between different versions of a daemon

Notice that the first 4 could easily be turned into parameters.  Device
names I don't have an answer for - yet.  Instances are going to be
dependent on the supervisor and system-state mechanism used, and frankly, I
think are beyond the scope of the project.  I don't have an answer for the
finish file at this time because that is a behavior dictated by
distribution policy; it too is outside of scope.  The last one, option
flags, can be overcome by making them a parameter.

The idea that we could turn the bulk of policy into parametric settings
that are isolated away from code is why I have not been as concerned about
separation of policy.  I've been messing around with using two parametric
files + simple macro expansion of the 10 longrun steps listed above to
build the run files as needed.  You would use it like this:

1. You download the source code.
2. You specify a supervisor in a settings file, which in turn provides all
of the correct names for various programs
3. You specify a distribution "environment" in a settings file, which
provides path information, device naming, etc.
4. You run a build process to create all of the run files, which
incorporate the correct values based on the settings from the prior two
files.
5. You run a final build step that installs the files into a "svcdef"
directory which contains all of the definitions ready-to-use; this would
correspond with the s6 approach of a definition directory that does not
contain t

Re: [Announce] s6.rc: a distribution-friendly init/rc framework

[Laurent Bercot]

In their defence, I don't think any mainstream distribution makes this
kind of modifications easy. IMO it's safe to assume a new init system
means a new distribution (possibly derived from something larger).


 And that is why I intend to start with smaller, more flexible,
less inertia-driven distributions such as Alpine and Adélie (and
maybe Void or Manjaro or... someday). Mainstream distributions will
only show interest when the work has already been done somewhere else,
multiple times, and has been proven to be significantly better than
what they already have.



Extra layers generally make things harder to work with, not easier.


 It really depends on the quality of the automation here. If the
abstraction is leaky and you have to look into the lower layer, then
yes, the upper layer is a hindrance more than anything. But if the
abstraction is airtight, as I intend to make it, then people can just
work with the upper layer and not worry about the inner mechanisms.
The inner mechanisms are still there and tweakable for power users,
but the point is to get a smooth, simplified interface on top.



Whoever may be building something new with s6 would probably benefit
more from a reference system that's simple enough to understand
and which could be used as a base for extending into a full blown
distribution. Having examples at hand would likely matter much more
than the format of service description.


 That is what I would naturally lean towards, but it is explicitly
*not* what I was told by distro maintainers. You need to be realistic
here and admit that file-based configuration with key=value lines is
more human-friendly than directory-based configuration. Old habits
die hard, and people who are used to the systemd or openrc way of doing
things just do not want to completely switch their mental model and
deal with something with directories and small files everywhere, and
what the heck is a service directory and why can't you tune your
daemon settings in a config file like everyone else.

 Fortunately, the s6 and s6-rc formats are easy to autogenerate; some
may even argue that they were *purposefully* designed to be easy to
autogenerate. Who knows. So, slapping a UI on top of them is, like,
exactly the kind of thing they're good at, so if that's what people 
want,

there's no reason not to give people what they want.



The reaction to the slew manual I'm afraid will likely be along the
lines of "that's all cool and stuff but how do I actually run this?".


 That is exactly most people's reaction when they first discover the
supervision paradigm. You need to somewhat know how it works to fully
appreciate its quality.

--
 Laurent

Re: [Announce] s6.rc: a distribution-friendly init/rc framework

[Casper Ti. Vector]

On Fri, Mar 23, 2018 at 03:05:53PM +0200, Alex Suykov wrote:
> The reaction to the slew manual I'm afraid will likely be along the
> lines of "that's all cool and stuff but how do I actually run this?".

I again confess that I am not good at writing tutorials; the current
manual is really more suitable for those who know the backgrounds of
s6/s6-rc.  For a reference system, see the attached tarball and online
VM image from the first message in this thread; the only difference
between its system and slew is that all references to `/etc/s6' /
`/etc/s6/lib/s6.rc' should now be `/etc/slew' /
`/etc/slew/lib/slew.rc'.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: [Announce] s6.rc: a distribution-friendly init/rc framework

[Casper Ti. Vector]

On Fri, Mar 23, 2018 at 10:51:57AM +, Laurent Bercot wrote:
>  That's all fine with me, but it may have connotations in English
> that you don't want to associate with a project aimed at stability
> and friendliness :)

See also project names like "git", "curses" and "snort"... :)

>  Bear in mind that - this is a simplified, but descriptive enough view
> of the political landscape of the current Linux ecosystem - distribution
> maintainers are *lazy*.

Now I understand.  What a pity for distro developers / users and us.

> I'm not turning s6 into a monolithic behemoth, don't worry. ;)

I (and probably many others) never worried about this :)

> To provide a reasonably lightweight but immediately usable and
> distro-adoptable init system, there is a golden mean to be found, and I
> think it's important to find it.

Well, slew.rc does not technically require much more than those provided
by what are usually shipped with the base system in common distros,
except for rc(1) and s6/s6-rc/execline.  But it requires the user to be
sufficiently familiar with s6/s6-rc and rc(1), so the main problem is
what you called the current political landscape.

>  But is it the software's job to determine the format of the
> network configuration file, or is it purely the distribution's?
> Is it an advantage for an init system to come with its own
> network configuration format, or a drawback? Is it possible, and
> worth it, to write hooks that call pre-existing network scripts,
> or should the whole network interface config mess be torn down and
> rebuilt from scratch? Those are the exact kind of questions we need
> to ask ourselves.

Good questions, and I certainly need more time to think about them.
I guess that, for the slew.rc-based netifrc replacement I imagined,
assuming it is sufficiently well-designed, it would not be too
difficult to port it to another daemontools-like init/rc framework
according to the underlying logic.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: [Announce] s6.rc: a distribution-friendly init/rc framework

[Alex Suykov]

Fri, Mar 23, 2018 at 10:51:57AM +, Laurent Bercot wrote:

>  Bear in mind that - this is a simplified, but descriptive enough view
> of the political landscape of the current Linux ecosystem - distribution
> maintainers are *lazy*. They already know systemd, or openrc, or
> sysvinit; they don't want to put in more effort. They don't want to have
> to significantly modify their packaging scripts to accommodate a wildly
> different format.

In their defence, I don't think any mainstream distribution makes this
kind of modifications easy. IMO it's safe to assume a new init system
means a new distribution (possibly derived from something larger).

> For instance, instead of having to provide s6-rc source definition
> directories for services, they would provide a descriptive file,
> similar to systemd unit files, that would then be automatically
> processed and turned into a set of s6-rc definition directories.

Extra layers generally make things harder to work with, not easier.

Whoever may be building something new with s6 would probably benefit
more from a reference system that's simple enough to understand
and which could be used as a base for extending into a full blown
distribution. Having examples at hand would likely matter much more
than the format of service description.

The reaction to the slew manual I'm afraid will likely be along the
lines of "that's all cool and stuff but how do I actually run this?".

Re: [Announce] s6.rc: a distribution-friendly init/rc framework

[Laurent Bercot]

What about using "slew.rc" and changing the installation path from
`/etc/s6' to `/etc/slew'?


 That's all fine with me, but it may have connotations in English
that you don't want to associate with a project aimed at stability
and friendliness :)



To be honest, I find the idea not very appealing to me.  I did not want
to say about this before, but now I find it necessary before more 
energy

is spent in goals that I consider less meaningful.  Fortunately, only
`s6-linux-init-maker' is affected by this issue as of now, and I will
use it as an example.

I think the direct downstream of init/rc systems is distributions, not
end users which demand "usability" in the usual sense (actually, one of
the many problems with systemd is that it attempts to bypass distros
and directly provide "unified" features to end users).  After s6/s6-rc
becomes mainstream, `s6-linux-init-maker' will perhaps have, IMHO, an
embarrassing role: most end users would begin with scripts provided by
their distros, which might often be very different from those generated
by `s6-linux-init-maker'.


 Yes, but I don't see this as a problem. The target audience for
s6-frontend would precisely be *distributions*, not end users; the
goal is to factor the work of putting all the s6 pieces together and
make it easier for distributions to tailor those pieces to their own
policies. For instance, instead of having to provide s6-rc source
definition directories for services, they would provide a descriptive
file, similar to systemd unit files, that would then be automatically
processed and turned into a set of s6-rc definition directories. So
they can focus on the functionality they want to achieve in the
service without having to learn the nuts and bolts of the s6
mechanisms.

 Bear in mind that - this is a simplified, but descriptive enough view
of the political landscape of the current Linux ecosystem - distribution
maintainers are *lazy*. They already know systemd, or openrc, or
sysvinit; they don't want to put in more effort. They don't want to have
to significantly modify their packaging scripts to accommodate a wildly
different format. They don't want to have to learn how things operate
under the hood. Ideally, they don't want to have to come up with
policy decisions themselves!
 You may not find the idea of s6-frontend appealing, but distro
maintainers really do. I've had distro maintainers say to me verbatim
that the *main*, and in some cases *only*, obstacle to s6 adoption was
its interface and how different it is from what they already have; but
apart from that, they love the idea.

 Remember that a few years ago I started a quest to learn what kind
of functionality people wanted in s6 in order for them to switch to
it? I expected technical answers, I expected weird systemd features;
I got literally none of those. What I did get, though, is requests
for a friendly UI and accessibility for "average" users (read: lazy
distro maintainers included). This is not the answer I wanted, but
it's the crystal clear answer I got; so it would be foolish of me to
ignore it. I don't like it any more than you do, and interface work
would definitely not be my first choice, but there it is.

 However, I'm absolutely committed to keeping the individual s6
pieces independent and maintaining them separately, and the frontend
will always be optional. So, power users like you will always be able
to use s6-linux-init, s6 and s6-rc separately, with your own scripting
around them; slew.rc or similar initiatives will not be obsoleted.
I'm not turning s6 into a monolithic behemoth, don't worry. ;)



And the difference is not only in the utilities used, but also in the
functionalities offered: eg. conditional (re-)mounting of `/run'
according to results of `mountpoint -q /run', loading/saving the clock,
saving the "uncaught" logs.  This is honestly not to show off the
features provided by my project, but to show that attempts to encapsule
factors in real-world use cases of an init/rc system into a code
generator can be largely unproductive.


 Well, obviously the frontend should offer knobs and settings to
allow customization of those aspects. And in the worst case, there
should be a "run this user-provided script as is" function, which
allows users to bypass the UI and provide snippets of code themselves.



Instead, I think a better way is to provide a full "reference
implementation" of what you have in mind, with the code generation 
knobs

converted into comments (cf. the `devtmpfs' line in `rc.boot' in my
project) or notes in the documentation.


 These two approaches are not mutually exclusive. :)



 I do not consider package dependencies
to be a big problem: distros (and those adventurous users) will
naturally handle them when customising the init/rc system.  No better
way has been proposed to my knowledge: `s6-linux-init-maker' basically
solves it by depending on skaware, and systemd "solves" it by using its
own extremely bloated implementations.


 That

slew: renamed from "s6.rc"

[Casper Ti. Vector]

On Fri, Mar 23, 2018 at 12:00:22PM +0800, Casper Ti. Vector wrote:
> What about using "slew.rc" and changing the installation path from
> `/etc/s6' to `/etc/slew'?  The change is not exactly trivial but already
> much smaller than the `/etc/s6-init' / `/etc/s6-rc' merge I did before
> releasing this project, and can be done in a few hours (the project itself
> is not complex; the complexity on my part comes from the several machines
> where it is delpoyed and has to be migrated :().

Done; new project URL: .
(You will be redirected if you visit the `s6-dot-rc' URL.)

The changes turned out to be very simple: batch changing `/etc/s6' to
`/etc/slew' (including the symlinks in `run/service/.s6-svscan'),
`s6.rc' (filename) to `slew.rc', and `s6.rc' (project name) to `slew';
the rest were cosmetics.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C