Re: The "Unix Philosophy 2020" document

[Avery Payne]

For those readers that meet the following critieria:



   - Are unfortunate enough to only speak a single language, English;
   - And simply want to read an English version of the document;
   - Are (un)fortunately running a current Debian installation with missing
   Latex dependencies;



Do the following:


mkdir -p ~/Projects/ ; cd ~/Projects

git clone https://gitlab.com/CasperVector/up2020.git

cd ./up2020 ; sudo aptitude install latexmk


Edit the first line of the latexmkrc file in the up2020 directory so it
looks like:


@default_files = ('up2020-en');


Then in the up2020 directory, execute this command:


latexmk


The resulting document, while incomplete, will be created as a
English-language PDF named up2020-en.pdf.


Sorry if this added noise to the mailing list.  It was just frustrating to
not have the document build because of missing software dependencies on my
system; doing this tweak allowed me to at least read it.

Re: [Announce] s6.rc: a distribution-friendly init/rc framework (long, off-topic)

[Avery Payne]

>
>  I see that s6.rc comes with a lot of pre-written scripts, from acpid
> to wpa_supplicant. Like Avery's supervision-scripts package, this is
> something that I think goes above and beyond simple "policy": this is
> seriously the beginning of a distribution initiative. I have no wish
> at all to do this outside of a distribution, and am convinced that
> the software base must come first, and then service-specific
> scripts must be written in coordination with distributions that use
> it; that is what I plan to do for s6-frontend in coordination with
> Adélie or Alpine (which are the most likely to use it first). But there
> is a large gray area here: what is "reusable policy" (RP) and what is
> "distribution-specific" (DS)? For instance, look at the way the
> network is brought up - in s6.rc, in OpenRC, in sysvinit scripts,
> in systemd. Is bringing up the network RP or DS? If it involves
> choosing between several external software packages that provide
> equivalent functionality, is it okay to hardcode a dependency, or
> should we provide flexibility (with a complexity cost)?
>
>  This is very much the kind of discussion that I think is important
> to have, that I would like to have in the relatively near future, and
> since more and more people are getting experience with semi-packaging
> external software, and flirting with the line between software
> development and distro integration - be it Olivier, Avery, Casper,
> Jonathan, or others - I think we're now in a good position to have it.
>
>
I'm still thinking this over, especially the distribution-specific
dependencies.  The tl;dr version is, we are really dealing with the
intersection of settings specific to the supervisor, the distribution's
policy (in the form of naming-by-path, environment settings, file
locations, etc), and the options needed for the version of the daemon
used.  If you can account for all three, you should be able to get
consistent run scripts.

The launch of a simple longrun process is nearly (but not entirely)
universal.  What I typically see in > 90% of cases are:

1. designation of the scripting environment in the shebang, to enforce
consistency.
2. clearing and resetting the environment state.
3. if needed, capture STDERR for in-line logging.
4. if needed, running any pre-start programs to create conditions (example:
uuid generation prior to launching udev)
5. if needed, the creation of a run directory at the distribution-approved
location
7. if needed, permission changes to the run directory
6. if needed, ownership changes to the run directory
7. as needed, chain loading helper programs, with dependencies on path
8. as needed, chain loading environment variables
9. specification of the daemon to run, with dependencies on path
10. settings as appropriate for the version of daemon used, with
dependencies on path

The few processes that can't do this, typically have either a design flaw
or a very elaborate launch process.  Either of those require a "special"
run file anyways, so they are already exceptions.

The following issues arise from distribution causing policy to be needed:

* The type of logging used, which can vary quite a bit
* The various names of the supervisory programs
* The path of the daemon itself
* The path of the daemon's settings file and/or directory
* Naming conventions for devices, especially network devices
* How to deal with "instances" of a service
* Handling of failures in the finish file
* Changes in valid option flags between different versions of a daemon

Notice that the first 4 could easily be turned into parameters.  Device
names I don't have an answer for - yet.  Instances are going to be
dependent on the supervisor and system-state mechanism used, and frankly, I
think are beyond the scope of the project.  I don't have an answer for the
finish file at this time because that is a behavior dictated by
distribution policy; it too is outside of scope.  The last one, option
flags, can be overcome by making them a parameter.

The idea that we could turn the bulk of policy into parametric settings
that are isolated away from code is why I have not been as concerned about
separation of policy.  I've been messing around with using two parametric
files + simple macro expansion of the 10 longrun steps listed above to
build the run files as needed.  You would use it like this:

1. You download the source code.
2. You specify a supervisor in a settings file, which in turn provides all
of the correct names for various programs
3. You specify a distribution "environment" in a settings file, which
provides path information, device naming, etc.
4. You run a build process to create all of the run files, which
incorporate the correct values based on the settings from the prior two
files.
5. You run a final build step that installs the files into a "svcdef"
directory which contains all of the definitions ready-to-use; this would
correspond with the s6 approach of a definition directory that does not
contain t

re: Incompatibilities between runit and s6?

[Avery Payne]

I am guessing the differences will be subtle, and most of the general
behavior you desire will remain the same.  You may be able to get a way
with a "sed 's/sv\ /s6-sv\ /' new-script-name" on some of
your scripts; give it a try, what could it hurt?

Also, for those systems not running CentOS, what are you currently using
for init + service management?

runit: is it still maintained and does it have a CVS repository?

[Avery Payne]

I have a slightly older (version 2.1.2) mercurial repository at this
address: https://bitbucket.org/avery_payne/runit

It should not be far behind whatever is "current".  That being said, unless
you have a specific need for runit, s6 & friends are probably the way to go.

[announce] release 0.3.0 of rc-shim

[Avery Payne]

I'm announcing the release of rc-shim v0.3.0, a small script that is useful
for adding supervision to existing installations already using SysV-styled
rc scripts.  The following has changed:

* there is a testing script that ensures the shim will work with your shell.

* confirmation that the shim works with several different shells, outlined
in the README file.

* there is now a "fancy" print variant, that uses colorized printing, etc.
that comes with your OS installation.

The script is in regular use and I have not had any major defects to be
concerned about.  There are a few known issues with output, but these
defects will not affect the operation of the shim.

The quality is now "beta" level and I have decided to continue to make
minor improvements; I retract my former statement that I am headed to a
maintenance release.

Source can be obtained with mercurial or git at
https://bitbucket.org/avery_payne/rc-shim. Incremental improvements will
continue with each release and feedback is always appreciated.

Re: A dumb question

[Avery Payne]

On 5/1/2017 2:11 PM, Francisco Gómez wrote:

And during the process, someone recently told me
something like this.

 "It's old software. Its last version is from 2014. If I have to
 choose between a dynamic, bug-filled init like Systemd and a barely
 maintained init like Runit, I'd rather use Systemd."

That sounds bad, doesn't it?


No, it doesn't sound bad at all, but the person you talked with does 
sound mis-informed.


To quote an (in)famous but well-meaning internet troll, "Functionality 
is an asset, but code is a liability."


Runit is simple enough, and small enough, that it does not require a VCS 
to maintain it.  The same goes for s6, and many other supervision suites.


Now contrast those with systemd: just how big is systemd, including all 
of its pieces?  How many people are needed to maintain it?


Ponder those three sentences, together, for a little while.

[announce] release 0.2.5 of rc-shim

[Avery Payne]

I'm announcing the release of rc-shim v0.2.5, a small script that is useful
for adding supervision to existing installations already using SysV-styled
rc scripts.

This is a very minor release.  The following has changed.

* now includes a primitive test script, useful for debugging shim behavior.

* now confirmed to work on 7 different shells, using the test script.

* fixed a typo with a variable name.

Source can be obtained with mercurial or git at
https://bitbucket.org/avery_payne/rc-shim.  Feedback is always appreciated.

[announce] release 0.2.3 of rc-shim

[Avery Payne]

I'm announcing the release of rc-shim v0.2.3, a small script that is 
useful for adding supervision to existing installations already using 
SysV-styled rc scripts.  At this point the shim appears to be fairly 
stable.  The following has changed:


* README has been expanded.

* supervisor settings have been consolidated.

I am working toward the final release, after which the project will 
enter "maintenance".


Source can be obtained with mercurial or git at 
https://bitbucket.org/avery_payne/rc-shim. Feedback is always appreciated.

[announce] release 0.2.2 of rc-shim

[Avery Payne]

I'm announcing the release of rc-shim v0.2.2, a small script that is 
useful for adding supervision to existing installations already using 
SysV-styled rc scripts.  I was going to withhold this announcement until 
the 0.3 release, but there were some bugs that needed to be addressed.  
The following has changed:


*  Debugging statements have been re-ordered so they fail immediately 
upon detecting a missed setting.


*  The STARTWAIT setting was moved in the script to fix a 
reference-before-declaration bug.


More importantly, the script is no longer considered experimental. It 
appears to be performing its functions consistently and I will be 
extending its testing.  I would rate it as "alpha" quality, meaning that 
some of the settings may be subject to change, but the function of it 
will continue as-is.


Source can be obtained with mercurial or git at 
https://bitbucket.org/avery_payne/rc-shim.  Incremental improvements 
will continue with each release and feedback is always appreciated.

Re: [announce] Release 0.2 of rc-shim

[Avery Payne]

Seems like I'm always forgetting something...

https://bitbucket.org/avery_payne/rc-shim

I need to rewrite the README.  You'll want to edit the settings in 
os-settings and supervisor-settings.


Feedback is always appreciated.  If you have questions, contact me 
outside of the mailing list.


On 11/13/2016 12:52 PM, Jean Louis wrote:

Did you forget the link?

And s6 scripts, just taught me to think myself, so I have adapted
everything to work nicely on my system, and rc-shim should help me
then not to think... :-)

Jean

[announce] Release 0.2 of rc-shim

[Avery Payne]

I'm pleased to announce the release of rc-shim v0.2, a small script that 
is useful for adding supervision to existing installations using 
SysV-styled rc scripts.  The script replaces existing /etc/init.d 
scripts with a shim that interfaces to a supervisor of your choice.  It 
should support any daemontools-alike supervisor.



Since the 0.1 announcement, the following has changed:

* Fixed several bugs in the 0.1 version that affected starting, 
stopping, and reporting status.


* The "reload" option has been removed as it was not compliant with the 
LSB 3.1 standard for arguments accepted by rc scripts.  It has been 
replaced with a stub for "force-reload".  The "force-reload" option 
requires customization to be used correctly, and currently performs a 
no-op.  This is by design.


* The shim header was altered to make it minimally compliant with LSB 
3.1 specifications.  It should allow the shim to work with tools that 
alter runlevel settings.  So far it has been successfully tested with 
Debian's update-rc.d program.


* The shim now correctly sets up and tears down symlinks in the service 
scan directory with each start/stop.


* The shim now has the option to use asynchronous start.  This is a 
trade-off between verification that the supervisor has started, and the 
speed at which the shim processes a start request.  It is disabled by 
default, but can be controlled per-script or system-wide.  Enabling the 
option skips verification in return for speeding up a start request, 
making the assumption that the service scan process will take care of it.


* Added debugging output, which is disabled by default.  This is useful 
during the installation process to confirm that the shim is working 
correctly with your supervisor and daemon.  It is set on a per-script level.



The following limitations still apply:

*  You will need to supply your own supervisor and run scripts for this 
to work.


* The run scripts must be organized into a set of definitions, a set of 
live run directories, and a set of symlinks in a service scan directory.


* The shim only supports starting a single daemon.  If you are replacing 
an rc script that starts multiple daemons, you will need to create a 
custom service scan directory and start that to emulate the behavior.


This script should still be considered experimental.  It continues to 
receive minor testing with a live system.  If you decide to test it, it 
is recommended that you simply rename your existing init.d scripts that 
you are replacing to allow for a rollback, should the shim not function 
correctly for your installation.  Future releases will have additional 
testing and incremental improvements. Suggestions are welcome.

[announce] Release 0.1 of rc-shim

[Avery Payne]

I'm announcing the initial release of rc-shim, a small script that is 
useful for adding supervision to existing installations using 
SysV-styled rc scripts.


The script replaces existing scripts in /etc/init.d with a shim that 
interfaces to an existing supervisor.  It should support any 
daemontools-alike supervisor.  You will need to supply your own run 
scripts for this to work.


The first release should be considered experimental.  It has had only 
minor testing with a live system, and the command set is limited.  
Future releases will have additional testing and incremental 
improvements.  Suggestions are welcome.


The script can be found here:
https://bitbucket.org/avery_payne/rc-shim/src

Re: Runit questions

[Avery Payne]

I hate using webmail.  It always eats your formatting.

The script shows up as a block of text in the mailing list because of this;
just click the link for the project and look for the run-svlogd script.
You'll find what you need there.

On Sun, Oct 16, 2016 at 1:16 PM, Avery Payne 
wrote:

>
>
> On Sat, Oct 15, 2016 at 3:28 PM, Andy Mender 
> wrote:
>
>> Hello everyone,
>>
>> I managed to solve some of my problems. It turned out that my terminal was
>> being spammed
>> with erroneous output, because I didn't add the "exec 2&>1" redirection to
>> my ./run files.
>>
>
> This behavior is fairly consistent among supervisors.  The use of STDERR
> typically is left to the administrator to decide; more commonly, it's
> simply tied off into the STDOUT stream for ease of use.
>
>
>>
>> Another problem popped up on my Devuan box, though. I was trying to add a
>> log directory with a ./run script inside to log dhcpcd per runit's design.
>> However, runsvdir complaints that "exec: not found".
>>
>
> This is probably a problem with your shell.  The shebang specified might
> point to an actual sh and not something like bash, ash, dash, etc.  I don't
> have a reference handy but I suspect that you are seeing failure messages
> for "exec" because the shell doesn't support that built-in.
>
> If you encounter this, change the first line to /bin/bash or /bin/ash or
> /bin/dash and see if that resolves the issue.
>
>
>> The same run scripts work on Gentoo, which makes it even more surprising.
>>
>
> Gentoo may be mapping /bin/sh to bash/ash/something  else, but I don't
> know.  I would suggest finding out.
>
>
>> Below the log/run script:
>> #!/bin/sh
>> exec chpst -u log svlogd -tt ./
>>
>
> I'll post a companion log script in a moment.
>
>
>> PS Earlier runsvdir was complaining that "log: user/group not found". I
>> created the "log" group, thinking it might help somehow.
>>
>
> You shouldn't need to create the group normally, which may indicate a
> separate issue.
>
> Try this run script in your ./svcdef/dbus/log directory.  It's lifted
> straight from the source at http://bitbucket.org/avery_
> payne/supervision-scripts as the ./svcdef/.log/run-svlogd script.  Please
> note that you may need to change the group name after chown, and the shell
> in the first line to meet your needs:
>
> #!/bin/shexec 2>&1# determine our parent's nameSVNAME=$( basename $( echo 
> `pwd` | sed 's/log//' ) )# bring up svlogd logging into a subdirectory that 
> has the parent's name.[ -d /var/log/$SVNAME ] || mkdir -p /var/log/$SVNAME ; 
> chown :adm /var/log/$SVNAME[ -d main ] || ln -s /var/log/$SVNAME mainexec 
> /usr/bin/svlogd -tt main
>
> If you are using Gentoo, I would recommend looking at Toki Clover's
> implementation at https://github.com/tokiclover/supervision.
>
> I hope this helps.
>

Re: Runit questions

[Avery Payne]

On Sat, Oct 15, 2016 at 3:28 PM, Andy Mender 
wrote:

> Hello everyone,
>
> I managed to solve some of my problems. It turned out that my terminal was
> being spammed
> with erroneous output, because I didn't add the "exec 2&>1" redirection to
> my ./run files.
>

This behavior is fairly consistent among supervisors.  The use of STDERR
typically is left to the administrator to decide; more commonly, it's
simply tied off into the STDOUT stream for ease of use.


>
> Another problem popped up on my Devuan box, though. I was trying to add a
> log directory with a ./run script inside to log dhcpcd per runit's design.
> However, runsvdir complaints that "exec: not found".
>

This is probably a problem with your shell.  The shebang specified might
point to an actual sh and not something like bash, ash, dash, etc.  I don't
have a reference handy but I suspect that you are seeing failure messages
for "exec" because the shell doesn't support that built-in.

If you encounter this, change the first line to /bin/bash or /bin/ash or
/bin/dash and see if that resolves the issue.


> The same run scripts work on Gentoo, which makes it even more surprising.
>

Gentoo may be mapping /bin/sh to bash/ash/something  else, but I don't
know.  I would suggest finding out.


> Below the log/run script:
> #!/bin/sh
> exec chpst -u log svlogd -tt ./
>

I'll post a companion log script in a moment.


> PS Earlier runsvdir was complaining that "log: user/group not found". I
> created the "log" group, thinking it might help somehow.
>

You shouldn't need to create the group normally, which may indicate a
separate issue.

Try this run script in your ./svcdef/dbus/log directory.  It's lifted
straight from the source at
http://bitbucket.org/avery_payne/supervision-scripts as the
./svcdef/.log/run-svlogd script.  Please note that you may need to change
the group name after chown, and the shell in the first line to meet your
needs:

#!/bin/shexec 2>&1# determine our parent's nameSVNAME=$( basename $(
echo `pwd` | sed 's/log//' ) )# bring up svlogd logging into a
subdirectory that has the parent's name.[ -d /var/log/$SVNAME ] ||
mkdir -p /var/log/$SVNAME ; chown :adm /var/log/$SVNAME[ -d main ] ||
ln -s /var/log/$SVNAME mainexec /usr/bin/svlogd -tt main

If you are using Gentoo, I would recommend looking at Toki Clover's
implementation at https://github.com/tokiclover/supervision.

I hope this helps.

experiment: making a shim for SysV-styled installs using rc scripts

[Avery Payne]

I know some people will cringe a bit, but hear me out.  There are still
lots of older installs out there, but none of them will give you a true
supervisor when you start a service.  The scripts basically start the
daemon, make a note of its PID, and then assume it's still running.

What if you could keep the existing structure and "service foobar whatever"
scripting, but replace the actual /etc/init.d scripts with a shim, one that
would activate the appropriate daemon and invoke supervision when you
called it?  A few of the existing problems would go away - there would be a
real supervisor and no stale PID files, etc.  In effect, gluing the
supervisor to the existing framework eliminates some of the most glaring
issues.

 I hacked together a little script to try it out.  The issues I've
immediately run into are:

* things like "reload" won't work, because there really isn't a consistent
method of signalling a daemon to "reload its settings".  The closest I can
get is to map "service foobar reload" to having a SIGHUP sent to the daemon
being supervised.

* you still have all the restrictions of the structure of the system, i.e.
6 runlevels, no parallel startup, etc.

I'm still exploring this a little bit.  Would this shim have any real
value, other than to help transition older systems into a full
supervisor-plus-system-state-manager?  Comments and ideas?

Re: Runit questions

[Avery Payne]

On Tue, Oct 11, 2016 at 3:09 PM, Andy Mender 
wrote:

> Hello again,
>
> I'm rewriting some of the standard sysvinit and openrc scripts to ./run
> scripts
>

I would look around a bit.  There are little pockets of pre-written scripts
out there, you just need to dig them up.

Some of the scripts on smarden.org may have minor issues with the daemon
flags they use, so if it doesn't work, go read the man page and compare the
flags in the script to the flags for your installed daemon.


> and I have some problems with dbus. I took the ./run script from Void Linux
> as the original runit documentation doesn't have an exemplary dbus script.
> Whenever I check the status of dbus via "sv status dbus", I get the
> following
> error: "warning: dbus: unable to open supervise/ok: file does not exist".
> This
> makes no sense, as both /etc/sv/dbus/supervise/ and
> /var/service/dbus/supervise/
> contain the "ok" file. Below the run script from Void Linux:
> #!/bin/sh
> [ ! -d /run/dbus ] && install -m755 -g 22 -o 22 -d /run/dbus
> exec dbus-daemon --system --nofork --nopidfile
>

Here is a hacked-up copy of my ./run script.  Be sure to change the
"messagebus" user name after the setuidgid to the proper daemon account for
your installation.  Sorry for the backslash, the word-wrap in the posting
would otherwise kill any formatting.


#!/bin/sh
exec 2>&1
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# must have a valid procfs
mountpoint -q /proc/ || exit 1

# create a unique identifier on each startup
dbus-uuidgen --ensure || exit 1

# start the service
exec pgrphack setuidgid messagebus \
   dbus-daemon --nofork --system --nopidfile



>
> Not sure what's wrong and why this run script needs to contain so many
> operands.
>

The daemon's runtime directory needs to exist before it is launched.  The
first line after the shebang basically does that.

Re: listen(1): proposed addition to the runit suite

[Avery Payne]

On 8/22/2016 6:57 AM, Gerrit Pape wrote:

But beware, directory layout, build and release processes are from
around 2001, and might not be as modern as expected ;).
With just a minor hack, I was able to get a clean build without 
slashpackage interference.  It might not need that much cleanup after all...

Re[2]: error on logging and how to correctly implement svlogd?

[Avery Payne]

Hi,

Thanks for replying. I don't use symlink, instead I put everything 
directly

on /etc/service/test, then sv start test


Try this:

mkdir /etc/svcdef
mkdir /etc/svcdef/test
mkdir /etc/svcdef/test/log

Put a copy of your test service ./run file into the new directory:

cp /etc/service/test/run /etc/svcdef/test/run

Now open an editor like this:

vi /etc/svcdef/test/log/run

and put this into it:

#!/bin/sh
exec 2>&1
# extract the service name
SVNAME=$( basename $( echo `pwd` | sed 's/log//' ) )
# create a logging directory if one isn't present
[ -d /var/log/$SVNAME ] || mkdir -p /var/log/$SVNAME ; chown :adm 
/var/log/$SVNAME

# create a hard-coded path name to reference
[ -d main ] || ln -s /var/log/$SVNAME main
# launch the logger
exec /usr/bin/svlogd -tt main

after that, save the file and exit the editor, and do the following:

mkdir /etc/sv
cp -Rav /etc/svcdef/* /etc/sv/
ln -s /etc/sv /service

Now start your supervision and make sure it's pointing at /service 
instead of /etc/service.  Type


ps fax

...and you should see a supervision "tree" complete with your test 
service and logger.  You don't have to use /etc/svcdef or /etc/sv or 
even /service, I'm just giving these as suggestions.  For that matter 
the logger could even be switched out, the logging done elsewhere, etc.


The logging needs to start using a subdirectory of the service.  In this 
case, the service is /etc/sv/test and the logger would be 
/etc/sv/test/log.  A ./run file needs to be present in the log directory 
to launch the logger, which is the script we just created at 
/etc/svcdef/test/log/run.


Hope this helps.

Re: runit kill runsv

[Avery Payne]

I am try to reproduce situation when runsv under some catastrophic 
failure,

when runsv got killed, it will restart, but my test daemon "memcached"
still running on background, eventually it will start memcached twice. 
How
could I avoid this from happening? Seems fault handling isn't that 
great on

this matter.

It almost sounds like you need to chain-load memcached using chpst.  If 
memcached has internal code to change its process group then it is 
"escaping" supervision, which means that runsv is not in direct control 
of it.  To fix this, your ./run script would be similar to:


#!/bin/sh
exec 2>&1
exec chpst -P memcached

See http://smarden.org/runit/chpst.8.html for details.  This would cause 
memcached to be "captive" to the runsv process.  Try the change with 
chpst and see what happens.  You may find other issues you're not seeing 
after you make this change; check the log with tail -f /path/to/log/file 
and see if it is restarting over and over (a "restart loop").

Re: [DNG] Supervision scripts (was Re: OpenRC and Devuan)

[Avery Payne]

Regarding the use of supervision-scripts as "glue" in distributions, yes,
the project was meant for that. Most - but not all of - the scripts are in
working order, as I use them at home on my personal server.  If you are
willing to take the time to remap the names (as needed), the scripts should
work out-of-the-box.  If you have questions, need help, or get stuck, write
to me and I'll do my best to give you answers.

Currently, these design problems remain:

* account name mappings that correspond to how the names are set up in the
installation,
* hard coded file pathing, which is inconsistent between distributions,
* handling of device names, which are inconsistent between kernels,
* handling of "instances", where one service will be "reused" over and over
(think multiple web servers of the same type on different ports),
* the "versioning problem", which I have (inadequately) described elsewhere
on the mailing list.  The current design addresses this.

My personal life has been very busy, and I needed a break, so there hasn't
been much movement.  Now that things are slowing, I can turn my attention
to it again this summer.  I have a plan to revitalize supervision-scripts
which addresses all (or most) of the listed design problems. The current
plan is:

* come up with clear definitions of the problems,
* a proposal, detailing solutions step by step, which will become a design
document,
* peer review of the document for inaccuracies and errors,
* close out the existing design and archive it,
* announce the old design as version 0.1, for historical interest,
* conversion of the existing design's data into new data structures, and
* finally, writing the code needed to generate the proper ./run files based
on the data provided.

The first step is mostly done.  The second one is just starting.


On Wed, May 4, 2016 at 9:39 AM, Steve Litt 
wrote:

> On Tue, 3 May 2016 22:41:48 -1000
> Joel Roth  wrote:
>
> > We're not the first people to think about supporting
> > alternative init systems. There are collections of the
> > init scripts already available.
> >
> > https://bitbucket.org/avery_payne/supervision-scripts
> > https://github.com/tokiclover/supervision
>
> Those can serve as references and starting points, but I don't think
> either one is complete, and in Avery's case, that can mean you don't
> know whether a given daemon's run script and environment was complete
> or not. In tokiclover's case, that github page implies that the only run
> scripts he had were for the gettys, and that's pretty straightforward
> (and well known) anyway.
>
> As I remember, before he had to put it aside for awhile, Avery was
> working on new ways of testing whether needed daemons (like the
> network) were really functional. That would have been huge.
>
> Another source of daemon startup scripts his here:
>
> https://universe2.us/collector/epoch.conf
>
> SteveT
>
> Steve Litt
> April 2016 featured book: Rapid Learning for the 21st Century
> http://www.troubleshooters.com/rl21
>

unit2run.py, a Python script for converting systemd units

[Avery Payne]

I'm announcing the 0.1 release of unit2run.py, which is a small
brain-damaged hack that attempts to create a ./run file from a systemd
unit.  It is released under the ISC license.

The following features are available:

* It will pump out a horrifically formatted shell script.  It might even be
legible.
* Sometimes the shell scripts actually work, regardless of the supervisor.
Sometimes.

It has the following limitations:

* It only understands about 4 statements, 2 of which do *absolutely nothing*
.
* It laughs at [sections].  Same goes for dependency handling, socket
activation, and other systemd features.  Ha-ha!
* It will eat your kittens and overwrite your existing run files in the
same directory if given the chance, so um, don't do that.
* It currently suffers from "procedural programming".
* Not if but when it breaks, you get to keep all the pieces.

Feel free to download it and have a laugh or two.  Yes, this was written in
my spare time, and it shows.

http://bitbucket.org/avery_payne/unit2run.py

P.S. make sure that there isn't already a file named "run" in the directory
you are running the script in.  You wouldn't like the results, trust me.

A small announcement

[Avery Payne]

The supervision-scripts project isn't dead, although several Monty 
Python fans might notice that it is pining for the fjords.  Perhaps I 
shouldn't have nailed it to the perch...


* As of March 3, 2016, the license has changed to the ISC license. Put 
simply, if you pull the changes as of today's date, you're on the new 
license.  Any pull from the repo prior to today's date is with the old 
license.


* The greeting page has been overhauled with the new project emphasis.  
The remaining pages will also be overhauled as I have time.


* The template system, while still personally used by me, will be 
discarded.  The concepts in it are viable but the arrangement used 
doesn't work well with system managers.  It's fairly sturdy code, but if 
it breaks, you get to keep both pieces.


* Peer-based-daemon-startup will be discarded.

* A new arrangement is being designed that will _generate_ the scripts.  
The details are still being worked out, especially the choice of 
language used for the script generator (which will influence how it can 
be deployed).


* Support for generating default daemon logging will continue to be 
included, but it is considered an option.


* More to come...

Re: The plans for Alpine Linux

[Avery Payne]

On 2/3/2016 9:30 AM, Steve Litt wrote:


Hi Laurent,

The situation you describe, with the maintainer of a distro's
maintainer for a specific daemon packaging every "policy" for every
init system and service manager, is certainly something we're working
toward. But there are obstacles:

* Daemon maintainer might not have the necessary time.

This is probably true.



* Daemon maintainer might not have the necessary knowledge of the init
   system or service manager.
Also true.  To them it's probably "just glue that needs to be there to 
support that thing-a-ma-jig".  Look at some of the Debian init.d scripts 
sometime and notice just how butchered some are.  Some are blatant 
cut-and-paste jobs with the bare minimum to make the launch happen...


I personally feel that the first two points made represent the 
"majority" of cases, although I don't have any proof to back that assertion.




* Daemon maintainer might be one of these guys who figures that the
   only way to start up a system is sysvinit or systemd, and that any
   accommodation, by him, for any other daemon startup, would be giving
   aid and comfort to the enemy.

Not entirely sure that would be the case, but hey, anything is possible.



* Daemon maintainer might be one of these guys who tries to (example)
   Debianize the run script to the point where the beauty and simplicity
   and unique qualities of the service manager are discarded.
Given how Debian works (examples: alternatives, package and repository 
licensing, etc.) I would be surprised if this DIDN'T happen.  Debian 
software really wants one way to deal with multiple choices, so there 
would probably be some kind of glue scripts that would make the 
appropriate calls, etc.


Just my .05 cents.

Re: Suggested Addition to nosh FGA

[Avery Payne]

Unless I am mistaken about what you are asking for, an 
outdated/incomplete list of command equivalents can be found at 
https://bitbucket.org/avery_payne/supervision-scripts/wiki/Comparison


It has not been actively maintained and there are a few gaps in it. 
However, I think most of the information still applies.


On 1/24/2016 10:06 AM, fREW Schmidt wrote:

I find
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/run-scripts-and-service-units-side-by-side.html
to be an incredibly helpful reference.  I think including maybe a
single table of directives/commands that are equivalent would be
useful.  So for example, runsvdir ~~ svscan ~~ s6-svscan.

[announce] supervision-scripts to be retired

[Avery Payne]

Since I began what amounts to my first open source project - ever - I 
have learned a lot in the process, met several interesting characters, 
and hopefully provided some insights to a few others as well.  To 
everyone over the last year and half that have put up with me, thank you 
for giving me an ad-hoc education, being patient with my silly and often 
inane questions, and tolerating some of my goofy responses.


When I started supervision-scripts, I had a vision of a set of ./run 
files that could be used with many different supervisors, in different 
environments.  It was, at the time, an admitted reactive approach to 
dealing with unpleasant actions in the Linux community. I have since 
changed my view and approach to this.


Along the way, I also found that there were issues that would prevent 
both the completion and use of the current scripts on other 
distributions.  One of those problems was the use of different user 
account names; different distributions use their own namings and none of 
them easily align with their peers in any fashion.  Another problem was 
the use of daemons that required "settings after installation", which I 
do not have a current provision for.  To make matters worse, much has 
happened in my personal life that has obstructed me from continuing work 
on the current version of supervision-scripts.  I have given some 
thought to this, and between the current lack of time and the 
constructive criticism received from various parties, I will be unable 
to continue adding new definitions as I had planned.


The existing arrangement I came up with, using indirection and program 
renaming, is viable for installations that use a supervisor only.  At 
first I thought I could incorporate it into system or state management, 
but I now see that will not be possible - yet another design flaw that 
prevents me from reaching the project's goals.  The other issue is the 
embedded user accounts used by various daemons, which currently are all 
Debian-mapped, making the project still intimately tied to Debian when I 
do not want it to be so.


Despite all of those limitations, it has been easy for me to create new 
definitions quickly, and use them for my own purposes at home. In the 
sense that this shows a proof-of-concept, it validates some of my 
assumptions about making portable ./run files.


So, the current project is entering "maintenance".  By this I mean that 
I may occasionally add new definitions to the project but overall, there 
will be no further code changes, and no changes to the structure of how 
it works.  The documentation will be adjusted to reflect this, along 
with the current design flaws.  Once the documentation is complete, the 
project will "retire", rarely updating.


I still believe that process supervision has a future; that for that 
future to become a reality, there needs to be an easy way for 
distributions, packagers, and end-user to incorporate supervision into 
existing arrangements; and that the current, easiest method for doing so 
is to have pre-written definitions that can be quickly installed and 
used.  I am not fully admitting defeat in this process.  This specific 
battle was lost, but this isn't over.


As time goes on, I will take what little spare time I have left and put 
it towards a new design.  The design will be fully implemented "on 
paper" first, and I will ask for peer review in the vain hope that more 
experienced eyes besides my own will be able to discern problems on 
paper before they solidify in code.  This new design will incorporate 
what I have learned from supervision-scripts, but it will take an 
entirely different approach, one that I hope achieves the original 
objective of a portable set of ./run files for supervision.


Until then, I will stay in the background, content to observe.

Re: machine api to supervise

[Avery Payne]

It would be nice to develop a common "grammar" that describes whatever 
is being probed.  If the grammar was universal enough, you could fill in 
empty strings or zeros for things that don't apply, but the interpreter 
for the state data need only be written once.


The following pipe-dream passes for valid json according to 
http://jsoneditoronline.org but some of the data might not even be 
available, so a lot of it is just wishful thinking...


{
  "servicelist": {
"myservice":
{
  "definition":
  {
"type": "s6",
"path": "/path/to/service/definition",
"run": "myservice"
  },
  "state":
  {
"uptime": "54321 seconds",
"last-start": "2015-10-16 18:27:32 PST",
"last-state": "unknown",
"number-restarts": 0,
"wanted": "up",
"currently": "up"
  },
  "env-vars":
  {
"A_STRAY_ENV_VARIABLE": "option-blarg",
"DAEMONSETTING": "--some-daemon-setting",
"OTHEROPTION": "myobscureoptionhere",
"PATH": "/sbin:/usr/sbin:/bin:/usr/bin:/path/to/service/definition"
  },
  "supervisor":
  {
"name": "s6-supervise",
"pid": 123,
"uid": 0,
"gid": 0,
"stdin": 0,
"stdout": 1,
"stderr": 1
  },
  "daemon":
  {
"name": "my-service-daemon",
"pid": 124,
"uid": 101,
"gid": 102,
"stdin": 0,
"stdout": 1,
"stderr": 1
  },
  "logger":
  {
"name": "logwriterprocess",
"pid": 125,
"uid": 99,
"gid": 999,
"stdin": 0,
"stdout": 1,
"stderr": 1
  }
}
  }
}

There should be enough there for the admin to diagnose most issues 
quickly.  I wrapped it in a "servicelist" so multiple definitions could 
be returned.  stdin/stdout/stderr, along with the env settings are 
probably not feasible, but I threw them in anyways.


On 10/16/2015 2:11 PM, Buck Evan wrote:

If you'll specify how you want it to look / work I'll (possibly) attempt to
provide a patch.

`s6-svstat --json` is my initial bid.

I'll try to match the nosh output as closely as is reasonable.

On Thu, Sep 24, 2015 at 4:27 AM, Laurent Bercot 
wrote:
On 24/09/2015 12:56, e-mail j.deboynepollard-newsgroups wrote:


  Unless some daemontools-family
developer goes and does something wrongheaded like breaking compatibility


  I did. There were good reasons for it: the way the bits are arranged in
the legacy daemontools status format is suboptimal, and getting the useful
information out of it requires a little extra work - and I got confused
a few times by the meaning of the bits.

  The status file was never an "official" interface, and the tools that
parse it are actually looking under the hood and breaking abstraction
layers. I'm surprised that you, of all people, advocate this procedure:
this is clearly bad programming practice.


There is something wrong with processing the output of svstat, and you are

right to be leery of doing so. The daemontools manual explicitly states
that it
is human readable, rather than machine-readable. And there are variations
in the
human-readable forms, which are fine for humans but no good for programs.


  Still, svstat is the only "official" interface, so for now it's the only
safe way to proceed. I agree that it is inconvenient, and that daemontools
should have provided a tool to get machine-readable status information;
but in the absence of such a tool, svstat is the entry point and should
be used as such.

  s6 provides a C API to read status: s6_svstatus_read(). The command-line
interface to it is s6-svstat. It's currently lacking options to output
machine-readable data; I plan to add that functionality, but in the
meantime, it *is* the interface.

--
  Laurent

Re: Built-in ordering

[Avery Payne]

Regarding the use of ordering during "stage 1", couldn't you just have a
single definition for stage 1, run through it to set up whatever is needed,
then transition to a new system state that doesn't include that defintion
with (insert system management here)?  What I am trying to ask is if the
down files are really necessary.

Re: Some suggestions about s6 and s6-rc

[Avery Payne]

With regard to having scripted placement of down files, if it was in a
template or compiled as such, then the entire process of writing it into
the definition becomes trivial or moot.  While there should always be a
manual option to override a script, or the option to write one directly, I
think the days of writing all of the definitions needed by hand have long
since past.

But there is an issue that would keep this idea from easily occurring. You
would need to find a way to signal the daemon that it the system is going
down, vs. merely the daemon going down.  I suppose you could argue that the
down and stop commands should be semantically different, and use those to
send the signal, but that's not how they are used today.  Beyond that, if
the placement of the down file was baked into all of the scripts, either by
compiling or templating, then there isn't an issue of repeatedly typing in
support for this.

supervision-scripts 2015-08

[Avery Payne]

Done:
- - - -

+ New definitions: clamd, cpufreqd.

+ Definitions are now versioned.  The ./envdir directory is now a 
symlink to another directory that contains the proper definitions for a 
given version of software.  This solves a long standing problem of 
"version 1 is different from version 2, so using the wrong definition 
breaks deployment", is minimally intrusive, only requires one additional 
symlink and directory in the definition's directory, allows for 
deployment on older systems (with older daemons), doesn't conflict with 
anything that I am aware of (at the moment), and keeps with the 
"filesystem is the database" concept.


+ Cleaned up some old references to /sv that should be /svcdef

+ Revised documentation.  Many concepts are being cleaned up and 
reorganized, including integration into state management systems like 
anopa and s6-rc.



In Progress:
- - - -

+ Renaming some of the internal environment variables to be SIG* instead 
of CMD* to better reflect their nature.


+ Create a test environment with various supervisors on it.  The idea is 
to have a minimal image with several boot options; each boot option 
brings up some combination of supervision + state management, i.e. 
OpenRC + (something), runit, s6 + s6-rc, etc.  So I would select an 
option and boot...and see the results.


+ More definitions.  I've noticed that I've reduced the flow of new 
entries to a trickle since I have had other commitments.  I need to make 
more; the project is nearly at a standstill but a lot of work is still 
needed.  At a minimum, one definition a week should be obtainable; 
ideally, I would like 4 a week.  To the few people who have been 
watching, yes, the project continues, no, it isn't dead.



To Do:
- - - -
Everything.

Re: [ale] systemd talk from July has slide deck online now

[Avery Payne]

On 9/9/2015 9:57 AM, Laurent Bercot wrote:

 Quoting the document: "This article is not meant to impart any technical
judgments, but to simply document what has been done".

 I don't think the author of the document is condoning XML configuration
any more than you and I are. (Context: VR is also the author of uselessd
and he eventually gave up on it because systemd was just too complex to
redesign.)



With regard to the redesign, I've noticed an ongoing method of derailing 
systemd discussion.  I'm only going to digress from the mailing list for 
a brief moment to point out this recurring format:


User1: "I don't like systemd that much because of reason X" (where 
reason X may or may not be a legitimate reason to dislike it)

User2: "Go design a better systemd"

Notice how the discussion is turned back to systemd directly.  It's 
always "go re-invent it", not "make something different".  I just wanted 
to point that out.


P.S. because of the flammable nature of this topic, please, do NOT 
respond to my posting on the mailing list; if you want to respond, send 
an email.

Re: s6: compatibility with make<3.82

[Avery Payne]

Thank you.  Thank you Thank you Thank you.  You just solved a HUGE 
problem for me.


I'm on Debian 7 with my home server, and Debian 7 comes with make < 
4.0.  Debian 8 does come with make >= 4.0, BUT, I do NOT want to update 
to Debian 8 for reasons I won't go into right now. That means I'm stuck 
with a make that won't build s6, unless I download, build, and install a 
separate make, which may or may not conflict with the existing install, 
leading to all kinds of "good times" that I really don't need.  I 
haven't had time to poke around Makefiles, so it's not practical for me 
to address this problem by editing the existing Makefile.  So, in a 
practical sense, there is no s6 deployment on my server until I can 
address this with blood, sweat, tears, and time.


Now with your contribution, I have a fighting chance of building s6 by 
glomming the Makefile into the build process.  That means the 
possibility of a s6 build on my existing install, and not worrying about 
having to do a complete distribution upgrade to Debian 8.


On 8/7/2015 1:50 PM, Buck Evan wrote:

Just to make it more concrete, this is what I had to do to eliminate the
questionable patches from my s6-packaging.

https://github.com/bukzor/s6-packaging/commit/0f3465c074446b73fac05defb6118925b92a3c5a

In essence I vendor and build a gnu-make tarball before starting the s6
build.
I'd prefer not to have to do this, but it's not the worst either.

supervision-scripts 2015-07

[Avery Payne]

Were vacations really meant to be a vacation, or just "time to fix 
things that you haven't had time for"?


Done:
 - - - -

+ New definitions: x2gobroker-authservice, x2gobroker-daemon.

+ Renamed the definition directory from /sv to /svcdef (service 
definitions).  This fixes a namespace issue where /sv could collide with 
existing runit definitions.  It also prepares the project for better 
integration into anopa, rc-s6, and other "state management" frameworks 
by allowing definitions to be kept separate from the runtime 
directories.  For simple installations, you can still just link /service 
to /etc/svcdef and get away with it if you like.  However, the plan also 
allows a copy from a read-only filesystem (/svcdef) to a ramdisk 
(/etc/livedef), useful for embedded devices, "thin" provisioning, 
"shared definitions" on a network, etc. as long as you copy the support 
directories with it (.bin, .run, .log, .finish).


+ Moved peer-level dependency startup into a separate script.

+ Fixed a bug for mapping s6-setsid.


In Progress:
 - - - - - - -

+ Preliminary support for FIFO files.  This is basically the same 
concepts from the state directory setup applied to a FIFO file when needed.


+ Definitions: clamd.


To Do:
 - - - -

+ Finish integrating an old patch that removes the need for awk and 
replaces it with cut.


+ Everything else from last month. Ugh.

Re: supervision-scripts 2015-06

[Avery Payne]

Thanks for all the tips.  I've tried to apply the updates to the wiki 
page as well as I can understand them.  My apologies if I missed 
anything.  You and Laurent both received attribution as well.


On 7/12/2015 2:26 PM, Guillermo wrote:

Re: supervision-scripts 2015-06

[Avery Payne]

On 7/13/2015 2:00 AM, Laurent Bercot wrote:


 There's generally no use for the pgrphack/s6-setsid family of tools.
The mistake of auto-backgrounding is common, so the fghack family
of tools has easily identifiable uses, and it *is* a hack all right;
but I've never seen a daemon misuse sessions and process groups,
probably because those are one of the hairiest parts of Unix, and
the least you hear about them, the better off you are. :)

 That's why I named s6-setsid as it is named: it's not a hack, it has
rare but useful legit uses. It's unlikely you'll find such legit uses
in a run script, however.

My Unix internals knowledge is slightly higher than zero.  Perhaps a 
decimal point and a digit.  So this - specifically process groups - is 
unfamiliar territory for me.  Learn-as-I-go, and all that.  My limited 
understanding was that it would through means not known to me cause all 
child processes to refer back to a common ancestor process (provide that 
the process is still around, which implies it is the parent of all 
others beneath it).  This is probably a completely erroneous assumption 
on my part.


That being said, for whatever reason, using pgrphack fixes a problem I 
had with mingetty as built against Debian 8.  Without pgrphack, mingetty 
would not function; the addition of pgrphack causes it to work 
correctly, and no other changes were needed.  Based on what you're 
saying, it implies that perhaps that my build (aka Debian's build) of 
mingetty is broken?  Or is there some low-level system requirement that 
is being fulfilled, and it should be using pgrphack to begin with?  Or 
is it something else entirely?


Also, thanks for pointing out the use of s6-setsid, which of course 
should be present in the use-s6 script when remapping pgrphack. I'll get 
it fixed as soon as I can.

supervision-scripts 2015-06

[Avery Payne]

I wish I could say I have new definitions but work commitments are just 
now slowing down.  I do have some vacation time in the near future, and 
perhaps a few new definitions will emerge from that.  Right now, some 
housekeeping and documentation is being addressed.


Done:
 - - - -
+ A small comparison table has been added to the wiki, showing 6 
different frameworks and their various "related concepts".  It is 
incomplete but somewhat usable, and contains various links to sites, 
downloads, etc.  Because there is still a lot of missing information, 
any additional hints, comments, or links sent to me are gladly 
accepted.  It can be viewed at 
https://bitbucket.org/avery_payne/supervision-scripts/wiki/Comparison


+ A new document has been started outlining the thought process behind 
the use of template scripts, and what decisions have been made.  It is 
incomplete as well but provides enough information to understand most of 
the structure of the project.  I am ashamed that it can be viewed at 
https://bitbucket.org/avery_payne/supervision-scripts/wiki/Templates ; I 
ask you hold critical comments until it is sufficiently complete.  For 
instance, the sequence in which concepts are introduced needs to be 
revised, as there is a bit of "but first we jump over here to discuss 
this, then come back later".


+ Bugfix: for reasons I haven't delved into, mingetty-tty* wouldn't 
launch. The addition of pgrphack to each definition resolved the issue.


In Progress:
 - - - - - - -
+ Renaming /sv to /svcdef to better reflect the meaning of the "service 
definitions" it contains (here service is loosely used as a meaning for 
"daemon").  This is because some installations will be needing to copy 
definitions into place, rather than linking directly to them.  The new 
name should be fairly agnostic to any existing frameworks or code, and 
removes a potential naming conflict with runit's use of /etc/sv as a 
directory name.


+ Adding support for x2go

To Do:
 - - - -
+ Preliminary support for versioned daemon options, as discussed on the 
mailing list.  I think I have settled on a format that is portable, 
lightweight, self-contained, and still supports "template switching" as 
a feature.  This will solve a few issues I've encountered with porting 
between environments (Environment A uses version 1.23 but Environment B 
uses older version 1.2 which is missing the newer option --foobar, you 
can't have it both ways in one script.)


+ Addition of support for set-up/tear-down of a named socket.  There are 
several definitions that are held up for only this reason. Example: 
clamd needs a socket that must exist prior to launch, or it will fail.  
Separate UID/GID environment variables will be used to convey ownership, 
with "inheritance" from the existing state directory UID/GID as a default.


+ Slight restructure to remove non-critical code from the common 
template script.  Specifically, peer-based dependency launching; yes, 
this idea was based on prior discussion in the mailing list, Thank You, 
you know who you are. :)  This will make "melding" with other 
environments much easier and cleaner.


+ Everything else from last month. Ugh.

Re: Proposal: Versioned Definitions - WAS Re: First thoughts on s6

[Avery Payne]

On 6/25/2015 2:28 PM, post-sysv wrote:
I'm honestly struggling to think of a practical use case for this, 
though I may be lacking sufficient context.
It actually does have a use case, as I've run into it.  (more on this in 
a moment)




The case of a daemon breaking semantics of existing options is quite 
seldom seen, and just a bad practice for reasons of maintaining least 
surprise and compatibility. Even when major versions are bumped, it's 
rare to see such a thing.
Try some of the settings as demonstrated on the samples for runit. Many 
of them will no longer work with the supplied flags because:


1. the option has changed meaning
2. the option has changed its designation, i.e. what was -a is now -b
3. the option was removed entirely

All of these issues persist in those definitions.  While the majority of 
the run scripts published are reasonably well written, they continue to 
"bake in" the settings for "that place and time", which means the above 
3 issues will sooner or later bite your hind-end when you aren't looking.




Now when new options are added, old configurations still tend to run 
the same (unless it's a scenario of this flag now needs that flag set 
in unison, which again is a breaking change).
This *used* to be the case, but with some of the things you see with 
RHEL and others, things tend to rev to the next version "to resolve an 
issue", regardless if reving will break the settings for that daemon.  I 
know this is a less than wonderful approach, but it is what it is, it is 
definitely coming from a commercial entity that provides this as a 
service to customers, and customers may or may not even care, all they 
know is that the upgrade/update broke something and they need to futz 
with it.


The proposal would do away with that.  With each update, you would be 
able to push out the "correct" settings and it would be up to the admin 
to ensure a smooth transition.




By all means just version and update the runscript. Symlink farms are 
always frustrating to deal with. It didn't work well for sysvinit at all.
With regard to symlink use, I think that SysV rc stuff had a problem 
with the creation and destruction of various links in various locations 
as services were sequenced, added, and removed, not to metion the entire 
start vs kill set of scripts.  Symlinks are fine when they are well 
managed, and I think in my not very humble and very uneducated opinion 
that SysV rc scripts were (and still are) not well managed.  I can only 
say that from having to fight with them for the last 5+ years to get 
them to do what I want, so I am no expert on their use, I only claim 
some experience.


Compare:

* there are symlinks created in /service, which are a given for most of 
the supervision frameworks, so it's not like we aren't using this approach.


* supervision-scripts uses symlinks for all of the run script 
definitions, but you never have to move or change them.  You only change 
the one link, located at a known location.


* The proposal would add one more symlink per definition total.

In this case, I'm asking for one additional symlink per definition, 
total.  The name and position of the link will be fixed and it only 
changes when you change what it points at.  There is no dynamic creation 
or desctruction, there is no need for sequencing, there is no need for 
context embedded in the file name - all of which made the SysV approach 
a headache.


Just playing Devil's Advocate here...

Proposal: Versioned Definitions - WAS Re: First thoughts on s6

[Avery Payne]

With yesterday's mention of the idea of a clearinghouse, the concept that
daemon options need to be versioned has been in the back of my head.  Here
is an idea: versioned definitions.  A crude example:

/etc/sv/sshd/envdir/DAEMON -> /etc/sv/sshd/envdir/current/DAEMON
/etc/sv/sshd/envdir/DAEMONOPTS -> /etc/sv/sshd/envdir/current/DAEMONOPTS
/etc/sv/sshd/envdir/current-> /etc/sv/sshd/versions/v3
/etc/sv/sshd/envdir/versions/v3/DAEMON
/etc/sv/sshd/envdir/versions/v3/DAEMONOPTS
/etc/sv/sshd/envdir/versions/v2/DAEMON
/etc/sv/sshd/envdir/versions/v2/DAEMONOPTS


So this was the seed of an idea that borrowed heavily from elsewhere.

The problem:
As software daemons mature, they acquire new options and change their 
behavior.  Because a service definition is tied to launching a version 
of a daemon, that means on any upgrade, the definition has the 
possibility of breakage due to those changes.


The concept:
Settings for daemons should be versioned so that, for a given release of 
a daemon, the "correct" options and settings will be used.


The proposal:
I've left the original rough idea at the top of the message for 
comparison only.  It is not the proposal.


The idea would be that, for a given definition, a ./version directory 
would exist that would house the version-specific information. A soft 
link that normally represents the settings to be used would then point 
to the correct version.  A sample layout appears below.


/etc/svcdef/sshd/env -> ../version/3.81
/etc/svcdef/sshd/version/3.81/
/etc/svcdef/sshd/version/3.81/DAEMON
/etc/svcdef/sshd/version/3.81/DAEMONOPTS
/etc/svcdef/sshd/version/3.81/PRELAUCH
/etc/svcdef/sshd/version/3.8/
/etc/svcdef/sshd/version/3.8/DAEMON
/etc/svcdef/sshd/version/3.8/DAEMONOPTS
/etc/svcdef/sshd/version/3.8/PRELAUNCH

While the definition names are specific to supervision-scripts, the 
layout is not, and could conceivably hold other files and data in them; 
for instance, Toki's supervision framework could store shell script 
settings there instead of envdir files, and the soft link would point to 
the specific script.  In this manner, we can define where and what is 
stored without caring about the how.


Advantages:
* Solves the version-breakage problem.
* The format is data-content-agnostic, it only requires that storage be 
in a file.
* The format can be universally supported with all existing supervision 
frameworks without a naming conflict.
* The format can be installed into legacy supervision arrangements 
without breaking the environment.
* The format is entirely optional.  The proposal is to recognize this as 
a common practice.
* Allows the administrator the possibility of creating custom settings 
instead of using default or system-supplied settings.
* Allows the administrator the possibility of rolling back to a prior 
version without a loss of settings.


Potential issues:
* It requires a filesystem that supports soft links.
* Storage requirements will grow with the number of versioned definitions.
* There is no guarantee that the data format will be consistent.
* For slower systems, some additional disk activity will be required.  
For most modern systems this isn't an issue, but for legacy systems it 
may have a slight impact.
* There may be some suite of programs that will conflict with this 
arrangement.

Re: A general sequence of events for init

[Avery Payne]

On 6/22/2015 6:42 PM, post-sysv wrote:
Handling stages 1 and 3 may need some additions to conditional logic, 
however. 
The idea would be that different plugins would represent some abstract 
notion at some stage in the boot process, i.e. "mount the root 
filesystem" would be abstracted away to a script that was 
correct/specific for the platform it was on, and the init would simply 
"call" the program/script/symlink at a pre-arranged location.  No 
conditional logic needed, just point to the right plugins. :)


Introducing shared objects in this would be overkill though, unless 
you're using the word "plugin" to mean a script using a common interface.


Yes, plugin in this context is an externally callable script or program 
of some kind, called by the init in the correct ordering/sequence.   A 
control script or program would do the coordination and "call" each 
"plugin".


You said there are others with a similar approach, I guess I need to do 
some additional homework.

A general sequence of events for init

[Avery Payne]

I have this crazy dream.  I dream that, for supervision-styled 
frameworks, there will be a unified init sequence.


*  It will not matter what supervision framework you use.  All of them 
will start properly after the init sequence completes.


*  It will not matter how sophisticated your supervision is.  It is 
independent of the features that are provided by the framework.


*  It will not matter if you only have process supervision, or if you 
have something that manages system state fully.  They are independent of 
the init start-up/shutdown.


* It will be scripted in a minimal fashion.  Each stage of the init 
would be a "plugin" called by a "master script".  The plugins would be 
straight-forward, so you could debug it easily.


* It will not matter if you are on Linux or *BSD anymore; the proper 
low-level initialization will take place.  All that would happen is a 
different plugin would be called.


* It would have a system-specific plugin for handling emergencies, so if 
the init fails, you drop into a shell, or reboot, or hang, or do 
whatever it is your heart desires.


I'm really trying to figure out why this can't exist.  What am I missing 
(beyond the shutdown portion)?  I know there will be the whole BSD 
rc-scripts / SysV rc-scripts / OpenRC debate, I'm trying to avoid any of 
those.  I've used BSD-styled scripts years ago on Slackware, and have 
dealth with SysV's crufty stuff recently.  I haven't tried OpenRC yet.

Re: runit maintenance - Re: patch: sv check should wait when svrun is not ready

[Avery Payne]

On 6/18/2015 6:24 PM, Buck Evan wrote:

Thanks Gerrit.

How would you like to see the layout, build process look? Maybe there's an
example project you like?
If it's easy enough for me to try, I'd like to.



I pulled Gerrit's stuff into bitbucket a few days ago.  The first step I 
would humbly suggest, would be to remove the tarballs embedded into the 
repository.Cleaning those out would reduce visual noise when looking 
at the file layout.  Because the entire source history is available, 
there is no reason you can't go back and recreate them on-demand; roll 
back the repo to a prior version and run a makefile outside of it to 
build a tarball.

Re: runit maintenance - Re: patch: sv check should wait when svrun is not ready

[Avery Payne]

On 6/20/2015 3:58 AM, Lasse Kliemann wrote:

Gerrit Pape  writes:


First is moving away from slashpackage.  Doing it similar to what
Laurant does in his current projects sounds good to me, but I didn't
look into details.  This even might include moving away from the djblib
(Public Domain files from daemontools included in runit) to skalibs.

Sorry to interrupt. Why moving away from slashpackage?

As far as I see, Laurent's build system is a super-set up slashpackage
functionality.

I think the whole idea of "registered namespaces for software packages"
is a good idea and should be supported. There are of course different
ways of implementing it.
Why not keep slashpackage as an "alternative" installation method? Is 
there any reason you can't package both "traditional" and "slashpackage" 
methods together?

Re: First thoughts on s6

[Avery Payne]

On 6/18/2015 6:04 AM, Steve Litt wrote:
You or somebody had better document that clearly and simply. 
I'm going to write up a document on the structure and contents of 
definition directories, as used by supervision-scripts.  However, I'm 
going to try and approach it from an evolutionary approach, so that each 
decision is laid out, and the role and purpose of each directory, file, 
and symlink is clear.  With a little luck it might also touch on the 
various aspects of different frameworks, what their expectations are, 
and how they interact with those files.


With regard to the double-symlink trick, that's something I borrowed 
from how I handle switching run scripts.  It was a crude layout but I 
think I can simplify it a bit more and make it less messy, while making 
it neutral to all of the frameworks.  I'm still thinking it over.



I think the difference between those two reactions is about a half a
page of text and maybe a diagram or two. If you incorporate this
system, when the time comes, send me back this email to remind me, and
I'll help with the document.
It will probably be just a markdown document (for expediency) but I'll 
take the same text and push it into the project wiki as well, so it's 
"pretty".




By the way, Avery, that's a slick solution to a real problem. Nice one!
Thanks!  The problem was lurking around in my head but I never gave it 
time, until now.

Re: First thoughts on s6

[Avery Payne]

On 6/17/2015 1:47 PM, Colin Booth wrote:
/usr/libexec/getty This is true for at least freebsd and openbsd. I 
don't have any net, dragonfly, or any of the derivatives to confirm 
but I'd be surprised if they moved their getty location. 

Thank you for the info.

This is actually a different name than the existing 
agetty/fgetty/mingetty that I have, so I can get away with making a new 
definition.  Nice. :)


It goes a bit deeper than that. For example, the maintenance burdon 
for a project like supervision-scripts goes up exponentially as more 
services are added since the maintainer needs to follow every service 
that they are providing scripts for on the off chance some options get 
deprecated or removed.


With yesterday's mention of the idea of a clearinghouse, the concept 
that daemon options need to be versioned has been in the back of my 
head.  Here is an idea: versioned definitions.  A crude example:


/etc/sv/sshd/envdir/DAEMON -> /etc/sv/sshd/envdir/current/DAEMON
/etc/sv/sshd/envdir/DAEMONOPTS -> /etc/sv/sshd/envdir/current/DAEMONOPTS
/etc/sv/sshd/envdir/current-> /etc/sv/sshd/versions/v3
/etc/sv/sshd/envdir/versions/v3/DAEMON
/etc/sv/sshd/envdir/versions/v3/DAEMONOPTS
/etc/sv/sshd/envdir/versions/v2/DAEMON
/etc/sv/sshd/envdir/versions/v2/DAEMONOPTS

Settings are now versioned and encapsulated to their definition; old 
definitions can be kept indefinitely, allowing you to support older 
"legacy" installs; you can still pull just one definition of 
/etc/sv/sshd (and all of the files and directories inside of it) out of 
a tarball, put it in place, and everything works; no changes are needed 
for any run scripts; and if you happen to be running an older version 
(for whatever reason) you can easily switch by changing just the 
"current" file.  Distro maintainers and packagers will not care that 
much, because they'll simply point the ./current symlink at whatever 
version their daemon uses and they're off to the races.


Drawbacks?  File count goes up.  Complexity goes up a smidgeon with the 
concept of "current".  Storage bloats a bit, it seems like symlinks 
require an entire disk block on ext3/ext4.  And yes, the problem of 
getting timely updates for new options still exists.  But it's now a 
write-it-one-time thing.


With sysvinit on Debian, the package maintainer for a given service 
was the person with the onus of making sure that the packaged init 
script was correct, but it's a lot easier to maintain the sshd init 
script if you're also handling packaging of that service as well. With 
systemd, assuming it wins the linux init wars completely, the .service 
file can be shoved farther upstream into the service itself. Yes the 
BSD package and port maintainers will need to still handle their own 
rc files but the maintenance burdon goes down for the packager because 
they no longer need to manage the corresponding .service files.
This begs the question - wouldn't the .service files have the 
corresponding daemon flags in them?  And if so, wouldn't that also imply 
that .service files would have the same level of upkeep and maintenance 
as run scripts, i.e. with a major version change the parameters change 
as well, requiring a new .service?  And wouldn't that also mean that 
they aren't as "portable" as they would sound, because the same 
versioning problem exists?


My theory about why systemd swept through the linux world so fast is 
because it's very attractive to package maintainers since one .service 
file will run everywhere (as opposed to there being a Debian sysvinit 
script, a Gentoo openrc script, an archlinux rc script, an Ubuntu 
upstart script, etc...).
To me service units are/Ash nazg durbatulûk/ but that's a personal 
opinion.  Those .service files seem fishy to me, when Ted T'so had 
problems getting one running.


I don't have any systemd-powered hosts right now, but I'd be highly 
surprised if systemd shipped with every .service file that it supported.

I think you are correct.

If we really want to make process supervisors into first-class 
citizens of the daemon management world, we need to get package 
maintainers excited about run scripts in the same way that they got 
excited about .service files. The goal should be to sit down at a 
supervised system, install a daemon, and have it work under your 
supervisor. 
Agreed.  But we have a chicken-and-egg problem; they won't start the 
process of converting to run scripts until someone shows up to "seed" it 
first.

Re: First thoughts on s6

[Avery Payne]

On 6/16/2015 7:25 PM, Colin Booth wrote:

This not working is part of the curse of the maintainer and why I
think a big collection of definitions that isn't tied to individual
packages is a mistake. In this case for the examples, Laurent went
with assuming getty is in the path (though it'll still fail on BSD
since /usr/libexec isn't in anyones path by default).
Supporting more than just Linux is important to my project, so this is 
of interest to me.  What is the full name of the getty (including path)?


Adding support for this isn't a big issue, so I somewhat disagree with 
your statement that a collection can't work vs everything must be 
included in a package.  For that definition specifically, I can override 
the default $PATH that the scripts use and include /usr/libexec as 
needed; this is by design.  Such a definition is stored as just another 
envdir setting so there really isn't some weird portability workaround 
involved.  At runtime, if the directory or program isn't there then no 
harm, no foul.


That being said, I do see your argument and understand that it would 
make more sense from a systems housekeeping perspective; why have every 
definition present, instead of just the ones installed?

Towards a clearinghouse

[Avery Payne]

On Jun 16, 2015 2:39 PM, "Steve Litt"  wrote:
>
> On Tue, 16 Jun 2015 14:12:48 -0700
> Avery Payne  wrote:
> >
> > In my not very humble opinion, we really need a single point of
> > reference, driven by the community, shared and sharable, and publicly
> > visible.  I could envision something like a single website which
> > would collect settings and store them, and if you needed settings, it
> > would build all of the envdir files and download them in one giant
> > dollop, probably a tarball.  Unpack the tarball and all of the envdir
> > settings are there, waiting to be used.  You could even be "fancy"
> > and track option flags through various daemon revisions, so that if
> > you have an older service running, you tell it "I have older version
> > x.y.z" and you get the "correct" flags and not the "current" ones.
>
> I must be too cynical. I see that after the above described collection,
> website, and envdirs (does that mean service directories) is somewhat
> operational, a well funded "Open Source" vendor will flood it with
> wheelbarrows of cash and a parade of developers, and that nice, simple
> collection and web app becomes unfathomable (and has a 20K word terms
> and conditions).

I think this concept could be made to fly on a shoestring budget.  While
part of me enjoys the illusion of "build it and they will come" the more
rational part of me is well aware that it will probably see less than 1,000
hits a month, 98% from crawlers updating their cache.

With regard to the perinnial issue of "buckets of money sway and manipulate
projects" the answer is simple, Don't Do That (tm) and it will be fine.
This would be a community project with public visibility, and any vendor
attempts to strong-arm it would be spotted fairly quickly.

We need to set the cadence and tempo here and I think having a central
resource would help.   I'm about this close > < to buying a cheap domain
name and then putting it up myself.  Except I know zero about website
design and anything I put up would be ugly.

I'm going to give it some more thought.  I can't promise anything beyond
that.

Re: comparison

[Avery Payne]

On 6/16/2015 1:32 PM, post-sysv wrote:
Soon systemd arrives with its promise of being a unified userspace 
toolkit that systems developers can supposedly just plug in and 
integrate without hassle to get X, Y and Z advantages. No more writing 
initscripts, no more setting policy because systemd will do as much as 
it can for you. A lazy package maintainer's dream, ostensibly.


That last sentence is telling.  It's also why a master catalog of 
settings is so badly needed.


In my not very humble opinion, we really need a single point of 
reference, driven by the community, shared and sharable, and publicly 
visible.  I could envision something like a single website which would 
collect settings and store them, and if you needed settings, it would 
build all of the envdir files and download them in one giant dollop, 
probably a tarball.  Unpack the tarball and all of the envdir settings 
are there, waiting to be used.  You could even be "fancy" and track 
option flags through various daemon revisions, so that if you have an 
older service running, you tell it "I have older version x.y.z" and you 
get the "correct" flags and not the "current" ones.


This service would not only act as a repository but it would also be a 
clearinghouse, a place where distributions could come and take what is 
needed.




Everyone hops on the bandwagon and there you are. Now we get to hear 
about how systemd solves so many long-standing problems with the 
distributions, but I can't shake the feeling that many of them were 
self-inflicted through indifference and/or incompetence.


After learning so much from my own odyssey, I'd like to be a little more 
forgiving and chalk it up to inexperience.  I have this scene running 
through my head as you describe the vendors looking for software nearly 
20 years ago: "We need an init...and something to start services...look, 
there's this Sys-V thingy lying around, grab it and make it work!"

Re: Readiness notification for systemd

[Avery Payne]

On 6/13/2015 11:48 AM, Laurent Bercot wrote:

It's
a wrapper for daemons using the simple "write a newline" readiness
notification mechanism advertised by s6, which converts that
notification to the sd_notify format.


This had me tossing around some ideas yesterday while I was headed home.

Most (but not all) daemons provide logging.

Logging generally (but not always) implies calling printf() with a 
newline at some point.


What if we could come up with a simple standard that extends your 
newline concept into the logging output?  A newline itself may be 
emitted as part of a startup string sent to a log, so I can't be assured 
that a daemon is really ready.  But what if we could ensure that a 
universally agreed pattern were present in the log? Something like 
"Ready.\n" when the daemon is up.  We literally could watch the 
stdout/stderr for this pattern and would solve the entire "readiness 
notification" problem in one go.


It would be an enormous problem politically because most daemon authors 
aren't going to add the 3-4 lines of code needed to support this.


But I could dream...

Re: comparison

[Avery Payne]

On 6/16/2015 5:22 AM, James Powell wrote:

Very true, but something always seems to say something along the lines of "if we had 
done #2 years ago, we might have avoided a huge mess that now exists".

Agreed.

The same applies to init systems. If there are ready to use feet wetting, taste 
testing scripts ready to go, the job of importing things just gets easier on 
the distribution.
Also agreed.  Actually, there's some discussion on the mailing list from 
a few months back about this.


From: Steve Litt
Sent: ‎6/‎16/‎2015 4:45 AM
To: supervision@list.skarnet.org
Subject: Re: comparison

On Tue, 16 Jun 2015 04:05:29 -0700
James Powell  wrote:


I agree Laurent. Though, even though complete init+supervision
systems like Runit exist, it's been nearly impossible to get a
foothold with any alternatives to sysvinit and systemd effectively. I
think one of the major setbacks has been the lack of ready-to-use
script sets, like those included with OpenRC, various rehashes of
sysvinit and bsdinit scripts, and systemd units just aren't there
ready to go.
The true problem is that each daemon needs its own special environment 
variables, command flags, and other gobbledygook that is specific to 
getting it up and running, and a master catalog of all settings doesn't 
exist.  Compounding that is the normal and inevitable need for each 
supervision author to do their own thing, in their own way, so tools get 
renamed, flags get mapped, return codes aren't consistent.  That's just 
the framework, we haven't talked about run scripts yet.  Who wants to 
write hundreds of scripts?  Each hand-cobbled script is an error-prone 
task, and that implies the potential for hundreds of errors, bugs, 
strange behaviors, etc.


This is the _entire_ reason for supervision-scripts.  It was meant to be 
a generic "one size fits most" solution to providing prefabricated run 
scripts, easing or removing the burden for package maintainers, system 
builders, etc.  All of the renaming and flags and options and 
environment settings and other things are abstracted away as variables 
that are correctly set for whatever environment you have.  With all of 
that out of the way, it becomes much easier to actually write scripts to 
launch things under multiple environments.  A single master script 
handles it all, reduces debugging, and can be easily swapped out to 
support chainload launchers from s6 and nosh.


The opposite end of this is Laurent's proposal to compile the scripts so 
they are "built into existence".  If I'm understanding / imagining this 
correctly, this would take all of the settings and with a makefile 
"bake" each script into existence with all of the steps and settings 
needed.  It would in effect provide the same thing I am doing but it 
would make it static to the environment. There's nothing wrong with the 
approach, and the end result is the same.


The only difference between Laurent's approach and mine, is that 
Laurent's would need to "re-bake" your scripts if your framework 
changes; in my project, you simply run a single script and all of the 
needed settings change on the fly.  I'm not sure of the pros/cons to 
either approach, as I would hazard a guess that any system switching 
between frameworks may also require a reboot if a new init is desired.


Here's the rub: in both cases, the settings for each 
service/daemon/whatever are key to getting things running.  Again, we 
come back to the idea of a "master catalog of settings".  If it existed, 
then half of the problem would be resolved.  There are lots of examples 
out there, but, they're not all in one place.


So I try to toil over supervision-scripts when I get time, and make that 
catalog.  Even if people don't like what I'm doing with the master run 
script itself, that doesn't matter.  *What matters is that I've managed 
to capture the settings for the various daemons, along with some 
annotations*.  Because I took the time to support envdir, and the 
settings for each daemon are stored in this format, those settings can 
be extracted and used elsewhere.  I'm slowly creating that "master 
catalog" in a plaintext format that can be read and processed easily.  
This is the real, hidden value of supervision-scripts.


By the way, I'm going to bite the bullet and switch off of MPL 2.0 soon, 
probably by month-end.

Re: patch: sv check should wait when svrun is not ready

[Avery Payne]

I'm not the maintainer of any C code, anywhere.  While I do host a 
mirror or two on bitbucket, I only do humble scripts, sorry.  Gerrit is 
around, he's just a bit elusive.


On 6/16/2015 9:37 AM, Buck Evan wrote:

I'd still like to get this merged.

Avery: are you the current maintainer?
I haven't seen Gerrit Pape on the list.

On Tue, Feb 17, 2015 at 4:49 PM, Buck Evan <mailto:b...@yelp.com>> wrote:


On Tue, Feb 17, 2015 at 4:20 PM, Avery Payne
mailto:avery.p.pa...@gmail.com>> wrote:
>
> On 2/17/2015 11:02 AM, Buck Evan wrote:
>>
>> I think there's only three cases here:
>>
>>  1. Users that would have gotten immediate failure, and no
amount of
>> spinning would help. These users will see their error delayed
by $SVWAIT
>> seconds, but no other difference.
>>  2. Users that would have gotten immediate failure, but could
have gotten
>> a success within $SVWAIT seconds. All of these users will of
course be glad
>> of the change.
>>  3. Users that would not have gotten immediate failure. None of
these
>> users will see the slightest change in behavior.
>>
>> Do you have a particular scenario in mind when you mention
"breaking lots
>> of existing installations elsewhere due to a default behavior
change"? I
>> don't see that there is any case this change would break.


Thanks for the thoughtful reply Avery. My background is also
"maintaining business software", although putting it in those terms
gives me horrific visions of java servlets and soap protocols.

> I have to look at it from a viewpoint of "what is everything
else in the system expecting when this code is called".  This
means thinking in terms of code-as-API, so that calls elsewhere
don't break.

As a matter of API, sv-check does sometimes take up to $SVWAIT
seconds to fail.
Any caller to sv-check will be expecting this (strictly limited)
delay, in the exceptional case.
My patch just extends this existing, documented behavior to the
special case of "unable to open supervise/ok".
The API is unchanged, just the amount of time to return the result
is changed.

> This happens because the use of "sv check (child)" follows the
convention of "check, and either succeed fast or fail fast", ...

Either you're confused about what sv-check does, or I'm confused about
what you're saying.
sv-check generaly doesn't fail fast (except in the special case I'm
trying to make no longer fail fast -- svrun is not started).
Generally it will spin for $SVWAIT seconds before failing.

> Without that fast-fail, the logged hint never occurs; the
sysadmin now has to figure out which of three possible services in
a dependency chain are causing the hang.

Even if I put the above issue aside aside, you wouldn't get a hang,
you'd get the failure message you're familiar with, just several
seconds (default: 7) later. The sysadmin wouldn't search any more than
previously. He would however find that the system fails less often,
since it has that 7 seconds of tolerance now. This is how sv-check
behaves already when a ./check script exits nonzero.


> While this is
> implemented differently from other installations, there are
known cases
> similar to what I am doing, where people have ./run scripts like
this:
>
> #!/bin/sh
> sv check child-service || exit 1
> exec parent-service

This would still work just fine, just strictly more often.

Re: comparison

[Avery Payne]

On 6/15/2015 9:00 PM, Colin Booth wrote:

I only know s6 and runit well enough to comment on for the most part but
filling in some blanks on your matrix:

Updated, thanks for the help.  As I said, it's a start.  It'll need some 
time to improve.  I mostly needed it for the project, to help me keep 
the mapping of "what tool does what action" straight so I can move 
forward.  I'd like to add some of the missing "specialty" tools that s6 
and nosh provides, and see if there are equivalent mappings elsewhere.  
Also, as new scripting frameworks are discovered, I'll add them as well.

Re: comparison

[Avery Payne]

I'm working on something similar, but you're asking for capabilities, 
and most of what I have is a mapping.  I've tried to include a few 
critical links in the comparison for the various homepages, licenses, 
source code, etc.  It's incomplete for now, but it's a start.


https://bitbucket.org/avery_payne/supervision-scripts/wiki/comparison



On 6/15/2015 5:37 PM, Buck Evan wrote:

Is there any resource that compares the capabilities of daemontools,
daemontools-encore, runit, s6, and friends?

Re: dependant services

[Avery Payne]

On 6/8/2015 2:15 PM, Steve Litt wrote:
I'm not familiar with inetd. Using sockets to activate what? In what 
manner? Whose socket?


~ ~ ~
Let's go back in time a little bit.  The year is 1996, I'm downstairs 
literally in my basement with my creaky old 486 with 16Mb of RAM and I'm 
trying to squeeze as much as I can into my Slackware 3.6 install that I 
made with 12 floppy disks.  There are some of these service-thingys that 
I'm learning about and they all take up gobs of expen$ive RAM, and while 
I can swap to disk and deal with that, swapping is a slooow 
afair because a drive that pushes 10 megaBYTES per second is "speedy".  
Heck, my drive isn't even IDE, it's ESDI, and being a full-height 5 1/2" 
drive is actually larger than a brick.  But I digress.  It would be cool 
if there was a way to reduce the RAM consumption...

~ ~ ~

Me: "There's got to be something that can free up some RAM...time to dig 
around documentation and aritcles online with my uber-kool 14.4 dialup 
modem!  Let's see herewhat's this?   Inetd?  Whoa, it frees up RAM 
while providing services!  Now I just need RAM to run inetd and all the 
RAM I save from not running other things can be used for mischief!"


~ ~ ~
What inetd does is:

1. Have a giant list of port numbers defined, with a program that pairs
   with each port number (/etc/inetd.conf)
2. Opens port numbers out of that list when the inetd daemon is run and
   listens to all of them.
3. When someone talks to the port, the corresponding program is
   launched and the port connected to the program.  If the program
   fails to launch, the connection is closed.
4. You only need RAM for inetd + any services that launch.
5. ...
6. Profit!

Meanwhile, in the same year, halfway across the country in Illinois, in 
a dark lab...

~ ~ ~

DJB: (swiveling around in a dramatic swivel chair, but no cat, because 
cats would shed hair on his cool looking sweater) "I shall take the old 
inetd concept, and make it generic and decoupled and streamlined and 
secure.  I shall gift this to you, the Internet, so that you may all be 
secure, unlike Sendmail's Security Exploit of the Month Club which keeps 
arriving in my inbox when I didn't ask for it.  Go forth, and provide 
much joy to sysadmins everywhere!" (queue dramatic music)


~ ~ ~
...and thus, UCSPI was born.  Fast forward to 2014while surfing 
various Linux news articles, I stumble into something that sounds like 
an infomercial...

~ ~ ~

 ...Systemd will now do socket activation with not only file sockets 
but also network sockets too!  NETWORK SOCKETS!  It's like an Armed Bear 
riding a Shark with Frickin' Laser Beams while singing the National 
Anthem with an Exploding Background!!  Get your copy today for THREE 
easy payments!!!  Order Now While Supplies Last OPERATORS ARE 
STANDING BY!!!


~ ~ ~
Yes, that juicy sound is the sound of my eyes rolling up into their 
sockets as I read that article, attempting to retreat to the relative 
safety of my skull as I Cannot Un-see What I Have Seen...as you can 
tell, this isn't exactly a new concept, and it's been done before, many 
many times, in various ways (inetd, xinetd, various flavors of UCSPI, 
and now systemd's flavor).

Re: dependant services

[Avery Payne]

On 6/8/2015 11:55 AM, Laurent Bercot wrote:

On 08/06/2015 16:00, Avery Payne wrote:

This is where I've resisted using sockets.  Not because they are bad
- they are not.  I've resisted because they are difficult to make
100% portable between environments.  Let me explain.


 I have trouble understanding several points of your message.

 - You've resisted using sockets. What does that mean ? A daemon
will, or will not, use a socket; as an integrator, you don't have
much say on the matter.
I'm not specifically speaking about a socket required by a daemon. I'm 
talking about using sockets for activation via UCSPI, similar to the old 
inetd concept.


 - What tools are available. What does that have to do with
daemons using sockets ? UCSPI tools will, or will not, be available,
but daemons will do as they please. If your scripts rely on UCSPI
tools to ease socket management, then add a package dependency -
your scripts need UCSPI tools installed, end of story.
Until now, I have been able to resist adding additional requirements.  
Each tool requirement, from outside the project, looks "lightweight and 
harmless", but you know as well as I do that it isn't; each external 
dependency increases complexity and reduces the chance that you can port 
your code.


I've tried, really, really hard to avoid this when possible.  It 
probably doesn't seem like that from the outside looking in, but that's 
been the intent.  Sometimes my decisions will appear to be silly; this 
is a learning process and as I go, silly decisions have been made, and 
much has been learned.  I don't claim to be an expert in any sense, just 
someone who took the time to work with something.



Dependencies
are not a bad thing per se, they just need to be controlled and
justified.
 "UCSPI sockets" does not make sense. You'll have Unix sockets and
INET sockets, and maybe one or two esoteric things such as netlink.
UCSPI is a framework that helps manipulate sockets with command-line
utilities. Use the tools or don't use them, but I don't understand
what your actual problem is.
The problem is that there isn't an assurance that UCSPI tools are 
/available/ at the point of installation; which means writing a shell 
script to handle using such tools becomes complicated, I can't just 
specifically code in a tool name and expect things to just work out of 
the box.  Compounding that is the lack of UCSPI tools being installed as 
a standard part of a software package, so things like runit may have a 
"standardized set" of tools that I can count on to be there, but not 
Gerrit's version of UCSPI.


I got around this problem with framework tools by abstracting away the 
toolset with symlinks and fall-back behavior, so to do UCSPI properly, I 
would most likely do the same again.  I'm not happy about my decision 
but it works across the widest range of tools and has the "least" 
impact.  It also makes extending support to other frameworks easier via 
abstraction.




 So where do
the sockets live?  /var/run? /run?  /var/sockets?
/insert-my-own-flavor-here?


 How about the service directory of the daemon using the socket ?
That's what a service directory is for.
Also true.  I was just pointing out that it's yet another decision that 
has to be made.  And as I pointed out elsewhere, things like anopa 
require some custom work to wedge the definitions in, complicating the 
process.  Adding UCSPI support just complicates it further.  Call me 
whiny for not wanting to put more effort in if you like; I'll admit to 
it on this specific topic.




* Make socket activate an admin-controlled feature that is disabled
by default.  You want socket activation, you ask for it first. The
admin gets control, I get more headache, and mostly everyone can be
happy.


 If all this fuss is about socket activation, then you can simply
forget it altogether.

Already did. :)



As a side note, I'm beginning to suspect that the desire for "true
parallel startup" is more of a "mirage caused by desire" rather than
by design.

 At least, if by "parallel startup" you mean "start things as soon as
they can be started without risk, without needless waiting times".

The question was along the lines of "sure it's something we can do, but 
does that mean we should do it to begin with?"  There is value to 
starting things quickly when possible, but the parallel start I'm 
talking about is "we can launch 2 or more processes at once and get a 
massive speed gain due to multiple cores" line of thinking.  That kind 
of magic makes sense when you're running a farm of 1,000 web servers; it 
makes zero sense when you have a large NAS/SAN and you're forcing a 
fsck-on-reboot as a default behavior because the data is too valuable to 
leave to chance.  Or put in shorter terms, if you have to wait for 
external factors, then what's the point of multi-core parallel launching?

Re: dependant services

[Avery Payne]

On 6/8/2015 10:44 AM, Steve Litt wrote:

Just so we're all on the same page, am I correct that the subject of
your response here is *not* "socket activation", the awesome and
wonderful feature of systemd.

You're simply talking about a service opening its socket before it's
ready to exchange information, right?
That is my understanding, yes.  We are discussing using UCSPI to hold a 
socket for clients to connect to, then launching the service and 
connecting the socket on demand; as a by-product the assumption is the 
client will block while the launch is occuring for the socket.  Of 
course, to make this work, there is an implicit assumption that the 
launch includes handling of "service is up" vs "service is ready".



Isn't this all controlled by the service? sshd decides when to open its
socket: The admin has nothing to do with it.
UCSPI is basically the inetd concept re-done daemontools style.  It can 
be a local socket, a network socket, etc.  So the UCSPI program would 
create and hold the socket; upon connection, the service spawns.





[Snip 2 paragraphs discussing the complexity of sockets used in a
certain context]


If I were to write support for sockets in, I would guess that it
would probably augment the existing ./needs approach by checking for
a socket first (when the feature is enabled), and then failing to
find one proceed to peer-level dependency management (when it is
enabled).

Man, is all this bo-ha-ha about dependencies?
Sequencing actually; I'm just mixing a metaphor here, in that "my 
version" of dependencies is sequential, self-organizing, but not 
manually ordered.  Order is obtained by sequentially walking the tree, 
so while you have a little control by organizing the relationships, you 
don't have any control over which relationship launches first at a given 
level.


=
if /usr/local/bin/networkisdown; then
   sleep 5
   exit 1
fi
exec /usr/sbin/sshd -d -q
=

Is this all about using the existance of a socket to decide whether to
exec your service or not? If it is, personally I think it's too
generic, for the reasons you said: On an arbitrary service,
perhaps written by a genius, perhaps written by a poodle, having a
socket running is no proof of anything. I know you're trying to write
generic run scripts, but at some point, especially with dependencies on
specific but arbitrary processes, you need to know about how the
process works and about the specific environment in which it's working.
And it's not all that difficult, if you allow a human to do it. I think
that such edge case dependencies are much easier for humans to do than
for algorithms to do.
Oh, don't get me wrong, I'm saying that the human should not only be 
involved but also have a choice.  Yes, I will have explicit assumptions 
about "X needs Y" but there's still a human around that can decide if 
they want to flip the switch "on" to get that behavior.




If this really is about recognizing when a process is fully functional,
because the process being spawned depends on it, I'd start collecting a
bunch of best-practices, portable scripts called ServiceXIsDown and
ServiceXIsUp.
This is of passing interest to me, because a lot of that accumulated 
knowledge can be re-implemented to support run scripts.  I may write 
about that separately in a little bit.

Sorry for the DP101 shellscript grammar: Shellscripts are a second
language for me.

The project is currently written in shell, so you're in good company.


Anyway, each possible dependent program could have one or more
best-practice "is it up" type test shellscripts. Some would involve
sockets, some wouldn't. I don't think this is something you can code
into the actual process manager, without a kudzu field of if statements.
It wouldn't be any more difficult than the existing peer code.  Yes, I 
know you peeked at that once and found it a bit baroque but if you take 
the time to walk through it, it's not all that bad, and I'm trying hard 
to make sure each line is clear about its intention and use.


Regarding an older comment that was made about relocating peer 
dependencies into a separate script, I'm about 80% convinced to do it, 
if only to make things a little more modular internally.


[snip a couple paragraphs that were way above my head]


Of course, there are no immediate plans to support UCSPI, although
I've already made the mistake of baking in some support with a bcron
definition.  I think I need to go back and revisit that entry...

I'm a big fan of parsimonious scope and parsimonious dependencies, so
IMHO the less that's baked in, the better.
The minimum dependencies are there.  If anything, my dependencies are 
probably lighter than most - there isn't anything in shell that is baked 
in (i.e. explicit service X start statements in the script outright), 
and the dependencies themselves are simply symlinks that can be changed.





As a side note, I'm beginning to susp

Re: dependant services

[Avery Payne]

On 5/14/2015 3:25 PM, Jonathan de Boyne Pollard wrote:
The most widespread general purpose practice for "breaking" (i.e. 
avoiding) this kind of ordering is of course opening server sockets 
early.  Client and server then don't need to be so strongly ordered. 
This is where I've resisted using sockets.  Not because they are bad - 
they are not.  I've resisted because they are difficult to make 100% 
portable between environments.  Let me explain.


First, there is the question of "what environment am I running in"? This 
can break down in to several sub-questions of "what variable settings do 
I have", "what does my directory structure look like", and "what tools 
are available".  That last one - what tools are installed - is what 
kills me.  Because while I can be assured that the bulk of a framework 
will be present, there is no guarantee that I will have UCSPI sockets 
around.


Let's say I decide to only support frameworks that package UCSPI out of 
the box, so I am assured that the possibility of socket activate is 100% 
guaranteed, ignoring the fact that I just jettisoned several other 
frameworks in the process simply to support this one feature. So we 
press on with the design assumption "it is safe to assume that UCSPI is 
installed and therefore can be encoded into run scripts". Now we have 
another problem - integration.  Using sockets means I need to have a 
well-defined namespace to locate the sockets themselves, and that means 
a well-known area in the filesystem because the filesystem is what 
organizes the namespace.  So where do the sockets live?  /var/run?  
/run?  /var/sockets? /insert-my-own-flavor-here?


Let's take it a step further and I decide on some name - I'll pull one 
out of a hat and simply call it /var/run/ucspi-sockets - and ignore all 
of the toes I'm stepping on in the process, including the possibility 
that some distribution already has that name reserved. Now I have (a) 
the assurance that UCSPI is supported and (b) a place for UCSPI to get 
its groove on, then we have the next problem, getting all of the 
services to play nice within this context.  Do I write everything to 
depend on UCSPI sockets so that I get automatic block?  Do I make it 
entirely the choice of the administrator to activate this feature via a 
"switch" that can be thrown?  Or is it used for edge cases only?  
Getting consistency out of it would be great, but then I back the admin 
into a corner with "this is design policy and you get it, like it or 
not".  If I go with admin controlled, that means yet another code path 
in an already bloaty ./run.sh script that may or may not activate, and 
the admin has their day with it, but the number of potential problem 
vectors grows.  Or I can hybridize it and do it for edge cases only, but 
now the admin is left scratching their head asking "why is it here, but 
not there?  it's not consistent, what where they thinking??"


Personally, I would do the following:

* Create a socket directory in whatever passes for /var/run, and name it 
/var/run/ucspi-sockets.


* For each service definition that has active sockets, there would be 
/var/run/ucspi-sockets/{directory} where {directory} is the name of the 
service, and inside of that is a socket file named 
/var/run/ucspi-sockets/{directory}/socket.  That is about as generic and 
"safe" as I can get, given that /var/run on Linux is a symlink that 
points to /run in some cases.  It is consistent - the admin knows where 
to find the socket every single time, and is assured that the socket 
inside of the directory is the one that connects to a service.  It is a 
reasonable name - the odds of /var/run/ucspi-sockets being taken for 
anything else but that is fairly low, and the odds of me stepping on top 
of some other construct in that directory are low as well, because any 
existing sub-directory in that location is probably there for the same 
reason.


* Make socket activate an admin-controlled feature that is disabled by 
default.  You want socket activation, you ask for it first.  The admin 
gets control, I get more headache, and mostly everyone can be happy.


We've answered the "where" and the "when", now we are left with the 
"how".  I suspect that you and Laurent would argue that I shouldn't be 
using sockets inside of ./run as it is, that it should be in the layer 
above in service management proper, meaning that the entire construct 
shouldn't exist at that level.  Which means I shouldn't even support it 
inside of ./run.  Which means I can't package this feature in my 
scripts.  And we're back to square one.


Let's say I ignore this advice (at my own peril) and provide support for 
those frameworks that don't have external management layers on top of 
them.  This was the entire reason I wrote my silly peer-level dependency 
support to begin with, so that "other folks" would have one or two of 
these features available to them, even though they don't have external 
management like nosh or s6-rc or anopa

supervision-scripts, 2015-05

[Avery Payne]

The project has slowed and I do not have nearly the number of updates. 
It's not dead, I tell you, it's merely resting.  And no I didn't nail 
its feet to the perch.


( for those that don't get the reference... 
https://www.youtube.com/watch?v=4vuW6tQ0218 )



Done:
- - - -
+ New definitions: arpalert

+ Experimental support for perp was added, using the "shim" described at 
the perp website.  It has not been tested, but should work all the same.



In Progress:
- - - - - - -
+ Switch to a BSD-styled license as the number of target definitions has 
decreased.


+ Work is temporarily on hold due to personal commitments. As those 
obligations are met, I will be able to resume adding definitions.



To Do:
- - - -
+ Revise perp support into its own run-perp.sh

+ Build full environments for testing, one for each framework.

+ Examine the definitions employed by nosh.

+ Everything else from last month.

Re: Arch Linux derivative using s6?

[Avery Payne]

On 5/14/2015 3:47 PM, Jonathan de Boyne Pollard wrote:
There are even more than that.  I mentioned back in January that the 
nosh Guide chapter on creating service bundles has pointers to the run 
file collections by Gerrit Pape, Wayne Marshall, Kevin J. DeGraaf, and 
Glenn Strauss.  I also pointed out that nosh came with some 177 
pre-built service bundles.  That figure has since risen to some 
230-odd (not including log services).


"We are supervision-scripts. Lower your firewalls and surrender your 
source. We will add your definitions and technological distinctiveness 
to our own. Your framework will adapt to service us. Resistance is futile."


Oops, sorry, don't know what came over 
me...


I will most assuredly pursue those 'service bundles' from *all* of the 
above authors when time permits... believe me, I've already scoured out 
most of github and bitbucket.  I've also done a few off of runit's 
definitions.


Nosh is still on my to-do list.  Near as I can tell, it shouldn't be too 
hard to include support for it, but I won't really know until I get a 
full VM cooked.  I think the quickest way to get this accomplished - for 
both nosh and s6 - is to install Debian 8 sans systemd into a VM image.  
From there I can add your new Debian packages to get nosh installed, and 

I will finally have GNU make 4.0 for building s6.

supervision-scripts, 2015-04

[Avery Payne]

April has shown some steady progress, but it is slowing due to personal 
commitments, namely a 2nd part-time job I've taken.Life is rather hectic 
right now, but I am trying to keep an eye on the mailing list and my email.


The number of target definitions has been reduced.  Several non-service 
definitions were found in the list that is used, and removed.  This 
brings the target number down to ~1,100, so technically I am approaching 
the 0.1 release candidate.  With luck, this will be announced next month.


PHK (a FreeBSD dev) is working on a replacment for the venerable ntpd 
daemon.  The initial work has appeared as ntimed-client.  I was able to 
build, run, and confirm it as working, despite that it is not production 
ready.  A definition is now available.


Lastly, there is a change in strategy.  In past months, I have taken the 
time to ensure that each service definition is tested as best I can, but 
the shear number of them makes this time prohibitive.  The existing 
run.sh script has proven itself reliable and I think I can safely reduce 
the amount of testing to random samples and key services.  So there will 
be an increase in the number of definitions made that have the 
"untested" marker, but the velocity of new definitions will start to 
increase.  Yes, this is why there are suddenly several untested 
definitions in alphabetical sequence this month. :)


Done:
- - - -
+ Added new definitions: pidentd, xtelld, NetworkManager, ntimed-client, 
bcron-update, bcron-sched, bcron-spool


+ Added new untested definitions: gdm, avahi-dnsconfd, aiccu, ajaxterm, 
all-knowing-dns, amavisd-new, amule-daemon, anacron, anon-proxy (aka 
mix), autolog, arpalert.


+ Fixes: added a missing ./needs to console-kit-daemon.

+ Revised README, SETUP and FEATURE documents.


Contributions:
- - - - - - - -

+ Fixed a bug in the definition for bind, based on a report from 
a...@clegg.com.


The following were implemented suggestions passed by drwi...@drwilly.de:
+ run-user-service.sh: Revamped to spawn fewer subshells.
+ run-user-service.sh: Fixed a bug that would cause the runit variant to 
crash.
+ run.sh: Removed two calls to pwd, instead using the shell's 
environment variable.

+ run.sh: Removed two spawned subshells, replaced with programs

Both contributors are greatly appreciated.


In Progress:
- - - - - - -
+ Held in the patch queue needing work: arpon, krb5kdc, 
krb5-admin-server, dropbear, cpufreqd, rpc.nfsd, rpc.svcgssd, rpc.gssd, 
rpc.idmapd, rpc.mountd


+ Conversion of bind to envdir format

+ Additional patches from drwi...@drwilly.de that could not be cleanly 
integrated.  The largest one would remove sed and awk, and use cut, 
reducing the number of external dependencies.




To Do:
- - - -
+ Finish perp support.  Because of how sv/.run/run works, this can 
easily be done by creating a new run-perp.sh script that supports perp's 
requirements.  This will also involve a use-perp support script.


+ I need to think through a kernel-neutral method of handling driver 
loading.  There are a handful of services that require this to 
succesfully launch.  For example, cpufreqd requires kernel modules to 
load first, but this is linux-specific.  Does FreeBSD have any 
requirements like this?  Or are these services so tied to the kernel 
platform that it doesn't really matter? (i.e. cpufreqd won't run on 
FreeBSD so there is no concern about incompatible module loading)


+ Re-think instance support. Attempting to encode a sane default isn't 
possible because the naming conventions aren't consistent, so the 
systems administrator needs to intervene. For example, there are a 
number of services that want to listen to "a specific network adaptor".  
Compounding this problem is that the device names of network adaptors 
not only changes between kernels (think FreeBSD device names vs. Linux 
device names), but also in the same kernel as well (due to systemd's new 
"network device names").


+ I need to wrestle with the new crop of init-startup-oriented projects, 
as well as existing ones.  I've read over some of anopa and even 
runit-for-lfs, but I haven't had time to really digest all of the 
designs, and the init for s6 hasn't even been released - yet.  While 
supervision-scripts is not meant to provide startup/shutdown support, it 
is meant to wedge into these environments and work within them, so 
having a passing knowledge of them is essential.  I've also wanted to 
interject a few thoughts and notes on "towards a universal 
startup/shutdown".


+ Everything else from last month.

Re: Thoughts on "First Class Services"

[Avery Payne]

Note: this re-post is due to an error I made earlier today.  I've gutted 
out a bunch of stuff as well.  My apologies for the duplication.


On 4/28/2015 11:34 AM, Laurent Bercot wrote:


I'm also interested in Avery's experience with dependency handling.


Hm.  Today isn't the best day to write this (having been up since 4am) 
but I'll try to digest all the little bits and pieces into something.  
Here we go...


First, I will qualify a few things.  The project's scope is, compared to 
a lot of the discussion on the mailing list, very narrow.  There are 
several goals but the primary thrust of the project is to create a 
generic, universal set of service definitions that could be plugged into 
many init, distribution, and supervision framework arrangements. That's 
a tall order in itself, but there are ways around a lot of this.  So 
while the next three paragraphs are off-topic, they are there to address 
those three concerns mentioned.


With regard to init work, I don't touch it.  Trying to describe a proper 
init sequence is already beyond the scope of the project. I'm leaving 
that to other implementers.


With regard to distributions, well, I'm trying to make it as generic as 
possible.  Development is done on a Debian 7 box but I have made efforts 
to avoid any Debian-isms in the actual project itself.  In theory, you 
should be able to use the scripts on any distribution.


With regard to the supervision programs used, the difference in command 
names have been abstracted away.  I'm not entirely wild about how it is 
currently done, but creating definitions is a higher priority than 
revisiting this at the moment.  In the future, I will probably 
restructure it.


~ ~ ~ ~ ~ ~ ~ ~

What
---
The dependency handling in supervision-scripts is meant to be used in 
installations that don't have access to it.  Put another way, it's a 
"Poor Man's Solution" to the problem and functions as a convenience.  
The feature is turned off by default, and this will cause any service 
definition that requires other services to run-loop repeatedly until 
someone starts them manually.  This could be said to be the default 
behavior of most installations that don't have dependency handling, so 
I'm not introducing a disruptive behavior with this feature.


Why
---
I could have hard-coded many of the dependencies into the various run 
scripts, but this would have created a number of problems for other areas.


1. Hard-coding prevents switching from shell to execline in the future, 
by necessitating a re-write.  There will be an estimated 1,000+ scripts 
when the project is complete, so this is a major concern.


2. We are already using the filesystem as an ad-hoc database, so it 
makes sense to continue with this concept.  The dependencies should be 
stored on the filesystem and not inside of the script.


With this in mind, I picked sv/(service)/needs as a directory to hold 
the definitions to be used.  Because I can't envision what every init 
and future dependency management framework would look like, I'll simply 
make it as generic as I can, leaving things as open as possible to 
additional changes.A side note: it is by fortuitous circumstance 
that anopa uses a ./needs directory that has the same functionality and 
behavior.  I use soft links "just because".  Anopa uses named files.  
The net effect is the same.


Each dependency is simply a named soft link that points to a service 
that needs to be started, typically something like 
"sv/(service)/needs/foobar points to /service/foobar".  In this case, a 
soft link is made with the name of the service, pointing to the service 
definition in /service.  This also allows me to ensure that the 
dependency is actually available, and not just assume that it is there.


A single rule determines what goes into ./needs, "you can only have the 
names of other services that are explicitly needed".  You can say "foo 
needs baz" and "baz needs bar" but NEVER would you say "foo needs baz, 
foo needs bar".  This is intentional because it's not the job of the 
starting service to handle the entire chain.  It simplifies the list of 
dependencies because a service will only worry about its immediate 
needs, and not the needs of dependent services it launches.  It also has 
the desirable property of making dependency chains self-organizing, 
which is an important decision with hundreds of services having 
potentially hundreds of dependencies.   Setup is straightforward and you 
can easily extend a service need by adding one soft link to the new 
dependency.  This also fits with my current use of a single launch 
script; I don't have to change the script, just the parameters that the 
script uses.  The new soft link becomes just another parameter.  You 
could call this "peer-level dependency resolution" if you like.



How
---
Enabling this behavior requires that you set sv/.env/NEEDS_ENABLED to 
the single character "1".   It is normally set to "0".  With the setting 
disabled (

Re: Thoughts on "First Class Services"

[Avery Payne]

Dang it.  Hit the send button.

It will be a bit, I'll follow up with the completed email.  Sorry for 
the half-baked posting.

Re: Thoughts on "First Class Services"

[Avery Payne]

On 4/28/2015 11:34 AM, Laurent Bercot wrote:


I'm also interested in Avery's experience with dependency handling.


Hm.  Today isn't the best day to write this (having been up since 4am) 
but I'll try to digest all the little bits and pieces into something.  
Here we go...


First, I will qualify a few things.  The project's scope is, compared to 
a lot of the discussion on the mailing list, and especially in regard to 
s6-rc and anopa, very narrow.  There are several goals but the primary 
thrust of the project is to create a generic, universal set of service 
definitions that could be plugged into any init, distribution, and 
supervision framework arrangement. That's a tall order in itself, but 
there are ways around a lot of this.


With regard to init work, I don't touch it.  Period.  Trying to describe 
a proper init sequence is already well beyond the scope of the project; 
and everyone has their own needs.  Compound this with things like, oh, 
the differences between Linux and FreeBSD starting up, and you have 
plenty of headache to go around.  So my work around for this is to 
completely avoid it; I'm leaving that to other implementers.


With regard to distributions, well, I'm trying to make it as generic as 
possible.  Development is done on a Debian 7 box but I have tried to 
make every effort to avoid any Debian-isms in the actual project 
itself.  In theory, you should be able to use the scripts on any 
distribution, and not just Debian.


With regard to the supervision programs used, this was the tricky one.  
While daemontools, runit, and s6 all have ./run, ./finish, and ./check, 
there's differences in the commands, signals, and capabilities 
involved.  So unless I want to write a bunch of case statements in a 
shell script for each time I touch a command, this is a problem.  
Fortunately, both runit and s6 have emulation of daemontools commands.  
You create a symlink of the daemontools name and point it at the runit/s6

Re: Thoughts on "First Class Services"

[Avery Payne]

On 4/28/2015 11:34 AM, Laurent Bercot wrote:


 If a lot of people would like to participate but don't want to
subscribe to the skaware mailing-list, I'll move the thread here.

Good point, I'm going to stop discussion here and go "over there", where 
the discussion belongs.

Re: Thoughts on "First Class Services"

[Avery Payne]

On 4/28/2015 10:50 AM, bougyman wrote:
Well at least we're talking the same language now, though reversing 
"parent/child" is disconcerting to my OCD. 


Sorry if the terminology is "reversed".


Here's the current version of run.sh, with dependency support baked
in:
https://bitbucket.org/avery_payne/supervision-scripts/src/b8383ed5aaa1f6d848c1a85e6216e59ba98c3440/sv/.run/run.sh?at=default


That's a gnarley run script. It's as big as a lot of sysvinit or OpenRC
scripts I've seen. One of the reasons I like daemontools style package
management is my run scripts are usually less than 10 lines.


This was my thought, as well. It adds a level of complexity we try to
avoid in our run scripts.
It also seems to me that there is less typing involved in individual
run scripts than the
individual things that have to be configured for this script. If on
goal of this
abstraction is to minimize mistakes, adding more moving parts to edit
doesn't seem to
work towards that goal.


Currently there are the following sections, in sequence:

1. shunt stderr to stdout for logging purposes
2. shunt supporting symlinks into the $PATH so that tools are "called" 
correctly.  This is critical to supporting more than just a single 
framework; all of the programs referenced in .bin are actually symlinks 
that point to the correct program to run.  See the .bin/use-* scripts 
for details.
3. if a definition is "broken" in some way, then immediately write a 
message to the log and abort the run.
4. if dependency handling is enabled, then process dependencies. 
Otherwise, just skip the entire thing.  By default, dependencies are 
disabled; this means ./run scripts behave as if they have no dependency 
support.
4a. should dependency handling fail, log the failing child in the 
parent's log, and abort the run.

5. figure out if user or group IDs are in used, and define them.
6. figure out if a run state directory is needed.  If so, set it up.
7. start the daemon.

Re: Thoughts on "First Class Services"

[Avery Payne]

On 4/28/2015 10:31 AM, Steve Litt wrote:

Good! I was about to ask the definitions of parent and child, but the
preceding makes it clear.


I'm taking it from the viewpoint that says "the service that the user 
wishes to start is the parent of all other service dependencies that 
must start".



So what you're doing here is minimizing polling, right? Instead of
saying "whoops, child not running yet, continue the runit loop", you
actually start the child, the hope being that no service will ever be
skipped and have to wait for the next iteration. Do I have that right?
Kinda.  A failure of a single child still causes a run loop, but the 
next time around, some of the children are already started, and a start 
of the child will quickly return a success, allowing the script to skip 
over it quickly until it is looking at the same problem child from the 
last time.  The time lost is only on failed starts, and child starts 
typically don't take that long.  If they are, well, it's not the 
parent's fault...


  

Here's the current version of run.sh, with dependency support baked
in:
https://bitbucket.org/avery_payne/supervision-scripts/src/b8383ed5aaa1f6d848c1a85e6216e59ba98c3440/sv/.run/run.sh?at=default


That's a gnarley run script.


Yup.  For the moment.


If I'm not mistaken, everything inside the "if test
$( cat ../.env/NEEDS_ENABLED ) -gt 0; then" block is boilerplate that
could be put inside a shellscript callable from any ./run.


True, and that idea has merit.


  That would
hack off 45 lines right there. I think you could do something similar
with everything between lines 83 and 110. The person who is truly
interested in the low level details could look at the called
shellscripts (perhaps called with the dot operator). I'm thinking you
could knock this ./run down to less than 35 lines of shellscript by
putting boilerplate in shellscripts.
I've seen this done in other projects, and for the sake of simplicity 
(and reducing subshell spawns) I've tried to avoid it. But that doesn't 
mean I'm against the idea.  Certainly, all of these are improvements 
with merit, provided that they don't interfere with some of the other 
project goals.  If I can get the time to look at all of it, I'll 
re-write it by segmenting out the various components.


In fact, you may have given me an idea to solve an existing problem I'm 
having with certain daemons...




You're doing more of a recursive start. No doubt, when there are two or
three levels of dependency and services take a non-trivial amount of
time to start (seconds), yours results in the quicker boot. But for
typical stuff, I'd imagine the old "wait til next time if your ducks
aren't in line" will be almost as fast, will be conceptually
simpler, and more codeable by the end user. Not because your method is
any harder, but because you're applying it against a program whose
native behavior is "wait til next cycle".


Actually, I was looking for the lowest-cost solution to "how do I keep 
track of dependency trees between multiple services".  The result was a 
self-organizing set of data and scripts.  I don't manage *anything* 
beyond "service A must have service B".  It doesn't matter how deep that 
dependency tree goes, or even if there are common "leaf nodes" at the 
end of the tree, because it self-organizes.  This reduces my cognitive 
workload; as the project grows to hundreds of scripts, the number of 
possible combinations reaches a point where it would be unmanageable 
otherwise.  Using this approach means I don't care how many there are, I 
only care about what is needed for a specific service.



And, as you said in a past email, having a run-once capability without
insane kludges would be nice, and as you said in another past email,
it's not enough to test for the child service to be "up" according to
runit, but it must pass a test to indicate the process itself is
functional. I've been doing that ever since you mentioned it.


At some point I have to go back and start writing ./check scripts. :(

Re: Thoughts on "First Class Services"

[Avery Payne]

On 4/28/2015 7:18 AM, bougyman wrote:

On Tue, Apr 28, 2015 at 8:38 AM, Avery Payne  wrote:

I guess I don't know what this means, in practice. My child services
generally know about the
parent in their ./run script and the parent (sometimes) has to know
about the children in his
./finish script.


I do the opposite using an optional flag that is initially disabled at 
installation.   The short version is, using 
parent-only-knows-child-needs makes the dependency chains self-organizing.


The long version is that, in supervision-scripts, each service (parent) 
has a list of immediate needs to be met (children).  It attempts to 
start each child in a subdirectory named ./needs (coincidentally, anopa 
uses a compatible arrangement with ./needs). Each child is iterated 
through and checked to see if it is running; if it is not running, it is 
started, and the next child examined, until all children are started.  
If all children start, the parent proceeds to start.  If a child fails, 
the parent writes a message to its log saying "my child X failed" and 
proceeds to spawn-loop.


As each child starts, if there is a ./needs in the child, it repeats the 
same behavior as the parent, calling other grandchildren and then 
eventually starting once all of its own dependencies are running.  The 
script is the same for parent and children alike, so the behavior is 
meant to be consistent.  The process is repeated for each grand-child, 
great-grand-child, etc. until the lowest dependencies are running, 
causing the entire dependency "tree" to be traversed.  This also means 
the "tree" can be a "chain" or any other arrangement.  "Leaf nodes" of 
that tree can be called more than once without harm because if they were 
already launched elsewhere, the start-up of that node is skipped.  And 
finally, I don't need to track the entire state of the tree, I only need 
to track what children are needed for any given parent, making the 
entire thing self-organizing, while keeping it simple.



None of the parents know anything about their children.
I'm just doing it backwards.  If I had definitions for postgresql and 
pgbouncer, I would only add the following soft links:


/etc/sv/app_which_uses_pg/needs/pgbouncer
/etc/sv/pgbouncer/needs/postgresql

That's pretty much it.  Nothing else would need to be added to ./run for 
each service.


I'd like to see the difference in a code example. I haven't had a 
chance to dig in to anopa yet enough to see how they couple it mouse 
loosely. Tj 


Here's the current version of run.sh, with dependency support baked in:
https://bitbucket.org/avery_payne/supervision-scripts/src/b8383ed5aaa1f6d848c1a85e6216e59ba98c3440/sv/.run/run.sh?at=default

Re: dependant services

[Avery Payne]

On 4/28/2015 7:11 AM, Steve Litt wrote:
Hi Buck and Avery, What's "first-class support"? Is there a way, in 
this thread, that the service behavior desired could be described, 
instead of calling it "first class support"? 
Then it's best that the term be abandoned.  I've tried to sum up the 
idea myself, but I realize (too late) that I've been bit by a buzzword; 
and I'm the one who opened their mouth and said it.


To me, first class support means "the most optimal within the given 
restraints", which is almost as vague-sounding once I write it. I've 
probed the idea a bit more this morning (see 
http://www.mail-archive.com/supervision@list.skarnet.org/msg00647.html*) 
*and found that first class support really just means "it does this, 
where others don't".  Simplify the idea again and you end up with "it 
works".  Well, duh, that's not too helpful, is it?  So, I am publicly 
retracting the utterance of  the phrase; either service dependency is 
supported or it isn't.

Re: Arch Linux derivative using s6?

[Avery Payne]

On 4/28/2015 6:56 AM, bougyman wrote:
Currently runit-void uses /etc/runit/1, 2, and 3 for those stages, 
with runlevels in /etc/runit/runsvdir, runsvdir watches 
/run/runit/runsvdir/current, a symlink to /etc/runit/runsvdir/current. 
Void linux is licensed under a 2 clause BSD style license, while 
runit-void (the supervision scripts) itself is public domain 
https://github.com/voidlinux/void-runit 
The list of directories in /services only shows entries for getties, are 
all of the other definitions inside of their respective packages?

Thoughts on "First Class Services"

[Avery Payne]

On 4/22/2015 5:56 AM, bougyman wrote:
What more is needed and what would making this "first class" add or 
improve upon? Tj 


This question has been in the back of my mind for several days now, 
giving it some off-and-on consideration.


The steps described are explicit and hard-coded into a service's ./run 
script.  This design works "just OK" and ensures complicated start-up 
sequences can be carried out, but it does not allow easy replacement of 
child services.  The catch is that the name of child(ren) will be 
"baked" into the parent ./run script.


The loosely-coupled approach taken by anopa (and my own work) allows 
easy replacement of a child service with some other service, at the cost 
of slightly increased complexity in the child's run script. The net 
effect is the same but the difference is subtle.  It also forces the 
set-up of the child to consistently be in the child's ./run script and 
not the parent,  because it isolates child set-up to its scope instead 
of potentially blurring it with the parent's ./run script; currently in 
my eyes this is important because it follows "separation of duty" and 
makes who-has-what-responsibility clear.

Re: dependant services

[Avery Payne]

On 4/21/2015 3:08 PM, Buck Evan wrote:



On Tue, Apr 21, 2015 at 2:46 PM, Avery Payne <mailto:avery.p.pa...@gmail.com>> wrote:


Alternatively, are there general-purpose practices for
breaking this kind
of dependency?

Strange as it sounds, renaming the child definition of a
dependency chain (which typically translates into the directory
name of the defintion) seems to be a regular issue.  Changing the
name of the definition typically causes various "links" to break,
causing the parent service to be unable to locate its children by
name at start-up.


Ah, I just realized you misunderstood me. You understood "breaking 
dependencies" to mean "the dependant system no longer works" where 
what I meant was "the dependency is no longer relevant".
With regard to practice or policy, I can only speak to my own project.  
I try to stick with "minimum feasible assumption" when designing 
things.  In the case of the run script handling dependencies, it only 
assumes that the child failed for reasons known only to the child, and 
therefore the parent will abort out and eventually spawn-loop.  Prior to 
exiting the script, a message is left for the systems administrator 
about which child failed, so that they can at least see why the parent 
refused to start.  Beyond that, I try not to assume too much.


If the dependency is no longer relevant, then that is a small issue - 
the ./needs directory holds the names of all the child processes that 
are needed, and if the child will fail because it's broken / moved / 
uninstalled / picked up its marbles and went home, then the parent will 
simply continue to fail to start, until the child's name is removed from 
the ./needs directory.  Again, you'll see a recorded message in the 
parent log about the child causing the failure, but not much more than 
that.  It can be easily fixed by simply removing the child symlink in 
./needs, which will cause the parent to "forget" about the child.  This 
possibility should be documented somewhere in the project, and I know I 
haven't done so yet.  Thanks for bringing it up, I'll try to get to it soon.

Re: dependant services

[Avery Payne]

On 4/21/2015 2:56 PM, Buck Evan wrote:
My understanding of s6 socket activation is that services should open, 
hold onto their listening socket when they're up, and s6 relies on the 
OS for swapping out inactive services. It's not socket activation in 
the usual sense. http://skarnet.org/software/s6/socket-activation.html


I apologize, I was a bit hasty and I think I need more sleep.  I'm 
confusing socket activation with some other s6 feature, perhaps I was 
confusing it with how s6-notifywhenup is used... 
http://skarnet.org/software/s6/s6-notifywhenup.html
So I wonder what the "full guarantee" provided by s6 that you 
mentioned looks like.
It seems like in such a world all services would race and the 
determinism of the race would depend on each service's implementation.
This I do understand, having gone through it with supervision-scripts.  
The basic problem is that a running service does not mean a service is 
ready, it only means it's "up".


Dependency handling "with guarantee" means there is some means by which 
the child service itself signals "I'm fully up and running", vs. "I'm 
started but not ready".  Because there is no polling going on, this 
allows the start-up of the parent daemon to sleep until it either is 
notified or times out.  And you get a "clean" start-up of the parent 
because the children have directly signaled that "we're all ready".


Dependency handling "without guarantee" is what my project does as an 
optional feature - it brings up the child process and then calls the 
child's ./check script to see if everything is OK, which is polling the 
child (and wasting CPU cycles).  This is fine for "light" use because 
most child processes will start quickly and the parent won't time out 
while waiting.  There are trade-offs for using this feature.  First, 
./check scripts may have unintended bugs, behaviors, or issues that you 
can't see or resolve, unlike the child directly signalling that it is 
ready for use.  Second, the polling approach adds to CPU overhead, 
making it less than ideal for mobile computing - it will draw more power 
over time.  Third, there are edge cases where it can make a bad 
situation worse - picture a heavily loaded system that takes 20+ minutes 
to start a child process, and the result being the parent spawn-loops 
repeatedly, which just adds even more load.  That's just the three I can 
think off off the top of my head - I'm sure there's more.  It's also why 
it's not enabled by default.

Re: dependant services

[Avery Payne]

On 4/21/2015 2:19 PM, Buck Evan wrote:

Does s6 (or friends) have first-class support for dependant services?
I know that runit and daemontools do not.  I do know that nosh has 
direct support for this. I believe s6 supports it through various 
intermediary tools, i.e. using socket activation to bring services up, 
so you could say that while it supports it directly and provides a full 
guarantee, it's not "first class" in the sense that you can simply 
provide a list of "bring these up first" and it will do it out of the 
box.  The recently announced anopa init system fills in this gap and 
makes it "first class", in the sense that you can simply provide the 
names of definitions that need to start and everything else is handled 
for you.



Alternatively, are there general-purpose practices for breaking this kind
of dependency?
Strange as it sounds, renaming the child definition of a dependency 
chain (which typically translates into the directory name of the 
defintion) seems to be a regular issue.  Changing the name of the 
definition typically causes various "links" to break, causing the parent 
service to be unable to locate its children by name at start-up.

Re: Another attempt at S6 init

[Avery Payne]

On 4/21/2015 7:34 AM, TheOldFellow wrote:


So I should need much less than Laurent has in his example.  (did I mention
the ancient grey cells?)
I'm no expert at execline, so I'm taking wild guesses here based on the 
little bits that I know from reading about it.



#close stdout and stderr
fdclose 1 fdclose 2
...would it hurt anything to move the fdclose 1 & 2 near the "connect 
stdin to /dev/null"?  Yes your console will be noisy but you at least 
get to see if anything is hurt or wounded prior to re-plumbing everything.


At Laurent's website, it mentions setting up an early getty so that you 
have access to something during the boot.

   The best suggestion is probably 'use
systemd', but please refrain.

(saying this while I cover my ears) You said the "s" word.

Re: Arch Linux derivative using s6?

[Avery Payne]

On 4/19/2015 7:03 AM, John Regan wrote:

It's not quite the same, but I think Alpine linux is pretty close to what 
you're looking for. They'd probably love to get more people involved, writing 
documentation, making packages, etc. It doesn't use s6, but I've submitted the 
s6 packages to the project. Maybe you could work on adding s6 init scripts to 
packages?
There's already a project for adding definitions for various daemons.   
http://bitbucket.org/avery_payne/supervision-scripts

Re: anopa: init system/service manager built around s6

[Avery Payne]

On 4/10/2015 6:41 PM, Aristomenis Pikeas wrote:

Laurent (s6-rc), Olivier (anopa), Toki (supervision framework), and Gorka 
(s6-overlay),

I'm having a lot of trouble figuring out the differences between your projects. 
The s6 suite of utils can be considered building blocks for a full init system, 
but what does each of your projects do on top of s6?


A breakdown of differences:

+ s6-rc: Laurent's version of an init system based on s6, meant to bring 
a machine up/down.


+ anopa: Olivier's version of an init system base on s6, meant to bring 
a machine up/down.


+ supervision framework: Toki's version of a complete framework-agnostic 
init system, partially geared towards OpenRC. Last I checked he had 
progressed into supporting a complete init (with OpenRC support).


+ s6-overlay: this is meant for Docker containers, and is most likely 
the one you want.


Yes, there is duplication of effort between Laurent and Olivier, that's 
ok though - I personally argue that choice is a good thing. :)  There 
are actually many, many projects out there, if you know what to look 
for, that might provide clues or insights.  Ignite on github I believe 
has some rudimentary init stuff in it, although it's runit based.



For a bit of context, my goal is the simplest init system that could possibly 
work, to be run inside of a docker container. I need to start services and 
gracefully handle SIGTERM/SIGKILL, with everything logged to standard out. 
That's about it. But this is proving to be difficult with s6. I've been 
chipping away at things, but it's slow going between understanding all of the 
tricky bash-isms and learning about all of the relevant s6 components.

If by "tricky bash-isms" you mean the shell redirections and exec and 
all of that, well...once you can visualize it, it's not that bad 
really.  I don't believe any of the projects use bash directly. Toki's 
project (as well as my own) assume /bin/sh, which at this time usually 
means an ash variant.  Laurent and Olivier have *nothing* done in bash 
(beyond the build process).  If anything, I think all of the projects 
are trying hard to avoid "bash specific" implementations; believe me 
when I say, I've looked at *a lot* of shell scripting in the last 5 
months, and I can say that a lot of projects with shell scripts are 
actually fairly clean.  (Yes, I sound a little surprised when I say that)

POLL RESULTS: what installations would you use process supervision in?

[Avery Payne]

There were 8 respondents.

[ 4 ] A hand-made / hand-customized Linux installation

[ 1 ] A commercial installation (HP-UX, AIX, Pre-Oracle Solaris)

[ 2 ] an installation made with LFS

[ 2 ] an installation made with Gentoo

[ 0 ] an installation made with Arch

[ 3 ] an installation made with Debian / Ubuntu

[ 2 ] an installation made with Fedora / Red Hat

[ 4 ] an installation made with NetBSD/OpenBSD/FreeBSD

[ 1 ] an installation made with DragonflyBSD

[ 0 ] an installation made with Android Open Source Project

[ 4 ] an installation not listed here (please give name and/or details)
for this category, responses are broken down as:
+ 1  condensed summary: runit within a larger project that uses Docker
+ 1 "Illumos-derived distros (e.g. SmartOS, OmniOS)"
+ 1 "Docker Images, Using Gorka's s6-overlay"
+ 1 "Ubuntu and Alpine Linux, but both inside docker :-)"

Method:
An open invitation to the supervision mailing list, with multiple choice 
responses being sent to my email address.  Responses were tallied on 
April 1 (no joke).  Each respondent is allowed +1 vote for a category, 
although multiple categories are allowed.


Summary:
The poll was meant to provide a broad picture of how process supervision 
is used by platform, and give a general feeling to how people are using 
it.  Even with just 8 respondents, this is informative.  From this, some 
(personally biased) observations:


* Docker, which is not an OS but a container solution, has a surprising 
amount of interest.  I haven't had time to play with Docker yet so I can 
only guess as to why - perhaps the low overhead and/or storage 
requirements that come with this model lend to making for "slim" containers?


* People have a very keen interest in using supervision with *BSD, with 
4 responders using it in some fashion.  Perhaps some "outreach" to those 
communities is in order...


* I was surprised to see Fedora/Red Hat listed, as these are 
traditionally systemd based, and systemd provides a superset of process 
supervision features.


* Some of the design decisions in my project came from the idea that the 
definition directories should be as portable as possible, because 
process supervision is a concept that extends to a large number of 
systems, and not just "the one".  As a result of that decision, 
development has been very slow and deliberate, probably slower than I 
would like.  Because I'm seeing a strong showing by non-Linux systems, I 
think it hints strongly that this was the right decision to make.


A big "Thank You" to everyone for your time, votes, and comments.

supervision-scripts, 2015-03

[Avery Payne]

March continues to be a busy month.  Most of the work done was internal, 
although a few definitions managed to make it out the door.


Done:
- - - - - - - -
+ New definitions: redis-server, ahcpd, mongodb, quake-server, famd, 
gunicorn, mdadm


+ Revised definitions: rpcbind (now includes a ./check script from Luke 
Diamand, thanks!)


+ The logging schema has been fleshed out to support daemontools and 
s6.  It remains untested at the moment.


+ New! Experimental option for logging called "autolog", a script that 
guesses at the logging to be used by looking at the framework in use.  
This option is not enabled by default and is considered experimental.  
If it works well, it may become the new default and replace the current 
policy of setting the sv/.log/run symlink to /bin/true.


+ Took a first pass at splitting up documentation into separate files.  
I also updated some of the troubleshooting documentation as it was 
referring to removed code.


+ Cleaned up remaining scripts, code, comments, and naming conventions.


In Progress:
- - - - - - - -

+ More definitions!  For instance, I'm sitting on a pile of patches for 
rpc-related services (read: support for NFS) and it still needs to cook 
a bit more before it's done.


+ The only non-template definitions left are: autofs, bind, bluetoothd, 
dbus, dhcpd, lwresd, postfix, wicd, and ypbind.  They all need some 
attention.


+ 3 services have broken flags: munin-node, quake-server, and 
redis-server.  Munin has structural issues, quake-server and 
redis-server need additional configuration.


+ Design notes.  I took an initial stab at it but I don't like the 
structure/layout.  Not enough "why", too much "the history of it means 
X".  Needs more refinement.


+ Clean up untested definitions, as part of the push for the 0.1 
release.  Many of the definitions have met the minimum test requirements 
and I've not cleared out the "untested" marker.  The current criteria 
for being tested is (a) the service launches cleanly (b) the service 
does not complain beyond warnings in its logs (c) the service appears to 
function as intended.


+ ...and of course, enough new definitions to justify a 0.1 release.  
This means at least 122 *tested* entries in sv/ must be present, which 
is about 10% of the count of init.d scripts used by Debian 7.  There are 
currently 96 entries but many are for getties, so after deducting 18 
redundant entries that leaves 78 definitions, which means I need to make 
44 entries.  Worst case, I end up with a 0.1 release candidate.



Still To-Do / Experimental:
- - - - - - - -
+ Think about ways to incorporate perp, which uses a different format of 
./run file. With the recent switch to support /bin/sh vs execline, this 
is a real possibility now.   That means sv/.run/run-sh, 
sv/.run/run-execline, and now sv/.run/run-perp will become potential 
targets for sv/.run/run.  Madness, I tell you, madness!


+ Examine nosh.  This is going take a bit of time to digest...

+ Re-think/re-work the ./needs directory.  While nosh already has a 
similar concept, I want to be able to support Laurent's future efforts 
as well.


+ Prepare to re-license the project as BSD when I approach the 0.2 
release, or approximately 244 entries.  The entire point of the current 
MPL2.0 license was to make contributions "sticky" enough to the project 
until such point that it had some critical mass. When I reach over 240+ 
definitions, the project should be able to accommodate the needs of a 
majority of people, so I won't have the same kind of need anymore, and 
can re-license the scripts to something much more permissive.


+ Once my project is BSD licensed, I might be able to merge / hybridize 
/ collaborate-on some of Toki's work (see below), completely supplanting 
the existing sv/.run/run-sh arrangement. Or not.  Or, something...we'll 
just have to wait and see.


+ Support a modified version of usersv[1].  While scripted support for 
user-defined process management will still be present, it would be nice 
to see usersv expanded to support other frameworks, and not just runit 
alone.  This would give a passive (admin controlled) or active (user 
controlled) option.


[1] https://github.com/eichlan/usersv

Is it worth having shell-agnostic ./run and ./finish?

[Avery Payne]

I'm asking because I went through some trouble to make my project 
somewhat shell-agnostic.  The intent is I can have separate support for 
/bin/sh, execline, or some other scripting environment, should it be 
desired.  But it's only worth the effort if I can get off of the 
sh-as-launcher dependency completely.  I can see how execline can be 
used to make run scripts, but I haven't tried making a finish (nor have 
I had a real need for ./finish scripts in shell).  So the question is: 
how difficult would it be to write a ./finish script in execline?  The 
answer will determine if it's worth the effort to avoid hard 
dependencies on /bin/sh.  If it's easy enough, then I go ahead with this 
plan.  If there are difficulties...well...


At  the moment, all of the ./run scripts for a service definition are 
simply symlinks to another symlink, which points to a script. Example:


/etc/sv/daemon/run >points to> /etc/sv/.run/run >points to> 
/etc/sv/.run/run.sh


Change the middle symlink and you get, for example:

/etc/sv/daemon/run >points to> /etc/sv/.run/run >points to> 
/etc/sv/.run/run.execline


That way you don't have to change all of the ./run symlinks in hundreds 
of definitions, you just change the one.

POLL: what installations would you use process supervision in?

[Avery Payne]

This is a simple straw poll.  Please do *not* reply to the mailing list 
- I don't want to clog it with answers.  Send the replies directly to my 
personal email address instead.  The poll will remain open until March 
31, and I will publish results after that time.


POLL: what installations would you use process supervision in?

[ ] A hand-made / hand-customized Linux installation
[ ] A commercial installation (HP-UX, AIX, Pre-Oracle Solaris)
[ ] an installation made with LFS
[ ] an installation made with Gentoo
[ ] an installation made with Arch
[ ] an installation made with Debian / Ubuntu
[ ] an installation made with Fedora / Red Hat
[ ] an installation made with NetBSD/OpenBSD/FreeBSD
[ ] an installation made with DragonflyBSD
[ ] an installation made with Android Open Source Project
[ ] an installation not listed here (please give name and/or details)

Re: svlogd log rotated tai64

[Avery Payne]

On 3/10/2015 4:42 PM, Sarkis Varozian wrote:

Hi All,

I'm not sure why tai64 was chosen as the default here, I suppose that is a
question for another time,

I'm a newcomer here, so take what I say with a large grain of salt.

Near as I can tell, by using TAI64, you avoid date-time conversion 
issues that have been around for awhile.  You can read more here: 
http://cr.yp.to/proto/utctai.html

Re: [NEWS/ANNOUNCE] Supervision Scripts Framework

[Avery Payne]

Supervision Scripts Framework is focused on a feature-rich drop-in set 
of scripts and support for runit and s6, although I think Toki has since 
added support for daemontools-alike supervisors.  The following list is 
based on what little I know about it, so please take it with a grain of 
salt if I misunderstand or mis-state something here:


It features OpenRC integration, single-file-per-definition 
configuration, "no-compile" installation, getty support, emerging cgroup 
support, the ability to easily roll your own definitions, and support 
for both the startup and shutdown phases.  The work has a Gentoo 
influence but it should be portable to other OpenRC environments.  Also 
there are a handful of base definitions for daemons, but it should be 
easy to add more as needed.  Toki's project is /rapidly /evolving on 
github, under a BSD 2-clause license:


https://github.com/tokiclover/supervision-scripts


The supervision-scripts project evolved out of an earlier project, 
providing a portable supervision/distro/kernel/userland-neutral set of 
definitions for use with daemontools (and daemontool-alikes), runit, and 
s6.  It currently features 70+ definitions, configuration by envdir, 
"no-compile" installation, getty support, the ability to easily roll 
your own definitions, admin-managed user-controlled daemons, and 
optional peer-level script-based daemon dependency startup.  It does NOT 
include startup or shutdown support, nor are there plans to include it.  
There's a list of planned features; each release will include 
approximately 122 additional definitions. *The project is in its infancy 
and is subject to major changes until the 0.1 release*.


My project is moving at a /leisurely/ pace on bitbucket with a mirror on 
github, under an MPL 2.0 license (minus rider "B").  NOTE: The license 
will switch to BSD once it reaches the 0.2 release, which will contain 
244 definitions:


primary: https://bitbucket.org/avery_payne/supervision-scripts
mirror: https://github.com/apayne/supervision-scripts



On 3/6/2015 9:12 AM, John Albietz wrote:

Excited to hear about all these projects. Are there links to the projects where 
I can find more information and also perhaps contribute? Thanks!

- John




On Mar 5, 2015, at 11:52 AM, Avery Payne  wrote:




On 3/5/2015 2:03 AM, toki clover wrote:
Hello fellow Supervision Users,

I was busy lately working on Supervision-Scripts[1] Framework which grew
up to be quite complete and efficient with a very simple API.

Well done!

HISTORY:

First, I only started working on this after I discovered Avery Payne's
lately on January when I was considering switching to Runit as an init system to
replace SysVinit. So, I took my time first... and then quickly transformed
Avery's plan into a simple and clean implementation with fewer directories
and files (run/finish). And finaly, I took another directory with merging
bits together aiming to get fewer files and comlexity by using a shell and 
OpenRC
for system boot/halt.

Because of the project names, I know there will be some confusion. For those 
wondering,  Toki's is named *Supervision Scripts Framework*; my project is 
named *supervision-scripts*.  They have similar names and internal concepts but 
very different goals.  If you're running a system with OpenRC and want a 
drop-in ready-to-go framework, give Toki's a try.

Re: [NEWS/ANNOUNCE] Supervision Scripts Framework

[Avery Payne]

On 3/5/2015 2:03 AM, toki clover wrote:

Hello fellow Supervision Users,

I was busy lately working on Supervision-Scripts[1] Framework which grew
up to be quite complete and efficient with a very simple API.

Well done!

HISTORY:

First, I only started working on this after I discovered Avery Payne's
lately on January when I was considering switching to Runit as an init system to
replace SysVinit. So, I took my time first... and then quickly transformed
Avery's plan into a simple and clean implementation with fewer directories
and files (run/finish). And finaly, I took another directory with merging
bits together aiming to get fewer files and comlexity by using a shell and 
OpenRC
for system boot/halt.
Because of the project names, I know there will be some confusion. For 
those wondering,  Toki's is named *Supervision Scripts Framework*; my 
project is named *supervision-scripts*.  They have similar names and 
internal concepts but very different goals.  If you're running a system 
with OpenRC and want a drop-in ready-to-go framework, give Toki's a try.

supervision scripts, 2015-02

[Avery Payne]

A household move consumed most of January and February, so not much was 
accomplished until later in the month.  Un-boxing things tends to do that...


Done:
- - - - - - - -
+ New definitions: lighttpd, minissdpd, knockd, nscd.

+ Merged all run templates that utilize /bin/sh into a single, master 
template at sv/.run/run-sh.  That master script now supports run 
directories and chmod/chown.  Other requirements will be placed there as 
well.


+ Use double indirection for ./run files to allow switching of template 
environments is complete for run scripts and logging.


+ All fgetty/agetty/mingetty are now converted to the envdir format.

+ Service definition names can now be changed to support runit's SysV 
shim mode.  NOTE: this feature breaks compatibility with peer-dependency 
support if you don't update the ./needs directory after a rename of a 
service.


In Progress:
- - - - - - - -
+ Revise the documentation.  The README file will be split into separate 
sections and placed into the doc/ directory.


+ Add design notes to the documentation.  There are a LOT of assumptions 
that aren't spelled out and people are misunderstanding why I have such 
a quirky & bland set of scripts.  This would help to explain the thought 
process behind all of it; it will also allow some (hopefully 
constructive and positive) criticism about the design.


+ Figure out how to get finish scripts fit into the schema for switching 
between sh/execline/perp (see below for details on how perp fits in).  
I'm not sure how this will turn out.


+ Clean up untested definitions, as part of the push for the 0.1 
release.  Many of the definitions have met the minimum test requirements 
and I've not cleared out the "untested" marker.  The current criteria 
for being tested is (a) the service launches cleanly (b) the service 
does not complain beyond warnings in its logs (c) the service appears to 
function as intended.


+ Clean up all remaining one-off definitions, as part of the push for 
the 0.1 release.  A bit of work has already been done in this regard, 
but there are still some that need to be brought up to current standards.


+ Finish the logging schema, as part of the push for the 0.1 release.  
At this point I won't have a full "logging chain" but it will have all 
the required features.  This also means proper support for daemontools 
and s6.


+ ...and of course, enough new definitions to justify a 0.1 release.  
This means at least 122 *tested* entries in sv/ must be present, which 
is about 10% of the count of init.d scripts used by Debian 7.  There are 
currently 96 entries but many are for getties, so after deducting 18 
redundant entries that leaves 78 definitions, which means I need to make 
44 entries.  Worst case, I end up with a 0.1 release candidate.



Still To-Do / Experimental:
- - - - - - - -
+ Think about ways to incorporate perp, which uses a different format of 
./run file. With the recent switch to support /bin/sh vs execline, this 
is a real possibility now.   That means sv/.run/run-sh, 
sv/.run/run-execline, and now sv/.run/run-perp will become potential 
targets for sv/.run/run.  Madness, I tell you, madness!


+ Examine nosh.  This is going take a bit of time to digest...

+ Re-think/re-work the ./needs directory.  While nosh already has a 
similar concept, I want to be able to support Laurent's future efforts 
as well.


+ Prepare to re-license the project as BSD when I approach the 0.2 
release, or approximately 244 entries.  The entire point of the current 
MPL2.0 license was to make contributions "sticky" enough to the project 
until such point that it had some critical mass.  When I reach over 240+ 
definitions, the project should be able to accommodate the needs of a 
majority of people, so I won't have the same kind of need anymore, and 
can re-license the scripts to something much more permissive.


+ Once my project is BSD licensed, I might be able to merge / hybridize 
/ collaborate-on some of Toki's work (see below), completely supplanting 
the existing sv/.run/run-sh arrangement.  Or not.  Or, something...we'll 
just have to wait and see.


+ Support a modified version of usersv[1].  While scripted support for 
user-defined process management will still be present, it would be nice 
to see usersv expanded to support other frameworks, and not just runit 
alone.  This would give a passive (admin controlled) or active (user 
controlled) option.


[1] https://github.com/eichlan/usersv

P.S. Toki Clover has been blazing a path with his own take on 
supervision scripts for OpenRC.  While there aren't nearly as many 
definitions, the scripts are nearing completion, and I'm sure it won't 
take much to extract the daemon settings from my project and port them 
into his.  Well done Toki!

Re: patch: sv check should wait when svrun is not ready

[Avery Payne]

On 2/17/2015 11:02 AM, Buck Evan wrote:

I think there's only three cases here:

 1. Users that would have gotten immediate failure, and no amount of 
spinning would help. These users will see their error delayed by 
$SVWAIT seconds, but no other difference.
 2. Users that would have gotten immediate failure, but could have 
gotten a success within $SVWAIT seconds. All of these users will of 
course be glad of the change.
 3. Users that would not have gotten immediate failure. None of these 
users will see the slightest change in behavior.


Do you have a particular scenario in mind when you mention "breaking 
lots of existing installations elsewhere due to a default behavior 
change"? I don't see that there is any case this change would break.


I am not so much thinking of a specific case as I am looking at it from 
an integration perspective.  I ask that you indulge me for a moment, and 
let me diverge from the discussion so I can clarify things.


My background is in maintaining business software.  My employer has the 
source code to their ERP system and I make large and small modifications 
to adapt to changing business needs.  During the process of working on 
"legacy code" in a "legacy language", I have to be mindful that there 
are side-effects to each change; I have to look at it from a viewpoint 
of "what is everything else in the system expecting when this code is 
called".  This means thinking in terms of code-as-API, so that calls 
elsewhere don't break.   Yes, I am aware of unit tests, etc., but trust 
me when I say it's not an option for the environment.  So that means 
lots and lots of careful testing by hand, and being very mindful of how 
things fit together.


With that viewpoint in mind, let's turn back to my words, which were 
admittedly overstated.  When I said "breaking lots of existing 
installations" I was trying to describe a point of view, for I was 
looking at it from a pragmatic standpoint of "if there is code out there 
that expects behavior X, but is given behavior Y, then the probability 
of something breaking increases".  From my point of view, when you run 
"sv check (something)", that's no different that making an API call 
because "sv check (something)" typically happens inside of a script, 
which in turn implies a language and environment.  The behavior of the 
"sv check" call, and specifically the side-effects of it, are taken into 
consideration "elsewhere"; I can't say where else because I can't see 
specific installations, and it's entirely possible that there is 
*nothing* out there that would be broken, and I'm writing this all for 
naught.  But the point remains - the API is set, the behavior of the 
"call" is set, and deviating from that requires that everyone downstream 
make changes to ensure that their scripts don't break.


So, I think it becomes a question of "can I guarantee that the side 
effect created by the change will not adversely impact something else, 
since I can't directly observe what will be impacted?"  Which is why I 
suggested the option switch.  Introducing a new switch means that the 
existing behavior will be kept, but we can now use the new behavior by 
explicitly asking for it.  In effect, we're extending our API without 
breaking existing "calls from legacy code".


The only example I could give would be my own project at the moment, 
although what follows is admittedly a weak argument. Blocking-on-check 
would pretty much destroy the script work I've done for peer-based 
dependency management, because a single dependency would cause the 
"parent" service to hang while it waited for the "child" to come up.  
This happens because the use of "sv check (child)" follows the 
convention of "check, and either succeed fast or fail fast", and the 
parent's script is written with the goal of exiting out because of a 
child's failure.  Each fail-to-start is logged in the parent, so it's 
clear that parent service X failed because child service Y was not 
confirmed as running.  Without that fast-fail, the logged hint never 
occurs; the sysadmin now has to figure out which of three possible 
services in a dependency chain are causing the hang.  While this is 
implemented differently from other installations, there are known cases 
similar to what I am doing, where people have ./run scripts like this:


#!/bin/sh
sv check child-service || exit 1
exec parent-service

A secondary example would be that the existing set of scripts in the 
project are written with an eye towards supporting three environments, 
which is possible due to their similar behavior.  This consistency makes 
the project possible for daemontools and s6, as well as runit.  A change 
in runit's behavior implies that I can no longer rely on that consistency.


Perhaps I am understanding the environment clearly but misunderstanding 
the intent of the change.  If I am not grokking your intentions, just 
send a short note to the effect of "sorry, wrong idea" and I'll stop. :)

Re: patch: sv check should wait when svrun is not ready

[Avery Payne]

On 2/16/2015 5:08 PM, Buck Evan wrote:

My check is running from outside the docker, racing against runit.


I'm not clear on this.  Am I understanding this correctly?

+ You have a service inside of a docker container
+ The service in the container is managed by runsv (which may or may not 
be under runsvdir)

+ You start the container
+ You start the service in the container (or it auto-starts as part of 
the container)
+ Your *host* for the container is attempting to check the status of the 
service inside the *guest* container

+ The check for the service is a ./check script inside the *guest*
+ You expect the ./check to block while waiting for the service, to 
confirm the service is ready and not "just starting".


If this a correct understanding, consider using some intermediary 
program at the *host* level to run "sv check some-service" inside of the 
*guest* container, and then have the intermediary report back to the 
*host*.  This would remove the need for a patch; the intermediary could 
block-while-waiting, etc. or perform any other strategy you need.  Added 
bonus: by having a single intermediary integrated with docker, you can 
re-use it with other docker containers.


If you really need the patch, I humbly suggest that you make the 
wait-for-check behavior into an option flag, instead of setting it as 
the default behavior.  You would gain your desired outcome without 
breaking lots of existing installations elsewhere due to a default 
behavior change.

Re: patch: sv check should wait when svrun is not ready

[Avery Payne]

On 2/16/2015 3:56 PM, Colin Booth wrote:

If you are running the sv check within a dependent service's run
script that was spawned first, the script will fail and runsv will
respawn it, with the call eventually succeeding once the required
service is up.

If you're calling this from a daemon script that is outside of the
general supervision framework, you'll need to write a simple wait loop
in shell to wrap the sv call similar to what Case suggested.
A third option is to loop the list of "child" dependencies and perform 
the check for each one.  If a check returns a failure, then fail the 
"parent" script and let the supervisor restart it.  You can then have 
./finish inject a quick message into the log about the failure.

Re: mailing list archive semi-broken

[Avery Payne]

On 2/10/2015 10:16 AM, Buck Evan wrote:

There's something preventing this message from being rendered:

http://skarnet.org/cgi-bin/archive.cgi?2:495

The one before it is fine:

http://skarnet.org/cgi-bin/archive.cgi?2:494

The one after is broken:

http://skarnet.org/cgi-bin/archive.cgi?2:496

But the one after that is fine again:

http://skarnet.org/cgi-bin/archive.cgi?2:497



I would guess that there's some non-ascii content in there causing an
exception.

It's been noted a few months back.  Laurent was a bit busy at the time 
and couldn't look into it.  Check the archives at 
http://www.mail-archive.com/supervision@list.skarnet.org/ or better yet, 
just subscribe to the list and file the messages into a separate folder, 
which is what I do.

Re: runit source control?

[Avery Payne]

On 2/10/2015 9:58 AM, Steve Litt wrote:

On Mon, 09 Feb 2015 19:22:26 +0100
Laurent Bercot  wrote:


On 09/02/2015 18:50, Buck Evan wrote:

Is there truly no public access to the source control for runit?
If so, it's decidedly odd.

   Not that odd.
   Small projects don't really need an SCM. Most pre-git SCMs are
cumbersome or impractical. runit predates git. Gerrit chose to
focus on the code, not on the packaging - that's all there is
to it.

I totally agree. As long as a project is primarily a one man band,
there's no need for all that stuff. It's probable that other peoples
contributions can be sent in as patches against the last tarballed
version.

The only reason I have *two* public repositories is, I'm after 
contributors. I really didn't want to do github, but I gave in so that 
others using git could contribute.  Having to sync two different repos 
isn't much fun, and can make some of the pull requests a little messy.


I do use Mercurial for other reasons, though.  Using the patch extension 
makes it easy to write a prototype, and leave it unapplied in the patch 
queue until I'm ready to refine and use it.

supervision scripts, 2015-01

[Avery Payne]

Lots of changes behind the scenes.  Not as many new definitions, 
although things will return to "normal" in the next month.



Done:
- - - - - - - -
+ New! Internal framework-neutral command grammar.  Upon selecting a 
given framework, all scripts automatically use the correct start and 
check commands.


+ New! "Broken" daemon support.  Daemons without proper "background" 
support now have a warning written to the daemon's log, and the run 
script aborts.  If you want to continue with the daemon, remove the 
./broken file from the definition and the script will launch.  Typically 
this is used for services that don't support foreground operation.


+ New! Optional framework-neutral peer-level dependency handling.All 
dependencies will be included out-of-the-box. Dependencies are defined 
as any external program required to be running before the daemon defined 
can launch. It is recommended that it only be used when no dependency 
manager is available older frameworks, and you desire automatic start-up 
of dependent definitions.  It is disabled by default, because it comes 
with certain caveats, notably: (a) you cannot change definition names 
and expect it to work, and (b) it provides a weak guarantee that 
dependencies are running, so it is possible to have race conditions or 
other failure states.


+ New! Dynamic run state directory creation.  Definitions will now 
create /var/run/{daemon-name} upon startup with the correct permissions.


+ New definitions: ypbind, autofs, rsyslog, mini-httpd, rpcbind, 
shellinabox, slpd, gdomap, rarpd


+ Reworked definitions: forked-daapd, avahi-daemon, dbus, mpd, mouseemu, 
ntpd, munin-node (broken)


+ Several tweaks to the README file, including a compatibility chart for 
user-land tools.



In Progress:
- - - - - - - -
+ Merged all run templates that utilize /bin/sh into a single, master 
template.  This was done after much deliberation - why spend time 
figuring out which one of a half-dozen different template scripts will 
have the feature you need, when you could just link one script and get 
on with life?


+ Use double indirection for ./run files to allow switching of template 
environments, i.e. it makes it possible to switch between /bin/sh and 
another arrangement.  The ./run file in the definition will point to 
../.run/run, which itself points to the actual template.


+ Look at supporting execline in parallel, although features like 
script-based dependency management will stop working.


+ Revisit all fgetty/agetty/mingetty and re-code them to support ./env 
settings.


+ Revisit all existing one-off scripts and see if they now qualify to be 
used with a template instead, and re-write them if possible.


+ Examine nosh to see if it can be supported as well.

+ Final push for 0.1 release, which will include 10% of defined 
definitions in Debian 7, all run templates stabilized, and all logging 
stabilized.



To-Do / Experimental:
- - - - - - - -
+ Think about ways to incorporate perp, which uses a different format of 
./run file.


... plus all of the To-Do stuff from last month

As always, suggestions, comments, and ideas are welcome.

Re: Could s6-scscan ignore non-servicedir folders? [provides-needs deps]

[Avery Payne]

On 1/22/2015 9:08 AM, post-sysv wrote:

On 01/22/2015 01:33 AM, Avery Payne wrote:
This brings to mind the discussion from Jan. 8 about "./provides", 
where a defining a daemon implies:


* the service that it actually provides (SMTP, IMAP, database, etc.); 
think of it as the "doing", the piece that performs work


* a data transport (pipe, file, fifo, socket, IPv4, etc.); think of 
it as how you connect to it


* a protocol (HTTP, etc.); think of it as a grammar for conversing 
with the service, with vertical/specific applications like MySQL 
having their own grammars, i.e. MySQL-3, MySQL-4, MySQL-5, etc. for 
each generation that the grammar changes.


I'm sure there are other bits and pieces missing.  With regard to 
relationships, if you had a mapping of these, it would be a start 
towards a set of formal (although incomplete) definitions.  From that 
you could say "I need a database that speaks MySQL-4 over a file 
socket" and you could, in theory, have a separate program bring up 
MySQL 4.01 over a file socket when needed.


But do we really need this?


The provides-needs relationship is one I've pondered myself (and it's 
how GNU dmd works),

but once again it stumbles conceptually the more I think of it.
Which is why I asked if we really need this.  I tried really hard to 
think of a good use case, and the only one I could come up with is 
similar to what you describe - basically, this would be used in 
conjunction with some kind of super-server.
Once again, provides-needs can be adequately hacked in by doing 
"start-single-instance foobard ||
exit 1", something to the effect of it. You're adding a thin and 
useless layer of sugar just to avoid

mixing some code with your configuration.

I'm not even sure we need to have much code to support this - see below.


Or you just order your services descendingly in the dependency chain, 
yet again.
This is what I tried to accomplish with the script-driven approach, 
using a "pull" rather than a "push".  Each service definition only knows 
what it "needs"; nothing else is furnished.  Needs are a directory in 
the definition named "./needs", and in that directory, a symlink 
pointing to each service definition that fulfills a "need".  As a 
service definition starts, it asks for those "needs" to come up; if they 
can't, then the definition can't come up and is considered to be 
failing.  When the definition asks for a "need", it does so by using a 
standard "servicetool up needed-service" command; it has no visibility 
of what is going on and frankly doesn't care. This was about as plain as 
I could make it, yet still work.  Looking beyond known concerns about 
"is the service really up" it has some advantages: it's fairly clean, 
low-overhead (only 1 symlink per need), it doesn't care much about 
broken or missing "needs" (because that triggers a failure anyways), it 
naturally builds dependency chains without having to explicitly spell 
them out, it can handle "trees" of dependencies, it handles instances 
where common leaf-nodes in a tree of dependencies won't clash, etc.  And 
the best part, this structure has the potential for re-use with a 
dedicated dependency manager, such as nosh, or the upcoming extensions 
to s6 - the information is unambiguous and clear, but no policy is set 
by doing it this way, beyond defining what is inside of "./needs".  So 
you could use this at a high or low level, and make it fly.
Provides-needs would work fine for a service-oriented architecture 
with explicit design and policies,
but in a generic supervision framework it just sounds unwieldy. Do try 
to describe a model where

it isn't, though.
I think the concept of a "needs" has a future, provided that it is close 
to what I described above.  But "provides" becomes a lot more vague the 
more I look at it.

Re: Could s6-scscan ignore non-servicedir folders?

[Avery Payne]

On 1/21/2015 7:19 PM, post-sysv wrote:


I'm not sure what effective and worthwhile ways there are to express 
service *relationships*,
however, or what that would exactly entail. I think service conflicts 
and service bindings might
be flimsy to express without a formal system, though I don't think 
it's anything that pre-start
conditional checks and finish checks can't emulate, perhaps less 
elegantly?


This brings to mind the discussion from Jan. 8 about "./provides", where 
a defining a daemon implies:


* the service that it actually provides (SMTP, IMAP, database, etc.); 
think of it as the "doing", the piece that performs work


* a data transport (pipe, file, fifo, socket, IPv4, etc.); think of it 
as how you connect to it


* a protocol (HTTP, etc.); think of it as a grammar for conversing with 
the service, with vertical/specific applications like MySQL having their 
own grammars, i.e. MySQL-3, MySQL-4, MySQL-5, etc. for each generation 
that the grammar changes.


I'm sure there are other bits and pieces missing.  With regard to 
relationships, if you had a mapping of these, it would be a start 
towards a set of formal (although incomplete) definitions.  From that 
you could say "I need a database that speaks MySQL-4 over a file socket" 
and you could, in theory, have a separate program bring up MySQL 4.01 
over a file socket when needed.


But do we really need this?

Fwd: Re: Could s6-scscan ignore non-servicedir folders?

[Avery Payne]

Ugh, sorry folks.  I keep forgetting to change the address. Forwarded a 
copy.


 Forwarded Message 
Subject:Re: Could s6-scscan ignore non-servicedir folders?
Date:   Wed, 21 Jan 2015 11:40:58 -0800
From:   Avery Payne 
To: Olivier Brunel 


On 1/21/2015 9:24 AM, Olivier Brunel wrote:

I'll have to setup some scripts for different init stages, using
s6-svscan as stage 2, as you've described elsewhere. But I also want to
have a system to start (and stop) services in order. I see this whole
idea of order/dependency is something that is being talked about, but
currently not supported.

Dependency handling is tricky, not from a nuts-and-bolts mechanical
perspective, but from a getting-it-right perspective.  There's older
discussion on the mailing list about this, and the short version is:
yes, it can be done, but if you don't have a good notification
mechanism, it's a very weak guarantee.  I've already tackled this with
my current set of scripts using an option flag that allows you to have
chain-dependencies for services.  It works fine, but there's no hard
guarantee that it will be consistent.



Furthermore, I want this system of mine to include other kinds of
services, that is one-time process/scripts that needs to be run once (on
boot), and that's it. And to make things simpler, I want to have it all
work together, mixing longrun services (s6 supervised) and oneshot
services when it comes to dependency/order definition.

So I'll have servicedir of sorts, for oneshot services. And I'm planning
of having one folder, that I tend to call runtime repository, but that
would also be the scandir for s6-svscan.

Obviously though, those aren't servicedirs in the s6 meaning, they
shoudln't be supervised, so I'd like for s6-svscan to check if a folder
does in fact have a file run, and if not to simply skip/ignore it.

That way I can have all my (longrun & oneshot) servicedirs under one
parent, and it shouldn't really break anything, since a folder without a
run file would not be really useful to supervise anyways, as it would
just try & fail to start it every 10 seconds.

The only case I see would be a folder created, scanned, and only
afterwards the run script be copied in there. But that sounds like a
very unlikely/rare scenario. (And in case it happened, one could just
trigger a rescan to fix it.)

So, what do you think of this? Would you be willing to have s6-svscan
ignore folders not containing a run file?

One-shot is something I've been mulling over for about two months now.
There isn't a quick and elegant way to just make it work.  But I might
have a solution, based on what I am currently thinking about doing in
supervision-scripts.   Here's a variant of that thought, for use with
"startup":

1. Create a service definition that has a s6-svscan launch in it. We
will call it "startup".
2. Mark the "startup" definition with a ./down file.
3. Create your separate, sequenced launches in a separate directory that
is not in your /service directory.  We will call it "startup-settings".
4. In "startup-settings" place all of your one-shot scripts to be run at
initialization.  Mark all of them with ./down files.
5. Use dependency resolution to chain the scripts inside of
"startup-settings" together, with the top-most scripts NOT having a
./down file.  Currently nosh allows for this, and s6 will have it in the
near future.  You can emulate it with supervision-scripts, but only as a
last resort, as it relies heavily on return codes and ./check scripts.
6. When s6 launches, do a "s6-svc start startup".  The "startup"
definition will wake up and launch s6-svscan.  The scan process will
bring up all of the definitions that do NOT have a ./down file in
"startup-settings".  Keep in mind that "startup-settings" is a separate
directory, but "startup" will be part of the existing supervision tree.
Those definitions in "startup-settings" will turn around and call all of
the others that are marked ./down with "s6-svc once (whatever)" - that
is their sole job in life.  Now for the heavy magic: Because the "run
once" definitions are dependency-chained, all of those will launch in
the correct sequence, in parallel.  Visualize it as a set of trees, with
the base of each tree as the non-down definition, and each branch or
leaf as a down definition.  Each branch, by virtue of dependency, calls
the next branch or leaf that it needs, waiting for its children to be
"up".  As each  leaf on the tree finishes, the leaf calls itself with
"s6-svc down (itself)", and the ./down file is left as-is.  This means
each leaf or branch will clean up after itself and will not re-launch.
Also, as each branch or leaf terminates, the "parent" will receive
notice 

Re: thoughts on rudimentary dependency handling

[Avery Payne]

On 1/19/2015 2:31 PM, Jonathan de Boyne Pollard wrote:

Avery Payne:
> * implement a ./wants directory.  [...]
> * implement a ./needs directory.  [...]
> * implement a ./conflicts directory.  [...]

Well this looks familiar.


I ducked out of ./needs and ./conflicts for the time being; if I spend 
too much time with making those features then the scripts won't move 
forward.  I'm already several weeks "behind" in my own schedule that I 
have set for scripts.




Before you read further, including to my next message, get yourself a 
copy of nosh and read the manual pages therein for service-manager(1) 
and system-control(1), paying particular attention in the latter to 
the section entitled "Service bundles".


Then grab the nosh Guide and read the new interfaces chapter.  On a 
Debian system this would be:


xdg-open /usr/local/share/doc/nosh/new-interfaces.html


Sounds like I have my homework cut out.  I will do so as soon as I can, 
although I warn you that it joins an already-long list of material to 
read and think about.

Re: first round of optional dependency support

[Avery Payne]

I just realized what you might have meant.  Sorry, it's late where I'm at.

Ok, so you want a guarantee that if A starts B and B starts C, that A will
only be up with B and C.  There is a weak guarantee here; B will only start
fully when C starts, so A will block B if B is waiting for C.  So yes,
although it's not a firm guarantee.  If you go back and read the discussion
from a little while back, you'll see there are limitations to this and it's
an option "of last resort".

On Thu, Jan 15, 2015 at 9:16 PM, James Powell  wrote:

>  Service scripts often need a lot of setup code before the actual daemon
> is executed. My question is, does it provide a fail-safe solution to
> dependency trees?
>
> Shutdown is only an issue if you need a finish script, otherwise the
> service supervisor will execute the kill signal and bring things down.
>
> Sent from my Windows Phone
>  --
> From: Avery Payne 
> Sent: ‎1/‎15/‎2015 9:11 PM
> To: supervision@list.skarnet.org
> Subject: first round of optional dependency support
>
>  Ok, admittedly I'm excited because it works.
>
> The High Points:
>
> + It works (ok, yeah, it took me long enough.)
> + Framework-neutral grammar for bringing services up and checking them, no
> case-switches needed
> + Uses symlinks (of course) to declare dependencies in a tidy ./needs
> directory
> + Can do chain dependencies, where A needs B, B needs C, C starts as a
> consequence of starting A
> + Chain dependencies are naive, having no concept of each other beyond what
> is in their ./needs directory, so you do NOT need to declare the kitchen
> sink when setting up ./needs
> + Is entirely optional, it is off by default, so you get the existing
> behavior until enabled
> + Simple activation, you enable it by writing a "1" to a file
> + Smart enough to notice missing definitions or /service entries, a script
> will fail until fixed
>
> The So-So Points:
>
> ~ Framework grammar makes the working assumption that it follows a
> tool-command-service format.  This might be a problem for future frameworks
> that require tool-service-command or other grammars.
> ~ Some distro maintainers may have situations where they compile out
> something that will be defined in a ./needs, or may compile in something
> that is missing from ./needs; this mismatch will bring tears, but for now,
> I am assuming that things are "sane enough" that these inter-dependencies
> will remain intact.
> ~ I'm not happy with handling of ./env settings, it could have been cleaner
> ~ Oversight of dependencies is based on the assumption that the supervisor
> for the dependency will keep the service propped up and running.
> ~ Once enabled, you need to start or restart services.  It doesn't affect
> running services.
> ~ Currently starting a scripts sends the commands up, then check.  Maybe it
> should do check, then up, then check?  That feels wrong - at what point
> does it turn into turtles all the way down?
>
> The Low Points:
>
> - Not true dependency management.  It only tackles start-up, not shut-down,
> and won't monitor a chain of dependencies for failures or restarts.
> - Enormous code bloat.  By the time I finished with the bulk of exception
> handling, I felt like I ran a marathon...twice.  The resulting script is
> *multiple* times the size of the others.
> - The number of dependent commands needed in user-space to run the script
> has gone upway up.  Every additional user-space tool included is
> another "does your install have X" that ultimately limits things -
> especially embedded devices.  Did I mention bloat earlier?
> - Way too many failure tests, which means...way too many failure paths.
> This makes testing much harder.
> - There's a bug (or two) lurking in there, my gut tells me so
> - Relative pathing is fine for a static install inside of /etc, but what
> happens when users try to spawn off their own user-controlled services?  I
> smell a security hole in the making...
>
> The Plan:
>
> This will become a part of avahi-daemon and forked-daapd definitions, but
> disabled by default.  From everyone else's perspectives, it will function
> like it always did, until enabled.  With sv/.env/ENABLE_NEEDS set to 1, for
> example, a launch of forked-daapd will bring up avahi-daemon, and
> avahi-daemon will bring up dbus.
>
> Constructive criticism welcome.  I ask that Laurent leaves his flamethrower
> at home - the urge to burn it with fire to purify the project may be
> strong. ;)
>

Re: first round of optional dependency support

[Avery Payne]

Depends on what you call fail-safe.

If you mean "will only run a service if all dependencies met", then yes.
The current mode is to log the dependency failure and exit the script.  The
service will of course run-loop until fixed; this isn't far from the
current non-dependency behavior anyways.  The emitted warnings in the log
should be descriptive enough to diagnose the problem and remedy it.

If you mean "attempt run regardless of dependencies", then no.  In my eyes,
that would be a ./wants directory, and I debated doing that as well.  I
figured, ./wants is a variant of ./needs, so I should get ./needs right
first.  Once I know ./needs is stable, I can adapt it, removing a lot of
the failure code, and it turns into ./wants, i.e. "attempt to start the
dependency, but if it doesn't start, don't worry about it, keep going"


On Thu, Jan 15, 2015 at 9:16 PM, James Powell  wrote:

>  Service scripts often need a lot of setup code before the actual daemon
> is executed. My question is, does it provide a fail-safe solution to
> dependency trees?
>
> Shutdown is only an issue if you need a finish script, otherwise the
> service supervisor will execute the kill signal and bring things down.
>
> Sent from my Windows Phone
>  --
> From: Avery Payne 
> Sent: ‎1/‎15/‎2015 9:11 PM
> To: supervision@list.skarnet.org
> Subject: first round of optional dependency support
>
>  Ok, admittedly I'm excited because it works.
>
> The High Points:
>
> + It works (ok, yeah, it took me long enough.)
> + Framework-neutral grammar for bringing services up and checking them, no
> case-switches needed
> + Uses symlinks (of course) to declare dependencies in a tidy ./needs
> directory
> + Can do chain dependencies, where A needs B, B needs C, C starts as a
> consequence of starting A
> + Chain dependencies are naive, having no concept of each other beyond what
> is in their ./needs directory, so you do NOT need to declare the kitchen
> sink when setting up ./needs
> + Is entirely optional, it is off by default, so you get the existing
> behavior until enabled
> + Simple activation, you enable it by writing a "1" to a file
> + Smart enough to notice missing definitions or /service entries, a script
> will fail until fixed
>
> The So-So Points:
>
> ~ Framework grammar makes the working assumption that it follows a
> tool-command-service format.  This might be a problem for future frameworks
> that require tool-service-command or other grammars.
> ~ Some distro maintainers may have situations where they compile out
> something that will be defined in a ./needs, or may compile in something
> that is missing from ./needs; this mismatch will bring tears, but for now,
> I am assuming that things are "sane enough" that these inter-dependencies
> will remain intact.
> ~ I'm not happy with handling of ./env settings, it could have been cleaner
> ~ Oversight of dependencies is based on the assumption that the supervisor
> for the dependency will keep the service propped up and running.
> ~ Once enabled, you need to start or restart services.  It doesn't affect
> running services.
> ~ Currently starting a scripts sends the commands up, then check.  Maybe it
> should do check, then up, then check?  That feels wrong - at what point
> does it turn into turtles all the way down?
>
> The Low Points:
>
> - Not true dependency management.  It only tackles start-up, not shut-down,
> and won't monitor a chain of dependencies for failures or restarts.
> - Enormous code bloat.  By the time I finished with the bulk of exception
> handling, I felt like I ran a marathon...twice.  The resulting script is
> *multiple* times the size of the others.
> - The number of dependent commands needed in user-space to run the script
> has gone upway up.  Every additional user-space tool included is
> another "does your install have X" that ultimately limits things -
> especially embedded devices.  Did I mention bloat earlier?
> - Way too many failure tests, which means...way too many failure paths.
> This makes testing much harder.
> - There's a bug (or two) lurking in there, my gut tells me so
> - Relative pathing is fine for a static install inside of /etc, but what
> happens when users try to spawn off their own user-controlled services?  I
> smell a security hole in the making...
>
> The Plan:
>
> This will become a part of avahi-daemon and forked-daapd definitions, but
> disabled by default.  From everyone else's perspectives, it will function
> like it always did, until enabled.  With sv/.env/ENABLE_NEEDS set to 1, for
> example, a launch of forked-daapd will bring up avahi-daemon, and
> avahi-daemon will bring up dbus.
>
> Constructive criticism welcome.  I ask that Laurent leaves his flamethrower
> at home - the urge to burn it with fire to purify the project may be
> strong. ;)
>

first round of optional dependency support

[Avery Payne]

Ok, admittedly I'm excited because it works.

The High Points:

+ It works (ok, yeah, it took me long enough.)
+ Framework-neutral grammar for bringing services up and checking them, no
case-switches needed
+ Uses symlinks (of course) to declare dependencies in a tidy ./needs
directory
+ Can do chain dependencies, where A needs B, B needs C, C starts as a
consequence of starting A
+ Chain dependencies are naive, having no concept of each other beyond what
is in their ./needs directory, so you do NOT need to declare the kitchen
sink when setting up ./needs
+ Is entirely optional, it is off by default, so you get the existing
behavior until enabled
+ Simple activation, you enable it by writing a "1" to a file
+ Smart enough to notice missing definitions or /service entries, a script
will fail until fixed

The So-So Points:

~ Framework grammar makes the working assumption that it follows a
tool-command-service format.  This might be a problem for future frameworks
that require tool-service-command or other grammars.
~ Some distro maintainers may have situations where they compile out
something that will be defined in a ./needs, or may compile in something
that is missing from ./needs; this mismatch will bring tears, but for now,
I am assuming that things are "sane enough" that these inter-dependencies
will remain intact.
~ I'm not happy with handling of ./env settings, it could have been cleaner
~ Oversight of dependencies is based on the assumption that the supervisor
for the dependency will keep the service propped up and running.
~ Once enabled, you need to start or restart services.  It doesn't affect
running services.
~ Currently starting a scripts sends the commands up, then check.  Maybe it
should do check, then up, then check?  That feels wrong - at what point
does it turn into turtles all the way down?

The Low Points:

- Not true dependency management.  It only tackles start-up, not shut-down,
and won't monitor a chain of dependencies for failures or restarts.
- Enormous code bloat.  By the time I finished with the bulk of exception
handling, I felt like I ran a marathon...twice.  The resulting script is
*multiple* times the size of the others.
- The number of dependent commands needed in user-space to run the script
has gone upway up.  Every additional user-space tool included is
another "does your install have X" that ultimately limits things -
especially embedded devices.  Did I mention bloat earlier?
- Way too many failure tests, which means...way too many failure paths.
This makes testing much harder.
- There's a bug (or two) lurking in there, my gut tells me so
- Relative pathing is fine for a static install inside of /etc, but what
happens when users try to spawn off their own user-controlled services?  I
smell a security hole in the making...

The Plan:

This will become a part of avahi-daemon and forked-daapd definitions, but
disabled by default.  From everyone else's perspectives, it will function
like it always did, until enabled.  With sv/.env/ENABLE_NEEDS set to 1, for
example, a launch of forked-daapd will bring up avahi-daemon, and
avahi-daemon will bring up dbus.

Constructive criticism welcome.  I ask that Laurent leaves his flamethrower
at home - the urge to burn it with fire to purify the project may be
strong. ;)

redoing the layout of things

[Avery Payne]

On Thu, Jan 8, 2015 at 3:08 PM, Luke Diamand  wrote:

> On 08/01/15 17:53, Avery Payne wrote:
>
>> The use of hidden directories was done for administrative and aesthetic
>> reasons.  The rationale was that the various templates and scripts and
>> utilities shouldn't be mixed in while looking at a display of the various
>> definitions.
>>
>
> Why shouldn't they be mixed in? Surely better to see everything clearly
> and plainly, than to hide some parts away where people won't expect to find
> them. I think this may confuse people, especially if they use tools that
> ignore hidden directories.
>

Ok, I'll take this as part of the consideration.


> Move everything down one level then?
>

I've given it a bit of thought.  I would be willing to remove the dots.
However, the current naming convention would create confusion if you were
to eliminate the support directories altogether.  Keep in mind the purpose
of the directories was to separate out functionality and clearly define
what a group of things does; a service template is vastly different from a
logging template.  The script names were meant as a reminder to how they
are used, along with the directories.  This is why there is a run-svlogd,
and not a log-svlogd.  However, I suppose I could rename things to better
match their intended use.  And while I don't want to drop the prefix (for
reasons of clarity when writing the script) as long as the directories
remain, I'm willing to drop those as well.  The proposal woudl be, inside
of sv/, something like:

/bin
/bin/use-daemontools
/bin/use-runit
/bin/use-s6
/env
/env/PATH
/env/FRAMEWORK
/env/ENABLE_DEPENDS
/finish
/finish/clean
/finish/notify
/finish/force
/log
/log/multilogd
/log/svlogd
/log/s6-log
/log/logger
/log/socklog
/run
/run/envdir
/run/getty
/run/user-service
/(definition 1)
/(definition 2)

...and so on, without the dots.  I'm not wild about the "messy appearance"
it will give but if it makes adoption easier, then I'll do it.  That, and
we now have five words that are reserved and can never be used by any
service (although I doubt that a service would use any of the above),
because the names exist alongside the definitions.  That was another reason
I wanted dot-files - it was one less thing to worry about, one less issue
that needed attention.

Good thing the bulk of the defintions are symlinks...makes it easy to
switch the directory name. ;)

Re: thoughts on rudimentary dependency handling

[Avery Payne]

On Thu, Jan 8, 2015 at 9:23 AM, Steve Litt 
wrote:
>
> I'm having trouble understanding exactly what you're saying. You mean
> the executable being daemonized fails, by itself, because a service it
> needs isn't there, right? You *don't* mean that the init itself fails,
> right?
>

Both correct.


> I'm not sure what you're saying. Are you saying that the dependency
> code is in the runscript, but within an IF statement that checks
> for ../env/NEEDS_ENABLED?
>

Correct.  If the switch, which is a data value in a file, is zero, it
simply skips all of the dependency stuff with a giant if-then wrapper.  At
least, that's the plan.  I won't know until I can get to it.


> > Like I
> > said, this will be a fall-back feature, and it will have minor
> > annoyances or issues.
>
> Yes. If I'm understanding you correctly, you're only going so far in
> determinint "really up", because otherwise writing a one size fits all
> services thing starts getting way too complicated.
>

Correct.  I'm taking an approach that has "the minimum needed to make
things work correctly."


>
> I was looking at runit docs yesterday before my Init System
> presentation, and learned that I'm supposed to put my own "Really Up"
> code in a script called ./check.
>

Also correct, although I'm trying to only do ./check scripts where
absolutely needed, such as the ypbind situation.  Otherwise, the check
usually looks at "is the child PID still around".


> If I read the preceding correctly, you're making service tool calls for
> runit, s6, perp and nosh grammatically identical.


Correct.


> Are you doing that so
> that your run scripts can invoke the init-agnostic commands, so you
> just have one version of your scripts?
>

Exactly correct.  This how I am able to turn the bulk of the definitions
into templates.  ./run files in the definition directories are little more
than symlinks back to a script in ../.run, which means...write once, use a
whole lot. :)  It's also the reason that features are slow in coming - I
have to be very, very careful about interactions.


>
> However you end up doing the preceding, I think it's essential to
> thoroughly document it, complete with examples. I think that the
> additional layer of indirection might be skipped by those not fully
> aware of the purpose.
>

I just haven't gotten around to this part, sorry.


>
> I can help with the documentation.
>

https://bitbucket.org/avery_payne/supervision-scripts
or
https://github.com/apayne/supervision-scripts

Feel free to clone, change, and send a pull request.

Re: thoughts on rudimentary dependency handling

[Avery Payne]

The use of hidden directories was done for administrative and aesthetic
reasons.  The rationale was that the various templates and scripts and
utilities shouldn't be mixed in while looking at a display of the various
definitions.  The other rationale was that the entire set of definitions
could be moved or copied using a single directory, although it doesn't work
that way in practice, because a separate cp is needed to move the
dot-directories.

The basic directory structure is as follows:

sv
-> .bin
-> .env
-> .finish
-> .log
-> .run

Where:

sv is the container of all definitions and utilities.  Best-case, the
entire structure, including dot directories, could be set in place with mv,
although this is something that a package maintainer would be likely to
do.  People initially switching over will probably want to use cp while the
project develops.  That way, you can pull new definitions and bugfixes with
git or mercurial, and copy them into place.  Or you could download it as a
tarball off of the website(s) and simply expand-in-place.  So there's a few
different ways to get this done.

.bin is meant to store any supporting programs.  At the moment this is a
bit of a misnomer because it really only stores the framework shunts and
the supporting scripts for switching those shunts.  It may have actual
binaries in the future, such as usersv, or other independent utilities.
When you run use-* to switch frameworks, it changes a set of symlinks to
point to what should be the tools of your installed framework; this makes
it "portable" between all frameworks, a key feature.

.env is an environmental variable directory meant to be loaded with the
envdir tool.  It represents system-wide settings, like PATH, and some of
the settings that are global to all of the definitions.  It is used within
the templates.

.finish will hold ./finish scripts.  Right now, it's pretty much a stub.
Eventually it will hold a basic finish script that alerts the administrator
to issues with definitions not launching, as well as handling other
non-standard terminations.

.log will hold ./log scripts.  It currently has a single symlink, ./run,
that points to whatever logging system is the default.  At the moment it's
svlogd only because I haven't finished logging for s6 and daemontools.
Eventually .log/run will be a symlink to whatever loggin arrangement you
need.  In this fashion, the entire set of scripts can be switched by simply
switching the one symlink.

.run will hold the ./run scripts.  It has a few different ones in them, but
the main one at this time is run-envdir, which loads daemon specific
settings from the definition's env directory and uses them to launch the
daemon.  Others include an optional feature for user-defined services, and
basic support for one of three getty.  I may or may not make a new one for
the optional dependency feature; I'm going to see if it can be standardized
within run-envdir first.

I can always remove the dots, but then you would have these mixed in with
all of the definitions, and I think it will add to the confusion more than
having them hidden.  As it stands, the only time you need to mess with the
dot directories is (a) when setting them up for the first time, or (b) when
you are switching your logging around.  Otherwise there's really no need to
be in them, and when you use "ls /etc/sv" to see what is available, they
stay out of your way.

If there is a better arrangement that keeps everything in one base
directory for easy management but eliminates the dots, I'll listen.
Although I think this arrangement actually makes a bit more sense, and the
install instructions are careful to include the dots, so you only need to
mess around with them at install time.

On Thu, Jan 8, 2015 at 8:20 AM, Luke Diamand  wrote:

> Is it possible to avoid using hidden files (.env) as it makes it quite a
> lot harder for people who don't know what's going on to, um, work out
> what's going on.
>
> Thanks!
> Luke
>
>

Re: thoughts on rudimentary dependency handling

[Avery Payne]

On Wed, Jan 7, 2015 at 6:53 PM, Laurent Bercot 
wrote:
>
>  Unfortunately, the envdir tool, which I use to abstract away the daemons
>> and settings, only chain-loads; it would be nice if it had a persistence
>> mechanism, so that I could "load once" for the scope of the shell script.
>>
>
>  Here's an ugly hack that allows you do that using envdir:
> set -a
> eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort |
> uniq -u)
> set +a
>

Thanks!  When I can carve out a bit of time this week I'll put it in and
finish up the few bits needed.  Most of the dependency loop is already
written, I just didn't have a somewhat clean way of pulling in the
$CMDWHATEVER settings without repeatedly reading ./env over and over.


>  It only works for variables you add, though, not for variables you remove.


It will work fine. I'm attempting to pre-load values that will remain
constant inside the scope of the script, so there isn't a need to change
them at runtime.

RE: thoughts on rudimentary dependency handling

[Avery Payne]

On Wed, Jan 7, 2015 at 7:23 AM, Steve Litt 
 wrote:
>
> I'm pretty sure this conforms to James' preference (and mine probably)
> that it be done in the config and not in the init program.
>
> To satisfy Laurent's preference, everything but the exec cron -f could
> be commented out, and if the user wants to use this, he/she can
> uncomment all the rest. Or your run script writing program could have an
> option to write the dependencies, or not.
>

I've pretty much settled on a system-wide switch in sv/.env (which in the
scripts will show up as ../.env).  The switch will, by default, follow
Laruent's behavior of "naive launching", ie. no dependencies are up,
missing dependencies cause failures, and the admin must check logging for
notifications. Enabling the feature would be as simple as

echo "1" > /etc/sv/.env/NEEDS_ENABLED

...and every new service launch would receive it.  You could also
force-reload with a restart command.  Without the flag, the entire chunk of
dependency code is bypassed and the launch continues "as normal".

The goal is the same but the emphasis has changed.  This will be considered
a fall-back feature for those systems that do not have such a tool
available, or have constraints that force the continued use of a shell
launcher.  It is the option of last resort, and while I think I can make it
work fairly consistently, it will come with some warnings in the wiki.  For
Laurent, he wouldn't even need to lift a finger - it fully complies with
his desires out of the box. ;-)

As new tools emerge in the future, I will be able to write a shunt into the
script that detects the tool and uses it instead of the built-in scripted
support.  This will allow Laurent's work to be integrated without messing
anything up, so the behavior will be the same, but implemented differently.

Finally, with regard to the up vs actually running issue, I'm not even
going to try and address it due to the race conditions involved.  The best
I will manage is to first issue the up, then do a service check to confirm
that it didn't die upon launch, which for a majority (but not all) cases
should suffice.  Yes, there are still race conditions, but that is fine -
I'm falling back to the original model of "service fails continually until
it succeeds", which means a silently-failed "child" dependency that was
missed by the "check" command will still cause the "parent" script to fail,
because the daemon itself will fail.  It is a crude form of graceful
failure.  So the supervisor starts the parent again...and again...until the
truant dependency is up and running, at which point it will bring the
parent up.  Like I said, this will be a fall-back feature, and it will have
minor annoyances or issues.

Right now the biggest problem is handling all of the service tool calls.
They all have the same grammar, (tool) (command) (service name), so I can
script that easily.  Getting the tools to show up as the correct command
and command option is something else, and I'm working on a way to wedge it
into the use-* scripts so that the tools are set up out of the box all at
the same time.  This will create $SVCTOOL, and a set of $CMDDOWN, $CMDUP,
$CMDCHECK, etc. that will be used in the scripts.  **Once that is done I
can fully test the rest of the dependency concept and get it fleshed out.**
 If anyone wants to see it, email me directly and I'll pass it along, but
there's not much to look at.

Unfortunately, the envdir tool, which I use to abstract away the daemons
and settings, only chain-loads; it would be nice if it had a persistence
mechanism, so that I could "load once" for the scope of the shell script.
Because of that, there will be some odd scripting in there that pulls the
values, i.e.

[ -f ../.env/CMDUP ] || echo "$(basename $0): fatal error: unable to load
CMDUP" && exit 99
CMDUP=$(cat ../.env/CMDUP)

with an entry for each command.


> In my 5 minute thought process, the last remaining challenge, and it's
> a big one, is to get the right service names for the dependencies, and
> that requires a standardized list, because, as far as I know, the
> daemontools-inspired inits don't have "provides". Such a list would be
> hard enough to develop and have accepted, but I expect our
> friends at Red Hat to start changing the names in order to mess us
> up.
>

Using a "./provides" as a rendezvous or advertisement mechanism I think is
nice-in-concept but difficult-in-practice. Give it a bit more thought and
you'll see that we're not just talking about the *service* but also any
*protocol* to speak with it and one or more *data transport* needed to talk
to the service.  Example: MySQL using a port number bound to 127.0.0.1, vs
MySQL using a file socket.  Both provide a MySQL database and MySQL's
binary client protocol, but the transport is entirely different.  Another
example: exim4 vs postfix vs qmail vs (insert favorite SMTP server here).
All speak SMTP - but some do LMTP at the same time (in either sockets or
ports), so there are a