Re: gmane archive

[Laurent Bercot]

 Posts to the list still aren't showing up on Gmane, although everything
seems to be working on the skarnet.org side.
 I have contacted the gmane.org administrator, but no reply so far.
 So I have set up the local archive again, and subscribed the list to
mail-archive.com.

 http://skarnet.org/lists.html updated.

--
 Laurent

Re: variables in the run script

[Laurent Bercot]

On 09/10/2013 08:58, Peter Pentchev wrote:

Can you post your *entire* exact script, including the shebang line?

If you do not have a shebang line, then try adding '#!/bin/bash' as
the first line of your script - and yes, since you're using
Bash-specific features, you do need to specify #!/bin/bash to make sure
that nothing tries to run your script using, say, /bin/sh, which might
be Bash or it might not be Bash, or it might be Bash behaving strangely
because it has been invoked as /bin/sh, or who knows what else :)


 My instinct tells me that Peter is spot on.
 Scripts behave differently when run from a user shell and from a run
script because those are two widely different environments - which is part
of the point of using a supervision tree to manage your processes - and
the devil usually is in the seemingly irrelevant details.

 As a stylistic note: you do not need to loop in a run script. Just execute
the script once, finishing with "exec sleep $zzz". Your supervisor will
run it again when sleep exits. Think of a run script as something you would
run from a crontab, except it performs the sleeping itself.

--
 Laurent

Re: CONFIRM subscribe to supervision@list.skarnet.org

[Laurent Bercot]

On 17/10/2013 08:40, supervision-h...@list.skarnet.org wrote:

Hi! This is the ezmlm program. I'm managing the
supervision@list.skarnet.org mailing list.

I'm working for my owner, who can be reached
at supervision-ow...@list.skarnet.org.

I respectfully request your permission to add

tho...@schwinge.name

to the subscribers of the supervision mailing list. This request
either came from you, or it has already been verified by
the potential subscriber.

To confirm, please send an empty reply to this address:

supervision-tc.1381995659.mlfbgnnbgdnanihmpame-thomas=schwinge.n...@list.skarnet.org

Usually, this happens when you just hit the "reply" button.
If this does not work, simply copy the address and paste it into
the "To:" field of a new message.

or click here:

mailto:supervision-tc.1381995659.mlfbgnnbgdnanihmpame-thomas=schwinge.n...@list.skarnet.org

If you don't approve, simply ignore this message.

Thank you for your help!


--- Administrative commands for the supervision list ---

I can handle administrative requests automatically. Please
do not send them to the list address! Instead, send
your message to the correct command address:

For help and a description of available commands, send a message to:


To subscribe to the list, send a message to:


To remove your address from the list, just send a message to
the address in the ``List-Unsubscribe'' header of any list
message. If you haven't changed addresses since subscribing,
you can also send a message to:


or for the digest to:


If you need to get in touch with the human owner of this list,
please send a message to:

 

Please include a FORWARDED list message with ALL HEADERS intact
to make it easier to help you.

--- Enclosed is a copy of the request I received.

Return-Path: 
Received: (qmail 26673 invoked from network); 17 Oct 2013 07:40:59 -
Received: from unknown (HELO smtprelay01.ispgateway.de) (80.67.31.39)
   by unknown with SMTP; 17 Oct 2013 07:40:59 -
Received: from [217.84.1.179] (helo=stokes.schwinge.homeip.net)
by smtprelay01.ispgateway.de with esmtpa (Exim 4.68)
(envelope-from )
id 1VWiCz-00041c-9M
for 
supervision-sc.1381993049.kocjpjmjifghfndcbfni-thomas=schwinge.n...@list.skarnet.org;
 Thu, 17 Oct 2013 09:41:25 +0200
Received: (qmail 22651 invoked from network); 17 Oct 2013 07:41:07 -
Received: from kepler.schwinge.homeip.net (192.168.111.7)
   by stokes.schwinge.homeip.net with QMQP; 17 Oct 2013 07:41:07 -
Received: (nullmailer pid 8543 invoked by uid 1000);
Thu, 17 Oct 2013 07:41:06 -
From: Thomas Schwinge 
To: 
supervision-sc.1381993049.kocjpjmjifghfndcbfni-thomas=schwinge.n...@list.skarnet.org
Subject: Re: confirm subscribe to supervision@list.skarnet.org
In-Reply-To: <1381993049.12672.ez...@list.skarnet.org>
References: <1381993049.12672.ez...@list.skarnet.org>
User-Agent: Notmuch/0.9-101-g81dad07 (http://notmuchmail.org) Emacs/23.4.1 
(i486-pc-linux-gnu)
Date: Thu, 17 Oct 2013 09:41:01 +0200
Message-ID: <87siw0l4z6@kepler.schwinge.homeip.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Df-Sender: dGhvbWFzQHNjaHdpbmdlLm5hbWU=




--
 Laurent

whoops

[Laurent Bercot]

Sorry, misclick. Please ignore the previous mail.
That's what happens when you have to use a GUI after 20 years of text 
interfaces.

--
 Laurent

Re: s6-svwait: fatal: unable to ftrigr_startf: No such file or directory

[Laurent Bercot]

 Hi Vallo,

 Thanks for the bug-report. It is indeed an oversight on my part, and an
interesting question: where should internal executables go when you want
to install them somewhere but do not want to make them globally accessible ?
slashpackage makes that easy: just don't export the command. FHS is different;
IIRC, /usr/libexec/$package_name is the traditional location for those binaries.

 But on the other hand, I also want people to be able to run s6-ftrigrd as a
local service, so the best solution is probably the simplest, i.e. export the
binary. My advice would be, just patch package/command.exported before 
compiling,
so that it includes a s6-ftrigrd line. I will update package/command.exported in
the next release.

 Note that the exact same problem will happen with skadnsd in the s6-dns 
package,
if you're also packaging it, and since it makes sense to run skadnsd as a local
service (even more so than s6-ftrigrd), the solution is to export that binary 
too.

 I will definitely add /usr/libexec support at some point, but for now, 
exporting
the binaries is a quick fix that is not even too ugly.

--
 Laurent

Re: s6-svstat

[Laurent Bercot]

 Hi Jorge,
 Sorry for the delay in responding.



The pid remains the same, consistently. So, what can be the meaning of
"0 seconds"? (The service is not working, no firewall logs received.)


 Has the system clock been modified after the service has been launched ?
"0 seconds" is what s6-svstat displays when it sees that the service has
been launched in the future.

 If it's not the case, please send an strace of what's happening.
(If your run file is written in execline, adding "fdmove -c 2 1 strace -vf"
at the top of your script, and restarting the service, will print the strace
in your logs.)

--
 Laurent

Re: s6-log n0 still creating archives

[Laurent Bercot]

I noticed that running s6-log as follows still creates few
archived logs:


 Thanks for the report. Does s6-log send any warnings to its
own stderr (which may or may not be caught by your supervision
infrastructure) ? Can you please send a strace of a s6-log
run where you observe the behaviour ?

--
 Laurent

Re: s6-log n0 still creating archives

[Laurent Bercot]

 OK, easy one: n was stupidly enforced as 2 or more.
 Thanks for the report and the trace.

 It's not possible to ignore the archive processor even with n0,
because processors may have side effects on the state files, and
may also do things like send the "current" file to other log
processing software, without local archiving - typically for
network logging. s6-log will process and create the archive file
in any case, then examine the directory and delete the archives
by creation time until there's only n left.

 Until the next s6 release, this patch should fix the behaviour.

--
 Laurent

--- src.old/daemontools-extras/s6-log.c
+++ src/daemontools-extras/s6-log.c
@@ -752,7 +752,6 @@
   case 'n' :
   {
 if (!uint320_scan(*argv + 1, &cur_n)) goto fail ;
-if (cur_n < 2) cur_n = 2 ;
 break ;
   }
   case 's' :

[announce] s6-1.1.2

[Laurent Bercot]

(I forgot to announce it here.)

 Hello,

 s6-1.1.2 is out.
 It fixes a stupid bug in s6-log, which enforced a minimum of 2
archived log files. (Thanks to Vallo Kallaste for the bug-report.)

 http://skarnet.org/software/s6/

 Enjoy,
 Bug-reports still and always welcome.

--
 Laurent

[announce] s6-1.1.3

[Laurent Bercot]

 Hello,
 s6-1.1.3 is out.

 It fixes a well-hidden bug in s6-log that made it busyloop in certain
circumstances when a log rotation fails. Users are encouraged to upgrade -
when something goes wrong, s6 is supposed to help you, not to help bring
your machine down!
 Thanks to Vallo Kallaste for the report.

 http://skarnet.org/software/s6/

 Enjoy,
 Keep those bug-reports coming.

--
 Laurent

Re: tac_plus with multi thread

[Laurent Bercot]

On 08/04/2014 22:59, Asif Iqbal wrote:

Hi All,

Can runit supervise multithreaded appliacation like tac_plus?

tac_plus: https://github.com/mkouhei/tacacs-plus

man tac_plus

-G Remain in the foreground, but not single-threaded nor logging to the
tty.


 Hi Asif,
 This is all a supervision framework needs. runsv will care of tac_plus -G
just fine.



-g Single threaded mode.  The daemon will only accept and service a
single connection
 at a time without forking and without closing file descriptors.
  All log messages appear
 on standard output.

 This is intended only for debugging and not for normal service.


 Ugh. Someone should definitely point the authors to:
 
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/unix-daemon-design-mistakes-to-avoid.html

--
 Laurent

Re: slapd crashing very frequently under runsv on FreeBSD

[Laurent Bercot]

OpenLDAP (slapd) is crashing very frequently under runsv (hundreds
of times per day, usually when it gets a burst of queries).


 It looks like there's a bug in slapd that manifests itself when it's run
under runsv and not when it's run under System V init. Signal 11 is SIGSEGV,
it means either a hardware error or a bug in the application; since your
slapd works fine under load when unsupervised, the hardware error is
unlikely.

 To track the bug, you'll need to check the exact differences between
running slapd under runsv and running slapd manually. The main differences
usually are:
 * working directory. That's a silly one, but I've seen problems because of it.
 * environment variables. Those are usually a smoking gun, especially PATH;
System V scripts usually pollute the environment with lots of variables,
some of which are actually needed for the daemon to run correctly.
 * open file descriptors. Typically, check whether stdin and stdout are
closed, or pointing to /dev/null, or pointing to something else (which is
probably a misconfiguration).
 * some options. Typically, a "log to stderr instead of syslog" option
and a "don't background yourself" option, which you don't set with sysvinit
but do set with a supervision system. Some daemons do horrible, ugly things
when you set those.
 * terminals and sessions, which are usually set wrong when you run a daemon
from your command line - but running it under runsv or an init.d script should
yield the same result (no controlling terminal) so it's not a likely culprit.

 If all else fails, a good way to gather information is to strace a run with
supervision, a run without, and compare the straces side by side.



My first guess is that slapd dies when it gets an errror writing to a
full logging pipe.  Is there a logger set up for this service?  Is it
writing log data?  Is it staying up?


 Writing to a full log pipe probably wouldn't cause a SIGSEGV. Most
programs would simply block there.


--
 Laurent

Re: B/LFS-s6 Project

[Laurent Bercot]

On 24/06/2014 09:03, Vincent de RIBOU wrote:

Can you summarize your feeling against systemd.


 It's not a question of feelings, it's a question of technical
merit. You can find various arguments for, or against, systemd
out there, and I very much do not wish to import *that* discussion
into this mailing-list; you can find plenty of it in other places.

 The technical arguments against systemd are very well summed up
by Rich Felker here: http://ewontfix.com/14/
 Less technical, more political, but still valid, arguments, are
summed up here: http://boycottsystemd.org/
 For equity: http://0pointer.de/blog/projects/why.html

 Now please, let me repeat that I do not want discussion about
systemd to happen here. It's a subject that people easily get
passionate and emotionally involved about, which is a good thing
- it shows that there's a real, important issue, that needs to
be solved - but it's really hard to keep a level-headed, civil
and constructive discussion. Most systemd threads out there turn
into flamefests after a few posts (if that), involve very little
technical knowledge,  and are basically a waste of time.

 We want to provide a real, usable alternative. Our goal is not
to stand against systemd, but to have a clean, well-designed,
lighter and less monolithic init system so users have a choice.
systemd is not our focus; discussing systemd takes us away from
our focus. If you (not specifically Vincent, but anyone on the
mailing-list) feel the need to argue that systemd solves world
hunger and we don't need anything else, please take it off the
mailing-list. If you feel the need to argue that systemd is a
horrible pile of crap, my heart is with you, but please also take
it off the mailing-list.

--
 Laurent

Re: B/LFS-s6 Project

[Laurent Bercot]

 I'm currently working with Jim to help him set up a way of
booting a LFS system with s6 while using standard utilities
and sticking to the FHS.

 Once the one-time initialization (which includes stage 1 and
a one-time initialization script that runs in stage 2, when
s6-svscan is already running) is complete, I expect a s6-based
system to be very close to a runit-based system. Service
directories are similar, the installation and administration
principles are similar. The main difference between s6 and
runit, for a distribution builder, is boot time.

 The shutdown script, .../.s6-svscan/finish, can be directly
translated from /etc/runit/3; and /etc/runit/1 can be the
basis for a s6-based /sbin/init, but since there is no fallback
with s6, /sbin/init should be minimal and the rest of the
one-time initialization happening in /etc/runit/1 should run
once s6-svscan has taken over.

--
 Laurent

Re: B/LFS-s6 Project

[Laurent Bercot]

The same will go for s6. Right now our first task is building a clean
room LFS to deploy s6 into. The only imported tools for s6 we will be
utilizing are the execline and skalibs packages. The rest will be
native GNU/Linux.


 In that case, GNU coreutils will have to be part of the base system,
because the /sbin/init script will call utilities such as cp or env.
 You will also have to decide what language to write /sbin/init in.
If you use /bin/sh, then your base shell (bash?) will also be part of
the base system.
 I wrote my stage 1 init script using only execline and s6-linux-utils,
which I wrote to avoid a dependency to coreutils or busybox until I
have s6-svscan running, to have the strict bare minimum on a tiny root
filesystem. But it was mostly a theoretical exercise since people will
usually need coreutils anyway, and I've since found that the size of
your root filesystem doesn't matter as long as it's read-only.



One hope we are wanting to test is if Runit service scripts can
actually be reused in s6. This would promote a high level of
compatibility with our current work, leaving only the native s6
logging tools to supplement the Runit logging tool, not to mention
save a lot of research time.


 Most runit service scripts should work as is with s6.

 What runit provides that s6 does not support:
  - The sv command supports the /etc/init.d LSB script interface.
The s6-svc command does not. However, it is trivial to write a script
that performs that functionality if desired.
  - the human-readable supervise/stat and supervise/pid files
  - the control/ customization subdirectory
  - the check script

 What s6 provides that runit does not support:
  - the event/ subdirectory and no-polling checks of the service
state via the s6-svwait command
  - the nosetsid file

 Service directories that do not make use of those functionalities
will be portable from one system to another.



However, as far as the one-shots of stage-1, while we recycled
sysvinit scripts for Runit, we would like to have s6's stage-1s
script contain all the one-shot init information if possible using
all the core functions of the base sysvinit scripts of LFS, while
invoking one-shots using checked triggers as previously described,
the same with stage-3s shutdown sequence also. We want s6 to be more
native to itself script-wise.


 One decision you'll have to make early: do you wish to implement
service dependencies ? If you do, it's a matter of scripting around
s6-svwait, but you need to take it into account in the init scripts,
and possibly in the run files as well - which would break runit
compatibility.
 Note that even with s6-svwait, there are race conditions when you
are waiting for a service to come up. s6-svwait can tell when
s6-supervise has successfully spawned its run script, but it cannot
tell when the run script has actually become an operational service.
To avoid race conditions, the service itself has to notify the
outside world when it is ready.



I should have the base system up and running within a week in my
spare time, so I'll start from the example implementation provided
and work my way out from there.


 Good luck, Jim. ;)
 Feel free to ask for details if there's something you don't
understand in the example implementation and it's not explained (or
not clearly enough) in doc/s6-svscan-1.html - there are definitely
a few tricky points, and at this fundamental level it's important to
get them right.

--
 Laurent

Re: B/LFS-s6 Project

[Laurent Bercot]

1. Write a stage-1 init script in standard Linux Bash shell
scripting. Because Bash is the stereotypical universal Linux shell
on most distributions, we felt that it's usage in the scripting
process would aid and ease s6 into the system faster and cleaner.
However, we are not above utilizing scripting using execline if
necessary as a learning tool, but our goal is to use standard Bash.


 My point is that there are a few shells out there, and people may
want to use something else than bash. (ash, dash, zsh, you name it,
someone swears by it.) So please use /bin/sh and avoid bashisms to
remain portable (if that means anything for a stage 1 init).

 The only place where you will need a bit more oomph than /bin/sh
provides is to open a fifo for writing when there's no reader yet.
(To redirect /sbin/init's stdout and stderr to the fifo that will be
read by the catch-all logger, but the logger isn't up yet since it
will be started by s6-svscan.)
 That shouldn't be a problem though: you have execline installed
and you can use redirfd in a shell script. You will just have to
write something like "exec redirfd -wnb 1 fifo sh -c 'rest of script'".
Chain loading works with the shell too, it's just cumbersome (and
a bit slower) when you need to execute into another shell.



2. As far as service scripts are concerned, we may use either
execline or bash scripting depending on which moved things along
faster.


 /bin/sh is perfectly fine as long as you remember to exec the
long-running process in the end to avoid keeping a dangling shell
and sending signals to the wrong process.
 The main reason I use execline for run scripts is that execline is
best suited for pure chain loading, and run scripts mostly involve
process state changes (fd redirections, environment setting, resource
limits setting, etc.) followed by an exec, and those are best
performed by chain loading: both runit and s6 provide process state
change programs for such use.
 Chain loading also makes it easy, for instance, to insert a
"strace -vf" at any point in your run script for debugging purposes.

--
Laurent

Re: Is runit not being maintained?

[Laurent Bercot]

On 22/07/2014 22:57, Joe M wrote:

I read that dragora moved from runit to perp as runit was not being
maintained.

Just wanted to check if runit is not being actively maintained and if
perp is a better option going forward.


 runit, as perp, don't need to be "actively maintained" because they
just work. I don't know how much maintenance effort Gerrit or Wayne are
willing to put into it, but very little is actually necessary.

 If you're looking for a similar alternative that is being actively
developed and enhanced, I can suggest s6 - I intend to keep working on
it for the foreseeable future, and it's currently undergoing a major
packaging overhaul so it will be easier to install and access.

--
 Laurent

Re: svlog processing

[Laurent Bercot]

On 22/07/2014 23:17, Joe M wrote:

I find svlogd log files hard to read as the lines have a UTC timestamp(-tt)
and the filenames with @s are hard to understand. I read that
svlog files are meant to be processed by a post-processor.

I use logrotate for other log files.

Just wanted to check if anyone could please share some insight on how
they manage/post-process svlogd generated logs.


 This is all policy, so my answer is by no means authoritative, YMMV, etc.

 The logging directory, since it's automatically rotated, is a good place to
store logs if you are limited by storage space, but a bad place to store them
if you're not and your policy is to keep all logs for a certain amount of time.
In that case, it's a good idea to have a processor that moves all your archived
log files to your bigger storage space every N rotations (you can count N with
the state file). That processor can rename the archive files to any format you
like.
 The TAI64N format for the archive file name is a way of easily keeping track of
all the archive files in the logging directory without parsing human-readable
dates: it's practical for machine processing. You can take advantage of this by
handling them via automation all the way to the point where they need to be read
by a human.



In the runit documentation, I saw multilog and svlogd both being
used. When do you use multilog over svlogd?


 I think the part of the documentation where multilog is mentioned, i.e. the
runit index page, predates the addition of svlogd to the runit package. I'm not
aware of a situation where multilog is better than svlogd.
 s6-log offers full regex pattern matching, but does not provide network 
logging.

--
 Laurent

Re: initialization vs supervision

[Laurent Bercot]

On 23/07/2014 20:16, Wayne Marshall wrote:

In the best of un!x traditions, a stronger
system may in fact be one that recognizes the fundamental
differences between the two functions, and provides purpose-specific
solutions for each of them.


 I absolutely agree with this, as with all the rest of your message.

 However, an initialization system can strongly benefit from being
tailored to use a supervision system, so close ties between the two are
not surprising.
 Doing all the one-time initialization without having supervision
support is complex: one-time initialization often needs early services,
that the rest of the initialization depends on - I think of a default
logger, and of udevd, for instance.
 The pstrees that are posted in this thread show a nice amount of
supervised services, and also some services that are *not* supervised;
the reason for this is probably that the unsupervised services are
started in /etc/runit/1, when runsvdir isn't yet started. This is a shame:
you want core services to be supervised, even more than non-core ones.

 I believe the right way for an initialization system to boot is:
- do the minimum necessary amount of early init so the supervisor can work
- fork the one-time initialization process, and block it so it can only
run once the supervision tree is active
- exec into the supervision system (so that it is either process 1 itself,
or supervised by process 1).

 This is the approach I have with s6. However, since I wanted to keep s6
system-agnostic and, as you said, initialization is system-dependent, I did
not provide an out-of-the-box /sbin/init - and it put too much work in the
hands of packagers. I intend to work on a s6-init package, which will
unfortunately have to be Linux-specific, to cover this flaw. The forked
one-time initialization process can safely be left to distribution packagers,
just like /etc/runit/1 is; but the tricky /sbin/init has to be provided in
order for an integrated init+supervision system to be usable.

--
 Laurent

Re: initialization vs supervision

[Laurent Bercot]

On 23/07/2014 23:45, James Powell wrote:

Now granted some things are not able to be supervised such as udev on my end. 
But honestly, does udev really require supervision?


 Yes, it does - why wouldn't it ? Or, if it doesn't, why would any other 
service ?

 We don't supervise services for the heck of it. We supervise services because:
1. the world is imperfect and programs sometimes crash, udevd included. Even
simple, failsafe programs can be killed by an administrator mistake, or by a
misconfigured OOM killer.
2. it makes it easy to control them with a minimal code path.

 "one-shot daemons that do not need supervision, they need to be ran, and just
left alone" is exactly what sysvinit does. It works, until it doesn't. We aim
to do better. udevd is no different from socklog or s6-tcpserver or qmail-send,
except that it's bigger and messier so it's more likely to die: why would you
want to stick supervision on "true" services, and leave udevd out ? The early,
fundamental services, that really hose your whole system if they crash, are
the ones that need supervision the most. A long-lived process is a long-lived
process is a long-lived process, there's no reason to treat udevd, dhcpd,
getty or sshd in different ways.

 Yes, a clear separation between initialization and supervision makes it
difficult to have everything under your supervision tree. This is why I am
suggesting a slightly more integrated approach, that solves this problem.

 s6 makes it easy (and it will be done automatically when I release s6-init)
but it can be done with runit too. Have /etc/runit/1 simply fork the real init
script then exit. Have the real init script block on something that will only
unblock when runsvdir is running. Now your real init script only executes
when runsvdir is already running, and you can add whatever service you want
to it.
 The scheme could probably be made to work with perp too, I just haven't
thought about it.

--
 Laurent

Re: initialization vs supervision

[Laurent Bercot]

On 24/07/2014 02:49, Alex Efros wrote:

udevd is only service started from /etc/runit/1. And, honestly, I think
it's much simpler to just kill it at end of /etc/runit/1 and (re-)start it
as a normal service when /etc/runit/2 will be executed, than try to
fork/delay parts of /etc/runit/1 - because most of other normal services
expect all parts of /etc/runit/1 already done before they'll start.

And I'll implement this right after udevd will crash for the first time.
Not because I think it's special and don't need supervision, but just
because I'm usually have other, more important things to do, than forcing
supervising to all and every service.


 What I'm reading is that you're willing to do treat udevd differently and
let it run unsupervised, and duplicate work (run udevd in stage 1 and again
in stage 2) because it's much easier. And you're right. My point is that if
it's so much easier and tempting to treat an early service differently, if
you have to jump through hoops to get something supervised, then we got the
design wrong, and I'm not satisfied with it.



- mount.ntfs-3g
   * this one uses FUSE and it's started by mount command - is it makes
 sense or even possible to supervise it?


 Not without redesigning mount to cooperate with a supervision infrastructure.
That would be the smart thing, but would tie mount to a given choice of
process management: it's a policy desision, and you can't embed policy in
software, unless you're going the systemd way. So that's not going to happen.



[other long-lived processes]


 Like mount.ntfs-3g, those daemons are started by various subsystems during
your machine's lifetime, and the smart thing for those subsystems would be
to tie to a supervision infrastructure, but that's their choice to make, not
ours. There's nothing for us to do here.



So, how udevd is more important to supervise than all these apps/services?


 It's not more important; it's just not part of an integrated applicative
system that forks uncontrolled processes all over the place no matter the
underlying infrastructure. It's a simple system daemon, and it can be
supervised like any other simple system daemon; the only reason why you don't
is that it's too hard with the framework you have. Well, it's the framework's
fault, and we can provide a better one.

--
 Laurent

Re: initialization vs supervision

[Laurent Bercot]

On 26/07/2014 20:47, Joan Picanyol i Puig wrote:

What "tricky" responsabilities are you thinking of for /sbin/init that
would make it Linux specific?


 s6-svscan wants a read-write directory to run in, and another to
run its logger in. I definitely want to support read-only root
filesystems, and it's too early to mount disks, so a tmpfs has to
be created - this is system-dependent. If you have a way to do
that in a system-agnostic way, I'm *very* interested. :)

 Also system-dependent is the list of early services you want
operational as soon as s6-svscan starts (and thus need to be copied
or linked to the scan directory before the exec): not every system
wants udevd, not every system has the same getty invocation, etc.
The linking itself can probably be done in a system-agnostic way,
though, and the early service definition left to the packagers, so
it's not as important.

--
 Laurent

Re: initialization vs supervision

[Laurent Bercot]

On 27/07/2014 15:30, Joan Picanyol i Puig wrote:

No magic wand here, I just see it as scripts all the way down...


 Of course it's scripts. But those scripts are not portable. You can't
make a tmpfs on FreeBSD the way you make one on Linux.



Creating and mounting a tmpfs and bringing up the network look pretty
much the same to me: initialization tasks orchestrated by scripts
invoking userland binaries. At this point in booting, if something fails
it's watchdog time...


 Yes, it's easy, but it still cannot be shipped in the s6 package if I
want it to be system-agnostic.



Regarding read-only root fs, isn't it just a matter of restarting
svscan's logger pointing to stable storage once it is up & running?


 The logger is not a problem, but you still need a read-write place you
can access before starting s6-svscan: the scan directory itself needs to
be read-write. There's no starting s6-svscan if you don't have a read-write
directory at all, for several reasons; and since you haven't started the
one-time init yet, there are no disks. So there needs to be a tmpfs
creation in /sbin/init, and I don't know how to do that portably.

--
 Laurent

Re: logpipe error handling patch

[Laurent Bercot]

On 29/07/2014 20:59, Jan Pobrislo wrote:

There are two possible resolutions - either make runsv create new pipe
when asked to start up again or make it die since it's unable to spawn
a child.


 Actually, there's only one possible resolution.
 When runsv is told to exit, it kills the service and closes the log pipe,
which is impossible to reopen. At this point, the service is dysfunctional
and there's no way to recover it: so runsv should not be accepting commands
anymore, and close its control pipe instead. Any sv command after "sv x"
should return an error, because the service is not in a state where it can
process commands; it will only be functional again when the logger has died,
runsv has died and runsvdir has restarted runsv.

 runsv could even exit right after sending the SIGTERM to the service, that's
the quick and easy solution that closes both the log pipe and the control
pipe. However, it also means runsvdir will restart it, and the new runsv
instance will try and spawn a new logger when the old one may still be hanging
around, and that could cause conflicts.
 A more advanced behaviour is to have potentially configurable timeouts in runsv
to ensure that the logger finally dies and the service can be restarted:
 - close the log pipe
 - give the logger some grace time
 - send it a SIGTERM (and a SIGCONT, just to be sure. Stopped processes don't 
die.)
 - give it some more grace time
 -  send it a SIGKILL
 - give it a bit more time
 - panic loudly.

--
 Laurent

Re: Has anybody build (or thought about) a supervisor with two log streams?

[Laurent Bercot]

I'd like to collect stdout and stderr separately. I could use this, for
instance, with web servers, to collect access logs from stdout and error
logs from stderr. Why has nobody done this? Or did I just not notice?


 Well there's only one real log stream, and that is stderr. stdout was
never made to produce logs - it's a part of the application flow.
"Access logs from stdout" sounds weird, why would anyone send access
logs to stdout ?
 Of course, there will always be software that just can't use stderr
the way it was intended to, and insists on logging stuff to stdout;
to me it's no different from software that insists on using syslog.

 Nothing prevents you from logging several streams, however; there
simply isn't any specific support in supervision suites. Just create
a fifo, redirect your service's extra stream to the fifo, and have
another service made exclusively of a logger reading on the fifo.
It won't be quite as reliable (as in, no data loss ever) as the
integrated pipe, but it will be enough for most use cases, especially
since you can still send the most critical logs to the integrated
pipe.

--
 Laurent

Re: The udev question & understanding what runsvdir "means"

[Laurent Bercot]

 Hi Avery,

 You may be interested in reading the thread at:
 https://www.mail-archive.com/supervision@list.skarnet.org/msg00081.html

 and especially my answer here:
 https://www.mail-archive.com/supervision@list.skarnet.org/msg00097.html

 The thing is, you only need one supervision tree, and it is possible
to run udevd, and other "internal" services, under it, but the runit
framework does not make it easy, since runsvdir is only started in
stage 2. My approach with s6 is to start the supervision tree as early
as possible, and run one-time initialization scripts once this tree is
available; the frontier between stage 1 and stage 2 is not as clear as
it is with runit.

--
 Laurent

Re: init.d script names

[Laurent Bercot]

On 02/10/2014 22:54, Avery Payne wrote:

Is it me, or are the Debian init.d service scripts unfortunately named?


 Oh, if only it was just a naming question. But it's much more complex
than that.



If the script names actually aligned with the services they controlled, we
could do all kinds of things to make transitioning off of SysV easier, such
as:


 Yeah, unfortunately that's not going to happen. You might get close for some
services, maybe even a majority, but in the general case there's no direct
mapping from a System V script to a supervision run script.

 The supervision frameworks we have are *process* supervision frameworks.
They can monitor long-running processes, and that's it. That's already great,
but nowadays machine state is more complex than that. A service can be
represented by something else than a process - for instance, is a network
card up or down ? - and real service management is a superset of process
supervision.

 System V init scripts do service state transition, not process state
transition: you have SysV scripts to start/stop network interfaces, and
they don't involve long-running processes. They don't *maintain* a state
per se - they can only do transitions between "runlevels", which really
are sets of service states, and they don't enforce that the running state
is the wanted state at all times; but still, they manage a broader definition
of "service" than just a long-lived process.

 And to add more complexity on top, in a SysV scheme, you can mix
one-time initialization and service startup. Which is not a bad thing
per se, it's sometimes needed, but this does not fit at all in a
runit-like scheme where you cleanly separate the one-time initialization
from the launching and supervising of the daemons.

 So yeah, there's no other way, for now, than doing manual conversion -
you need to really study the SysV scripts to know exactly what they do,
and split them into 1. one-time initialization, and 2. zero or more
processes to be supervised. A lot of the time, it's going to be
straightforward, but you just can't rely on it. You gave the example of
Samba, but Samba isn't even that hard - it's just 2 processes. I'm sure
some scripts out there are much more evil than that.

 In 2015, I plan to extend s6 to perform service management in a clean
way. I believe it's the only way we can provide a plausible alternative
to the _other_ "frameworks" out there, if you catch my drift.

--
 Laurent

Re: init.d script names

[Laurent Bercot]

I've already come up with an interesting plan to template out the entire
logging aspect.  Default system-wide logging methods with your choice of
back-ends and per-service manual overrides, here we come...


 That sounds perfectly reasonable. It's what I describe as the logging chain:
 http://skarnet.org/software/s6/s6-log.html#loggingchain



I've noticed this in a round-about way but haven't take the time to ponder
it.  After reading your description, the problem has been clarified and I
suspect my approach is probably not going to fit well.  But I might have a
temporary solution.


 Yes, you can emulate service management with processes. But it's kludgy
and at the end of the day you'll have to maintain processes that do
nothing, just as a boolean flag. Processes are cheap, but not cheap
enough that it's ok to maintain a live process to store one bit of
information.
 Believe me, we've been here before. You *can* use a process supervision
suite to watch everything that needs to be watched, but you're fitting
a square peg into a round hole and it's not going to be pretty.



There is a reasonable assurance that the interface will remain
up, granted, but there is no guarantee that it will *stay* up.  If some
other program brings it down, or an administrator accidentally brings it
down, or even if the cable is unplugged, then the "passive" nature of this
approach will not catch it, and the state of the interface is out of sync
with the expectations of the framework.


 Oh, yes, services need to be supervised, of course, and that is one of the
big flaws of System V. But not every service will fit nicely in your
"supervise this process" squares. Try it and see for yourself.
 As for your iptables rules example: this is a textbook case of
one-time initialization mixed with service startup. When you install
a new set of rules, it's a change in machine state that happens only
once, and there's no change in process state - so is a given iptables
configuration a service ? Does it need to be supervised ? It doesn't
fit that modelization. My conclusion is that the modelization is
insufficient and I'm setting time aside to work on something clean.



I think one-time setups can be accommodated by state tracking in the
service directory itself.  Something like ...

do-setup()
{
   setup stuff here
   if $error then
 return 1
   else
 touch ./setup
 return $?
   fi
}

[ ! -f ./setup ] && do-setup || exit 1


 And where do you put that, in your run file ? Does that mean that
when you try and up the service, if the one-time setup fails, the
supervisor will try to run it again and again ? That's very
dangerous. If a one-time initialization fails, you usually want
to make loud noises on the screen or in the logs, and wait for
user input. If you can't do that, and you retry stuff instead,
then at least you should make sure never to re-execute commands that
have already succeeded. That implies heavy state tracking in your
one-time initialization.
 It's a harder problem than it looks, really, and all the easy
way outs are hackish and will fail horribly when you least expect it.

--
 Laurent

Re: init.d script names

[Laurent Bercot]

On 03/10/2014 06:38, Avery Payne wrote:

So, should I stop work on runit-scripts and s6-scripts?  If this is all for
naught, then why put the effort in?


 I didn't say that, duh. I said it was a hard problem to solve in the general
case, that there was no simple shortcut, and that the framework needed to
support more flexible weaving of one-time initialization and service startup.
That doesn't mean converting as many scripts as possible from SysV to a
supervision scheme isn't worth doing - on the contrary, it's a very laudable
initiative, and thanks for the effort you're putting in! Just don't expect
perfect results for now - and please stay tuned, because I'll try to improve
the framework soon, and hopefully make your work easier.

--
 Laurent

Re: run templates for runit/s6

[Laurent Bercot]

 For simple services, it's looking good, except for this part:


The "probe to see if it really is installed" is an unfortunate necessity as
Debian allows the init.d script to be left in place even when the program
is removed, and one of my goals is to support the "drop in and replace SysV
init scripts" feature; so this is a rough emulation of that behavior.  Yes,
the init.d scripts really do silently abort if the program is not present.


 Well you really don't want to do this. In SysV the failure just happens once,
but here the supervisor will try and start the service again and again, every
second, using up CPU and filling up your logs with error messages.
 The right way to proceed is to remove the service directory (or symlink)
from the scan directory when you remove the package, the same way you add
such a link when you install the package. You can keep the scripts, and your
whole set of service directories, as a part of your "sysvrc drop-in
replacement" package, but you have to make sure you only launch a supervisor
on the services that are actually installed. You cannot have a supervisor
running for every potential service under the sun - that would be way uglier
and wasteful than the original SysV, which would defeat the purpose :)

 Yes, that means you'll have to edit the post-install and pre-removal scripts
in every service package. I'm afraid there's no way around it: SysV does not
care whether your daemon is "wanted up" or not, so it can get away with
trying and silently failing, but supervision does, and cannot.



I haven't gotten the s6 image up and running yet, so I don't have anything
for s6-scripts.  Hang in there - I hope to have a image up and get the
source pulled in and built sometime this weekend.  I looked briefly at how
the service directories are set up, and I'm hopeful that I might be able to
cross-pollinate both with similar (or the same) scripts when possible.


 Run scripts should work the same for runit and s6. The differences are in
the way the supervisors themselves work: you'll need porting work, for instance,
if you want control file support (runit-specific) or startup/shutdown
notification support (s6-specific), but the run and finish files should be
reusable as is.

--
 Laurent

Re: run templates for runit/s6

[Laurent Bercot]

Ugh.  I'm not familiar with all of the details, but I would hazard a guess
that I would need to contact each and every package maintainer and submit a
patch.  At this point it becomes a political/social problem and not a
programming/design problem.  Not every maintainer is going to want to go
along.


 Yes, that's been the bane of supervision frameworks from the start - the
way they start and stop services is fundamentally different from the SysV
way, so it does require support from the packager. The adoption of
supervision has always been a political issue, never a technical one. And
the reason why systemd became so prevalent is that the systemd people
applied political pressure; they were much better at that than we were.

 You could look at how Alpine Linux does things. As far as I know, it's a
runit-based distribution.



Also, I'm still contemplating how to tackle error notification, so for
example, if .run/dependent-service fails because a service dependency, yes
it will be silly and loop, but at least it will yell to the admin "I'm
having trouble" while it's looping.


 That's how things have traditionally been running with supervision schemes.
Just start all the services, and they will fail and get restarted as long as
their dependencies aren't met, so eventually everything will work. It has
the advantage of simplicity: no explicit service dependency tracking is
needed. It has the drawback of being a bit messy, and using more resources
than needed at startup time (and flooding logs with transient error messages).



I like the idea, but I think it could be better.  Service dependency could
be done with state tracking, and that suggests that it should be in C and
not in shell script.


 Yes. Dependency management is a project on its own. I've had my sights on
it for quite a while, but I want to do things right, not as a quick and
dirty hack; that's part of my plan for 2015. And in any case, dependency
management, as a part of general machine state management, should not be
integrated into a process supervision suite, but be built *on top* of it;
that is why I have added notification hooks to s6 (and more hooks are
needed, so the next release will have more).



 The inclusion of state tracking in C is probably best discussed
in a separate thread.


 C or otherwise - there is no reason at all for the supervisors to
perform state tracking themselves. It is a different job, so a
different tool must be created. But please, yes, let's talk about
it.

--
 Laurent

Re: About the nosh package

[Laurent Bercot]

 Hi Casper,

 Very interesting. I knew about nosh, but last time I checked it, it
was not developed as much. The systemd unit file offline converter, in
particular, looks promising.
 I will definitely have to study how it's made. Jonathan is crazy, but
he usually has good design ideas :)

 Regarding "socket activation", I made a post about it on the gentoo forums:
 
http://forums.gentoo.org/viewtopic-p-7581522.html?sid=627c5a133142d01747000349627547c6#7581522
 In short: the idea is valuable, the way systemd does it is broken.

 Thanks for the update !

--
 Laurent

Re: another distro using runit...

[Laurent Bercot]

Le 20/10/2014 19:52, John Albietz a écrit :

- With nearly all of my services, I create enable scripts that check for,
and if necessary set up directories and perhaps even default passwords or
databases. And I haven't found an elegant way yet to integrate this into
runit. I think it would be useful to separate out a command for 'enable'
that would run successfully only once for a service.


 runit, like other supervision frameworks, doesn't have a notion of machine
state: its point is to supervise processes, not to perform machine state
management.
 This is exactly what supervision frameworks are currently lacking when
compared to popular init systems: a vision of the global machine state
involving more than processes. We've discussed about it before, check the
recent list archives.

 A summary of my position is that machine management can be done in a right
and elegant way, but should not be integrated *into* supervision frameworks
- it should be implemented *above* them.

--
 Laurent

Re: another distro using runit...

[Laurent Bercot]

Would you then be in support of moving the pre/enable logic into a separate
utility (something similar to `chpst)? So for example:

runwith --ExecStartPre "mkdir /var/log/mysql" --ExecReload
"/root/drain_reload.sh" mysqld


 It is unclear to me whether the "pre" scripts are supposed to be
state-checked and maintained along with the long-lived process or
if they are supposed to be run only once; and I suspect different
script authors to have different expectations.

 The thing is, everything that needs to be maintained should go into
the ./run script, because it is what the supervisor maintains. So if
you need to check that /var/log/mysql is present everytime, you should
add "mkdir -p /var/log/mysql" at the beginning of your ./run script.
 And everything that needs not be maintained should be handled outside
of the supervisor, in a one-time script.

 Supervisors only know one bit of information: is the long-lived process
alive or not ? They can't process states such as "the filesystem has been
correctly prepared but the daemon is currently down". So it's up to you
whether you want to maintain the whole state along with the daemon, or part
of the state, or nothing at all.

 The issue I have with utilities such as your suggested "runwith" is
where and when you would run it. If your suggestion is to have SysV-style
(or OpenRC-style, or anything else of the kind) init scripts making calls
to "runwith", which would exec the right one-time stuff and also prepare the
appropriate directories to be supervised, then it's a totally valid approach,
but it needs support from the init system to actually launch a supervision
tree *prior to* running all those scripts, and also support in the supervision
system itself to avoid race conditions. This is my current plan for evolving
s6, but it's not there yet, and neither is runit; it's very doable, but you
can't just hack it in a day - it requires design to get things right and
adaptable to distributions' chosen ways of starting stuff. If that was not
your suggestion, then I'm confused about how this "runwith" command would
be used.

 For now, I'd say you have to do things manually: analyze the "pre" stuff,
separate what needs to be run only once and what needs to be maintained,
move the run-once things into runit's stage 1, and move the maintained
things into ./run scripts. (Problems arise when you need to mix and match
run-once and maintained across different services; I've described this in
previous posts.)



Alternatively, using current runit functionality, what would you think
about  if I simply add additional service directories that get loaded in
order. So I could have all enable/pre scripts run as one-time runit
services in a separate svdir:


 You don't need a supervision tree for one-time stuff: those are all
short-lived processes. Just run the one-time scripts and be done with it.

--
 Laurent

Re: License selection for process scripts

[Laurent Bercot]

 Licensing is only an issue if you want to distribute code together with
pre-existing licensed code. Since it's unlikely that your scripts will be
distributed together with runit, s6, daemontools, perp, or anything else,
the fact that they all have different licenses should not be a problem for
you - you don't have to worry about compatibility, you can just choose
whatever license you like best for your script packages.

 As for what kind of license is "the best", it's a highly religious subject
and I very much wish to avoid this kind of debate on the mailing-list.
 I've found the wikipedia article at
http://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licenses
as well as http://opensource.org/licenses to be nice starting points.

--
 Laurent

Re: License selection for process scripts

[Laurent Bercot]

Le 24/10/2014 02:55, Avery Payne a écrit :

At this point, I will probably dual-license everything as BSD 3-clause for the
scripts themselves, and if needed a 2nd license that matches whatever
framework they are running under - just for the sake of expediency.


 That's my point: you don't have to match licenses. The supervisor and the
run scripts do not share any code, and most likely will not be
distributed together, so the framework's license is totally irrelevant in
the choice of the scripts' license. It would be akin to saying that all
shell scripts running on GNU should be licensed with the GPL, because they
will be interpreted by bash.
 But of course, it's your code, so you can license it however you want :)

--
 Laurent

Re: Process Dependency?

[Laurent Bercot]

 First, I need to apologize, because I spend too much time talking and
dismissing ideas, and not enough time coding and releasing stuff. The
thing is, coding requires sizeable amounts of uninterrupted time - which
I definitely do not have at the moment and won't until December or so -
while writing to the mailing-list is a much lighter commitment. Please
don't see me as the guy who criticizes initiatives and isn't helping or
doing anything productive. That's not who I am. (Usually.)

 On to your idea.
 What you're suggesting is implementing the dependency tree in the
filesystem and having the supervisor consult it.
 The thing is, it's indeed a good thing (for code simplicity etc.) to
implement the dependency tree in the filesystem, but that does not make
it a good idea to make the process supervision tree handle dependencies
itself!

 (Additionally, the implementation should be slightly different, because
your ./needs directory only addresses service startup, and you would also
need a reverse dependency graph for service shutdown. This is an
implementation detail - it needs to be solved, but it's not the main
problem I see with your proposal. The symlink idea itself is sound.)

 The design issues I see are:

 * First and foremost, as always, services can be more than processes.
Your design would only handle dependencies between long-lived processes;
those are easy to solve and are not the annoying part of service
management, iow I don't think "process dependency" is worth tackling
per se. Dependencies between long-lived processes and machine state that
is *not* represented by long-lived processes is the critical part of
dependency management, and what supervision frameworks are really lacking
today.

 * Let's focus on "process dependency" for a bit. Current process
supervision roughly handles 4 states: the wanted up/down state x the
actual up/down state. This is a simple model that works well for
maintaining daemons, but what happens when you establish dependencies
across daemons ? What does it mean for the supervisor that "A needs B" ?

   - Does that just mean that B should be started before A at boot
time, and that A should be stopped before B at shutdown time ? That's
sensible, but it simply translates to "have a constraint on the order
of the wanted up/down state changes at startup or shutdown". Which  can
be handled by the init and shutdown scripts, without the need for direct
support in the supervision framework; an offline init script generator
could analyze the dependency tree and output the proper script, which
would contain the appropriate calls to sv or s6-svc in the correct order.

   - Or does that mean that every time A is started (even if it is
already wanted up and has just unexpectedly died) the supervisor should
check the state of B and not restart A if B happens to be down ? What
would be the benefit of that behaviour over the current one which is
"try and restart A after one second no matter what" ? If B is supposed to
be up, then A should restart without making a fuss. If B is wanted up but
happens to be down, it will be back up at some point and A will then be
able to start again. If B is not wanted up, then why is A ? The dependency
management system has not properly set the wanted states and that is the
problem that needs to be fixed.
 
 * Supervisors currently have no way of notifying their parent, and they

don't need to. Their parent simply maintains them; supervisors are pretty
much independent. You can even run s6-supervise/runsv without s6-svscan/
runsvdir, even if that won't build a complete supervision tree. The point
is that a supervisor maintains one process according to its wanted state,
and that's it. (With an optional logger for runsv.) Adding a notification
mechanism from the supervisor to its parent (other than dying and sending
a SIGCHLD, obviously) would be a heavy change in the design and take away
from the modularity. It can be done, but not without overwhelming benefits
to it; and so far I've found that all the additional stuff that we might
need would be best handled *outside of* the supervisors themselves.

 The more I think about it, the more convinced I am that script generators
are the way to go for dependency management, and service management in
general. Script generators can take input in any format that we want, and
output correct startup/shutdown sequences, and correct run scripts, using
the infrastructure and tools we already have without adding complexity
to them. It's something I will definitely be looking into.

--
 Laurent

Re: Process Dependency?

[Laurent Bercot]

Le 31/10/2014 21:04, Avery Payne a écrit :


The reason I was attempting to tackle process dependencies is
actually driven by dbus & friends.  I've stumbled on many "modern"
processes that need to have A, and possibly B, running before they
launch.  Of course, you are right, in the perspective that dbus
itself is encouraging a design that is problematic, and that problem
is now extending into the issues I face when writing my (legacy?)
scripts.


 But it's okay to have dependencies !
 If there's a problem with D-Bus, it's not that it needs to be up
early because other stuff depends on it. It's that it's way too
complex for what it does at such a low level and it bundles together
functionality that has no business being provided by a bus. But you
cannot blame software for having dependencies - system software is
written to be depended on.



I haven't had time to think the entire state diagram through but I
did give it some brief thought.  Personally, I see current frameworks
as potentially having an incomplete process state diagram.  Most of
them have a tri-state arrangment, and I think this is where part of
the problem with dependencies shows up.  We currently have down -> up
-> finish -> down as a cycle.  In order for dependencies to work, we
would need a 4-state, i.e. down -> starting -> up -> finish -> down.
(...)


 You are thinking mechanism design here, but I believe it's way too
early for that: I still fail to see how the whole thing would be beneficial.
The changes you suggest are very intrusive, so they have to bring matching
benefits. What are those benefits ? What could you do with those changes
that you cannot do right now ?



The starting state is where magic happens.  During starting, other
dependencies are notified to start.  If they all succeed, we go to
up.


 So you are subjecting the starting of a process to an externally checked
global state. Why ?
 A process can start 1. when the global service state changes and this
process is now wanted up, or 2. when it has died unexpectedly and the
supervisor just maintains it alive.
 In case 1, there's no need to modify the supervisor at all : the existing
functionality is sufficient. Service management can be done on top of it,
possibly via generated scripts.
 In case 2, the global state has not changed, the process is simply wanted
up and should be restarted. If it is wanted up, then its dependencies are
*also* wanted up, and should be up. Why would you need to perform a check
at that point ? Just restart the process. If something goes wrong, it will
die and try again; it's only a transient state, so it's no big deal.
Heavyweight applications for which it *is* a big deal to unsuccessfully
start up several times can have a safeguard at the top of their run script
that checks dependencies for *this* service. The dependency checking can also
be auto-generated.



If we fail, we go to finish, where the dependencies are notified
that they aren't needed by our process; it's up to either B or B's
supervisor (not sure yet) to decide if B needs to go down.


 And thus you put service management logic into the supervisor.
I hate these #blurredlines. :P



 Looping A
seems undesirable until you realize that your issue isn't A, it's B.
And when A fails to start, there should be a notification to the
effect "can't start A because of B".


 But what are you trying to achieve with this that you cannot already
do ? Why can't you just let A be restarted until it works ?
If A is too heavy, then specific guards can be put in place, but that
should be the exception, not the rule.

 What I can envision is keeping the global "wanted" state somewhere
along with the global "actual" state, and writing primitives, to be
called from run scripts, that block until the needed "actual" sub-state
matches the needed "wanted" sub-state. That way, processes that choose
it can block, instead of loop, until their dependencies are met. But
that's a job for a global dependency manager, not a supervision suite,
and there is still no need to modify the supervisor.



So you can see, supervisors don't talk to each other, the overlord
process pretty much stays as it is


 Not at all - the changes you suggest would be quite heavy on the
overlord and the supervisor. Today, overlords can send signals to
supervisors and that's it; whereas what you want involves real two-way
communication. Sure, it's simple communication with a trivial protocol,
but it's still a significant architectural change and complexity
increase.



Yes, indirectly.  Just because A wants to start, but can't because B
is having its own issues.  I see it as separation of responsibilities
- B has to get itself off the ground to start running, but
B-the-process doesn't have to be aware of A's needs, that's a problem
for B's supervisor.


 Yes. So let B do its thing, and let A do its thing, and everything will
be all right eventually. If looping A is a problem, then add something in
A's run script that prevents it from do

ezmlm-cgi bug (was: runit-scripts transitions to supervision-scripts)

[Laurent Bercot]

Le 31/10/2014 23:10, Avery Payne a écrit :

Just curious, but does anyone else have issues seeing some messages on the
mailing list?  My last message won't load through the web interface on
skarnet.


 Sorry about that. It's a bug somewhere in ezmlm-cgi. :(
 Some messages cause it to segfault when generating HTML from them. I have
tried tracking the bug, but could not isolate it; I haven't reported it yet
to the ezmlm-idx authors because I wanted to narrow it down more before
reporting it, but ran out of time.

 I don't think there's a security risk, so I did not take down the local
mail archive, but it is inconvenient indeed as some messages will not
display. Please use mail-archive.com instead for now.

--
 Laurent

Re: Transition to supervision-scripts

[Laurent Bercot]

On 07/11/2014 21:00, Avery Payne wrote:

I'm still in the process of getting a build of s6 going, mostly because I'm
still getting my head wrapped around the /package directory concept, and
getting the builds to play nice with it.


 Don't get stuck with that. /package is entirely optional - using it should
be the admin's decision, not the developer's. You can already do without it
with the current s6 release; it will be easier with the next one (scheduled
at some point in December).
 Using /package should make your life easier, not harder. If it's an
impediment for you, then don't use it.

--
 Laurent

Re: Consensus on process "instances"?

[Laurent Bercot]

 Hi Avery,



The simple approach to obtaining this behavior would be to write a
definition for each instance.  This provides a very fine control over what
is and isn't running but the maintenance of it is increasingly prohibitive
with each instance added.  For larger installations, I don't see this as
practical to maintain.


 If you have a fixed number of instance, it's not significantly harder.
 The main problem if is you have a dynamic number of instances. Changing the
main service tree dynamically is probably abusing the framework, and some
additional tool is probably needed here.



A possible solution would be to create a separate svscan directory, use the
same script for each instance with separate per-instance settings, and
start all of the instances as a process sub-tree of the parent svscan.


 Yes.
 As soon as you think of using a supervision tree to do that kind of thing,
more glue work is needed. Creating a specialised subtree sounds like a good
idea, because it gives you separate places to 1. factor all the commonalities
and 2. instantiate.
 There are more advantages to this than you give it credit for. For instance,
you can run all your subtree under a dedicated user, with dedicated quota. You
can factor more than the run script - most of the configuration can be done
at the "master" level.
 However, yes, dynamically creating, configuring and removing instances is
still heavy maintenance; but is certainly possible to automate a large
part of the grunt work.



Is there a better way to accomplish this, or a common consensus about how
to handle this situation?  Or do people simply deal with the pain of
writing all of the instances as they are needed?


 So far, I'm not aware of any consensus. Dynamic instantiation isn't something
lots of thoughts have been given about yet; thanks for bringing it to our
attention.
 What kind of utilities would you need to make it easier to manage and maintain
dynamic instantiation ?

--
 Laurent

[announce] s6-2.0.0.0

[Laurent Bercot]

 Hello,
 Merry Christmas! s6-2.0.0.0 is out.

 * Build system changed to ./configure && make && sudo make install.
Distributors rejoice! Packaging s6 will now be easy and won't go against
your guidelines, no matter how strict they are.
 * You need brand new skalibs-2.0.0.0 and execline-2.0.0.0, both s6
build-time dependencies.
 * Reassessed portability. s6 should run as is on at least Linux, FreeBSD,
OpenBSD, NetBSD, Solaris and MacOS X. If you're trying to make it work on
another architecture and adjustments are needed, please report it.
 * No major functional changes. Cosmetic fixes left and right.
 * One added feature: readiness notification support in s6-svwait, and a new
program, s6-notifywhenup, to perform fifodir notification of service readiness
without having to link the daemon to a specific library. Some minimal daemon
support is still needed: daemons signal readiness by writing a line to a file
descriptor. tcpserver and s6-[tcp|ipc]server*, among others, do this.

 Readiness notification support is an important tool to have. It's a
requirement for dependency management.

 Lots of future work to come in 2015. I can't wait to start!

 Enjoy, and please send bug-reports.

--
 Laurent

Re: [announce] s6-2.0.0.0

[Laurent Bercot]

On 22/12/2014 20:45, John Albietz wrote:

What is the upgrade path from pre 2.0.0.0 s6 installs to 2.0.0.0. Are there
any major compatibility issues?
With the new major version in the release number, not sure if I should
expect an upgrade to just *work*.


 As far as I can tell, things should just work. The release number is
simply aligned with the skalibs release number; skalibs has changed a
lot, but the s6 code itself, not so much.

 What *is* important to check, though, is that you do not mix binaries
from different releases. If you're using --enable-slashpackage, you're
safe: intra-package dependencies are versioned. If you're not, make sure
to always have all binaries of a single release accessible via PATH search.
s6-svwait from 2.0.0.0 will not work if the s6-ftrigrd program it calls is
from 1.1.3.2.

--
 Laurent

Re: s6 and friends 2.0

[Laurent Bercot]

 Hi Toki,
 Grats on trying out s6.
 Since most of your troubles have to do with skalibs and not with s6 itself,
please use the skaw...@list.skarnet.org list. (Reply-To: set.  I'd set
Mail-Followup-To: if I knew how to tell Thunderbird to do so. >.>)

 My answers below.



First, a minor note: Why do you keep those two extra digits in the release
version? I guess MAJOR.MINOR.PATCH are enough most of the times and
then sometimes a tiny fix would require a SUB-PATCH digit, honestly, one
can just delay a release a few days, to make sure everything is alright, and
increment the patch digit.


 That was my opinion as well, until I found that it didn't work in practice.
 The major number hardly ever changes - it basically takes a whole rewrite or
redesign of the project to warrant a major number change. So this one is out
of the picture for day-to-day versioning.
 The minor number, then, should be used for ABI versions. When the minor
number changes, the ABI may change.
 The third number should then be used for API augmentations, minor releases.
 And so a fourth number is needed for bugfixes.
 Basically, since the major number - which I really want to call an
'incarnation number' - is so static, it pushes everything to the right. If I
had my way, I'd use the "incarnation.major.minor.patch" terminology, but I'm
afraid it would be confusing - people are too used to calling the first number
"major".



Second, why is there this long list of configure directory:


 Because some of them are needed by the functionality, and some of
them are FHS legacy. What part do you think is superfluous ?



which doe not even respect ABI string when I end up with this:


 ABI string ? What do you mean ?



  * Contents of dev-libs/skalibs-2.0.0.0:


 I'm sorry, but if you're using a packaged version of skalibs,
I can't be held responsible for the way it is packaged. Figuring
out the way the configuration system works and using it to put
the files where they want them is the packager's job.



Even with configure command line, I still got the double usr mess because
of the way the configure script hardcode a few paths.
--
./configure --prefix=/usr --build=x86_64-pc-linux-gnu


 This configure is not autotools-generated. Those --build and --host
options will not do anything, and it does not necessarily interpret
every option exactly as an autotools-generated configure does. Please
read the output of ./configure --help carefully.

 Also, skalibs is meant to be used by low-level software, which might
even run as process 1. You should not use --prefix=/usr for it: the
dynamic libraries, if you build them, should go into /lib, not /usr/lib.
The static libraries still go by default into /usr/lib/skalibs, because
there's no reason for them to go into /lib - nobody does development
before mounting /usr.
 This is also true, at least, for execline and s6, whose binaries should
go into /bin and not /usr/bin.

 I know there are some distributions out there that use /bin and /usr/bin,
/lib and /usr/lib indiscriminately. This is a mistake. The assumption that
/usr will be on the root filesystem, as well as the assumption that it is
on the local machine at all, is youthist and flippant; skalibs aims to be
portable to platforms where those assumptions may not be true.

 Do not use --prefix for skalibs, execline and s6, and you'll be fine.
The fine-tuning installation directories should allow you to do exactly
what you want. If there is something you cannot do, please post exactly
what you tried, what you expected, and what you got.



I did not touch s6 main package because I invested too much time in the
others...


 I commend you for your dedication. However, I daresay it was premature.
I spent a lot - and I mean a lot - of time on the configure script to make
it both sane and adaptable to anyone's needs. (Two conflicting goals.)
No matter what you want to do, if it's not too demented - and that may be
a big if with today's distributions - then there's a way for you to do it,
without having to patch the script. If you can make an autotools-generated
configure script work for you, which is more than I can say for myself,
then there's no way you won't be able to make my configure script work as
well. Please explain what you need as clearly as you can.

--
 Laurent

Re: s6 and friends 2.0

[Laurent Bercot]

 On 28/12/2014 12:56, Casper Ti. Vector wrote:

Seconded.  This can help a lot with sandboxed building.  For example,
Gentoo Portage often make use of ${DESTDIR} in the install phase:
files are "install"ed to /var/tmp/portage/blah/blah/ and then merged
into ${EPREFIX} (which usually means `/'), and do some Gentoo-specific
post-processing (inter-package file collision checking, stripping,
documentation compression, etc., the degree of which can be fine-tuned).
So this feature will be *practically* handy :)


 ${DESTDIR} is different, it's just for staging. The skarnet.org build
process supports DESTDIR, and Portage should have no problem there: leave
--prefix alone, use "make install DESTDIR=/var/tmp/portage/blah/blah",
done.

 --prefix should only be set if the final installation needs to be done
in another filesystem tree. As I said in the skaware mailing-list, this
only means that --includedir and --libdir should also be set if you don't
want the extra /usr. But it does exactly what ./configure --help says it
does. (And if it doesn't, it's a bug and I'll fix it.)



And my apology to harish badrinath for replying twice.  I am spoiled by
the `Reply-To' header in other mail lists and have not get used to check

(...)

Perhaps Laurent can add this config if appropriate?


 Nope, this is called Reply-To munging and is a bad thing. The other
mailing-lists are misguided. See http://cr.yp.to/proto/replyto.html
and http://www.unicom.com/pw/reply-to-harmful.html

 You can configure your MUA, Mutt, to identify this list as a mailing-list
and use the "reply to list" function. ('l', IIRC.) You can also
configure it to support the Mail-Followup-To header.

--
 Laurent

Re: runit-scripts gone, supervision-scripts progress

[Laurent Bercot]

 Hi Avery,
 Happy new year to you !

 Congratulations on the achievements so far, even if they're not reaching
the bar you set for yourself.

 Just a little note:


+ The ./finish concept needs development and refinement.

+ Need to incorporate some kind of alerting or reporting mechanism into
./finish, so that the sysadmin receives notifications


 ./finish is a delicate beast. It is not only run when the admin brings
the service down, which is fine, but also when the service stops in an
untimely fashion; and the service cannot start again as long as ./finish
is running. So, if anything time-consuming, or worse, blocking, happens
in ./finish, the service can be totally hosed.
 Services should do all their necessary work in ./run, before executing
into the long-lived process: when they are in ./run, it's a known and
manageable state, they are up, even if they are not ready yet. But in
./finish, it's kind of a limbo state that shouldn't be drawn out. The
service is down, but it's still doing something, can't be brought up
right now, etc. Having a service stuck in "finish" state is about as
infuriating as having a process stuck in "D" state on Linux.

 s6-supervise has a built-in protection against misbehaving ./finish
scripts: if ./finish is still around after 5 seconds, it kills it.
(With a SIGKILL. When a service is down is not the time to be polite.)
AFAICT, runsv does not have such a protection, which makes it even more
important to pay attention when writing ./finish scripts.

 One way or the other, ./finish should only be used scarcely, for clean-up
duties that absolutely need to happen when the long-lived process has died:
removing stale or temporary files, for instance. Those should be brief
operations and absolutely cannot block.
 So, if you're implementing reporting in ./finish, make sure you are using
fast, non-blocking commands that just fail (possibly logging an error
message) if they have trouble doing their job.

 The way I would implement reporting wouldn't be based on ./finish, but on
an external set of processes listening to down/up/ready notifications in
/service/foobar/event. It would only work with s6, though.

--
 Laurent

Re: runit-scripts gone, supervision-scripts progress

[Laurent Bercot]

 Hi James,



I have had a though, why not include symlinkable functionality for
halt, poweroff, shutdown, and reboot directly in s6-svscanctl


 s6-svscan can be used as a normal process, not only as process 1,
and there can be more than one scan directory on the system.
Calling s6-svscanctl "shutdown" would only be valid when applied
to the main scan directory: this implies embedding some policy
into the software. s6 is not the place to do that. However, the
s6-linux-init package I'm working on will provide "shutdown"
compatibility binaries.



and move s6-pause into s6 itself to simplify the packages


 s6-pause is a hack to have a live process that does nothing. I'm
using it as a test tool and as a placeholder for real run scripts.
But in a real installation, it should not be needed. If you need
s6-pause in a real configuration, you are probably using a
process supervision framework to implement supervision of services
that do not need a long-lived process, and this is ugly. I'm
working on finding a better design.

 Think of s6-pause as a long "sleep" process. You wouldn't want
that in your scripts, would you ? And if you *really* need that
functionality, use "sleep 2147483647". I guarantee your system will
reboot before that sleep exits. :P



Anyways, I'll be posting more frequently about getting
init-stage-1/2/3 drafted correctly and in execline script language.


 Don't let execline steal your focus. If you want to write scripts
and are more comfortable with sh, write in sh and get something
running. Converting scripts to execline is always possible later.
The only place where execline is important to have is the early
logging pipe in stage 1, and then again you only need redirfd.

 I like writing in execline because it makes chain loading a lot
easier than sh does, it's more predictable, on embedded systems the
resource savings are not negligible, and most important, I've
grown accustomed to it and can now speak it as fluently as sh.
But for early development, if you are more familiar with sh, by
all means use what you are familiar with and focus on the job, not
on the tool.

--
 Laurent

Re: runit-scripts gone, supervision-scripts progress

[Laurent Bercot]

On 03/01/2015 01:13, Avery Payne wrote:

I'm thinking "spawn to background and exit just after that".


 That solves the problem I mentioned, but creates other ones.

 If ./finish is about cleanup, and you background it, then ./run
may start again before the cleanup has completed, so there will
be competition for resources, and race conditions.
 If the service starts again, then dies again, you will have two
concurrent ./finish processes. More race conditions, unless they
are reentrant, which is a heavy constraint on a finish script.
 If the service is in a failure loop and dies faster than
cleanup completes, you will have an accumulation of ./finish
processes, which will end up eating a lot of resources. You do
not want to risk cascading failure.

 All in all, I think it's safer not to background ./finish, and
just make sure it doesn't block.



Right now I'm having an internal dialog about if I should have an
environment variable that "hints" the framework to the scripts, which in
turn would allow me to support framework-specific features.  I like the
idea but I'm concerned that it will be unmaintainable without templates.


 Welcome to the wonderful world of integration!
 As you must have guessed by now, DJB, Wayne, Gerrit and I regularly
meet in secret to find new ways to make you pull your hair out, and
we've decided that we really went too easy on you in 2014, so expect
more work in 2015. ;)

--
 Laurent

Re: Logging daemon with ISO8601 filenames

[Laurent Bercot]

On 05/01/2015 20:12, Caleb Spare wrote:

We're really happy with using runit/svlogd generally, but over time
we've found the TAI-timestamped filenames that svlogd produces, which
aren't human-readable, makes dealing with the logs too difficult.


 Can't you run a simple filter like tai64nlocal or s6-tai64nlocal when
you need to read the logs ? Doesn't it fit your workflow, is it too
inconvenient ?

 (I'm not asking this to be a jerk: I'm genuinely interested in learning
what is acceptable or not to users. As a programmer, I find the advantages
of TAI64N stamps obvious and I feel it's a very minor inconvenience to
have to use a preprocessor when reading logs; but other users have
different expectations and evaluation criteria, and it helps me to know
where they're coming from.)

--
 Laurent

Re: s6 init-stage1

[Laurent Bercot]

On 06/01/2015 02:03, James Powell wrote:

The initial init bootscript that I'm currently drafting is in
execline using the template provided by Laurent. I was going to take
the advice on using /bin/sh rather than /bin/execlineb but I recanted
that decision due to the fact I wanted the using the FIFO handling
execline provides.


 Far be it from me to discourage you from your noble quest ! But
you could write it in sh and just use the 'redirfd' command from
execline, which does the FIFO magic.
 That said, performing multiple chain loads in shell is a pain,
and you'd have to write something like
"... ; exec redirfd -wnb 1 fifo /bin/sh -c '...'" which quickly
becomes a quoting nightmare for the second part of the script. So
maybe learning execline is the right choice. ;)

 

If I am reading things correctly, assumingly, init-stage1 using the
template, I only need to correct the paths and include any mounts of
virtual kernel file systems not listed as well as get cgroups ready,
and stage any core one-time services like copying core service
scripts to the service scan directory on the tmpfs, correct, before
passing off to init-stage2 to load drivers, start daemons, etc.?


 That is correct - it's how I designed the example init-stage1 in
the s6 tarball.
 It is not the only way to proceed, but I find it the safest way,
even safer than runit, because it puts the supervision infrastructure
in place extremely early on.

 The idea is that unlike svscan and runsvdir (that poll their scan
directory every five seconds no matter what), s6-svscan can be told
to scan for services immediately (with the s6-svscanctl -a command).
So it's possible for a s6-based system to start s6-svscan on an
empty, or nearly empty, scan directory, to have it running as early
as possible; then to populate the scan directory with other
services, and trigger s6-svscan so those services start immediately.

 So, stage 1 just becomes:
 a) Do what is strictly necessary to prepare the scan directory for
s6-svscan
 b) Fork a background process that will block until s6-svscan is
guaranteed running
 c) exec into s6-svscan, which will run as process 1 until shutdown.

 a) involves virtual filesystems, maybe cgroups, and setting up
a few basic services in the scan directory: s6-svscan's own logger,
udev, and an early getty for debugging, for instance.
 b) can be done in different ways, but I've found it simple to just
open s6-svscan's logging FIFO for writing in the *normal* way. That
operation will block until there's a reader for the FIFO, and such
a reader will appear when s6-svscan's logger is up, which means that
s6-svscan is up and running and has started its own logging service.
At this point, the background process can unblock, and run what I
call init-stage2, which can mount filesystems, populate the scan
directory and call s6-svscanctl -a to start services, etc.
 c) needs a way to *not* block on the FIFO while there is no reader,
and that's why redirfd is handy here.

 Note that this scheme allows exactly every service on the system to
be supervised. Even udev. Even s6-svscan's own logger. I believe this
is the cleanest way to use a supervision infrastructure.

 There is no theoretical objection to using that scheme with runit or
daemontools too. There simply are two practical roadbumps:
 * The FIFO trick. This can be solved by using execline's redirfd even
in a daemontools/runit installation, or by adding a few lines of
special code to daemontools/runit.
 * The fact that runsvdir/svscan can wait for up to 5 seconds before
launching a requested service: there is no way to trigger a scan like
there is with s6. This makes it hard to populate a scan directory
sequentially, because the waiting times (before runsvdir picks up the
new service directory and starts the service) will add up, and the
boot process will be very slow.
 But as a proof of concept, it could be done.

--
 Laurent

Re: s6 init-stage1

[Laurent Bercot]

On 06/01/2015 09:00, Colin Booth wrote:


1. Depending on your initramfs and your on-disk layout you can skip
mounting proc and sys. I know this is the case with Debian, probably
true elsewhere as well.


 It all depends on the assumptions that init-stage2 makes, but yes,
now that you're mentioning it, mounting /proc and /sys may be
delayed, as long as none of the very early services need them.
Make sure the login process and interactive root shell do not need
them either, because if init-stage2 fails very early, being able to
log in will make debugging/recovery a lot easier.



2. If you aren't starting udev until init-stage2, you'll need to
manually mknod null and console devices before the "Reopen
stdin/stdout/stderr" comment.


 That only applies to people who want a static /dev. Most people
will run some flavour of udev, and will probably want to keep the
devtmpfs mounted on /dev, in which case the kernel exports
/dev/null and /dev/console itself. (Probably with the wrong rights,
but they're functional enough to get by until udev runs.)



3. You'll need to either symlink /tmp into your tmpfs, mount a tmpfs
on /tmp as part of init-stage1, or remount / to rw before s6-svscan is
loaded. Otherwise the catch-all logger won't be able to do its thing
as written. Same deal with /service, though that one is documented and
expected.


 Actually, neither of those 3 things are needed for /tmp. :)
 What *is* needed is a writable-by-root-only directory, to store the
information init needs:
 - The scan directory, which must be rw
 - rw places to store the supervise/ and event/ subdirectories of
the service directories, or a copy of the service directories
themselves
 - a rw place for the catch-all logger to run

 /tmp is not ideal for this, for several reasons. One of which is
as soon as stage 2 begins and user stuff runs on the system, creating
files in /tmp isn't absolutely secure anymore, because filenames can
be predicted and DoSsed. Another reason is conceptual: the information
we need to store is not exactly temporary, it's not the throwaway
stuff you'd expect to see in /tmp - on the contrary, it's vital to the
system. So it's very unsightly to put it in /tmp.

 I very much dislike having / read-write. In desktops or other systems
where /etc is not really static, it is unfortunately unavoidable
(unless symlinks to /var are made, for instance /etc/resolv.conf should
be a symlink to /var/etc/resolv.conf or something, but you cannot store,
for instance, /etc/passwd on /var...)
 But on servers and embedded systems, / should definitely be read-only.
Having it read-write makes it susceptible to filesystem corruption,
which kills the guarantee that your machine will boot to at least a
debuggable state. A read-only / saves you the hassle of having a
recovery system.
 So, it should be the admin's choice, and I do not want s6 to force
the admin to mount / rw.

 That is why I'm saying that s6 needs a tmpfs, distinct from /tmp,
made in stage 1. Having a "private" tmpfs allows init to store the
scan directory, the copies of service directories, and the catch-all
logger directory, without impacting the rest of the system.
 Since that tmpfs is needed anyway, /tmp might as well be a symlink
to a public (mode 1777) subdirectory of it: it makes /proc/mounts
cleaner. But it's not a requirement, and /tmp may be mounted as a
separate tmpfs at some point in stage 2.

 If you are reckless, totally insensitive to gracefulness, and you
absolutely cannot deal with creating a tmpfs just for the sake of s6,
you may try to use a subdirectory of the devtmpfs in /dev as an
early root-only read-write place.
 You will now forget I suggested that. *flash*



4. If you don't want to have your dev mount in /mnt/tmpfs/dev (mostly
to keep ps output non-ugly and to kind-of stick to the FHS)


 Eh, the FHS doesn't say that /dev should be a real directory. It can
be a symlink all right. I checked. :P
 Most Linux people will use udev, though, and for them /dev will be a
devtmpfs: a real directory, and a mountpoint.



5. I made a few more classes of services for init-stage2 to copy into
the service directory. Specifically for things that I wanted running
ASAP and were udev agnostic. Those were: syslogd (using s6-ipcserver
and ucspilogd), klogd, cron, and udev. Mostly that was because I
needed udev running (and supervised) before bringing up dbus, and I
wanted to make sure /dev/log had a reader before I started bringing
anything up that might not want to talk to stdout instead (openssh,
I'm looking at you).


 The order in which init-stage2 starts services and interleaves them
with one-shot commands should mirror your dependency graph. This is
where a dependency management system would come in handy; I plan to
work on a program that takes a dependency graph as its input (format
TBD) and outputs a suitable init-stage2 script.

 (Crazy idea brewing. Dependency graph management is a solved problem:
it's exactly what "make" does. So my program could simp

Re: s6 init-stage1

[Laurent Bercot]

On 06/01/2015 13:12, Peter Pentchev wrote:

Even better: most modern systems have a tsort(1) utility for this kind of
topological sorting; BSD-derived systems have had it for ages.


 Interesting. Thanks for the heads-up - I had heard of tsort, but didn't
know exactly what it does.

 However, I'd like a tool that knows what steps it can parallelize.
A sequential output is great for functions name in a piece of code,
but for services, the point is to start as many as possible in
parallel, and minimize the amount of synchronization points.

For instance, given
1 2
3 4
meaning 2 should happen after 1, and 4 should happen after 3,
tsort gives
1
3
2
4
but instead, I need something like
1 3
2 4
because 1 and 3 can happen in parallel, and same for 2 and 4.

 AFAICT, tsort cannot do that. (make might not be able to either,
but since it's more complex, it's harder to tell.)

--
 Laurent

Re: thoughts on rudimentary dependency handling

[Laurent Bercot]

 I'm not sure exactly in what context your message needs to be taken
- is that about a tool you have written or are writing, or something
else ? - but if you're going to work on dependency management, it's
important that you get it right. It's complex stuff that needs
planning and thought.



* implement a ./needs directory.  This would have symlinks to any
definitions that would be required to run before the main definition can
run.  For instance, Debian's version of lightdm requires that dbus be
running, or it will abort.  Should a ./needs not be met, the current
definition will receive a ./down file, write out a message indicating what
service blocked it from starting, and then will send a "down service" to
itself.


 For instance, I'm convinced that the approach you're taking here actually
takes away from reliability. Down files are dangerous: they break the
supervision chain guarantee. If the supervisor dies and is respawned by
its parent, it *will not* restart the service if there's a down file.
You want down files to be very temporary, for debugging or something,
you don't want them to be a part of your normal operation.

 I firmly believe that in order to keep boot and shutdown procedures fast
and simple, and avoid reinventing the kitchen sink, any dependency
management on top of a supervision system should work *offline*. Keep the
dependency manager out of the supervisor's way in normal operation; just
use it to generate state change scripts.

 If your dependency manager works online, you *will* bring services down
when you don't want to. You *will* have more headaches making things work
than if you had no dependency manager at all. I guarantee it.

 I don't know how to design such a beast. I'm not there yet, I haven't
given it any thought. But a general principle applies: don't do more, do
less. If something is unnecessary, don't do it. What a supervision
framework needs is a partial order on how to bring services up or down
at boot time and shutdown time, and other global state changes; not
instructions on what to do in normal operation. Stay out of the way of
the supervisor outside of a global state change.

--
 Laurent

Re: thoughts on rudimentary dependency handling

[Laurent Bercot]

On 06/01/2015 18:46, Avery Payne wrote:

1. A service can ask another service to start.
2. A service can only signal itself to go down.  It can never ask another
service to go down.
3. A service can only mark itself with a ./down file.  It can never mark
another service with a ./down file.

That's it.  Numbers 2 and 3 are the only times I would go against what you
are saying.


 So does number 1.
 When you ask another service to start, you change the global state. If
it is done automatically by any tool or service, this means the global
state changes without the admin having requested it. This is trying to
be smarter than the user, which is almost always a bad thing.

 Number 2 is in the same boat. If the admin wants you to be up,
then you can't just quit and decide that you will be down. You can't
change the global state behind the admin's back.

 Number 3 is different. It doesn't change the global state - unless there's
a serious incident. But it breaks resiliency against that incident: it
kills the guarantee the supervision tree offers you.

 I firmly believe that a tool, no matter what it is, should do what the
user wants, even if it's wrong or can't possibly work. If you cannot do
what the user wants, don't try to be smart; yell at the user, spam the
logs if necessary, and fail. But don't do anything the user has not
explicitly told you to do.

 Maybe a dependency manager needs to be smarter than that. In which case
I would call it a "global state manager". There would be the "current
state", which starts at "everything down" when the machine boots, and
the "wanted state", which starts at "everything up"; as long as those
states are not matched, the global state manager is running, implementing
retry policies and such, and may change the global state at any time,
bringing individual services up and down as it sees fit, with the
ultimate goal of matching the global current state with the global wanted
state. It could do stuff like exponential backoff so failing services
would not be "wanted up" all the time; but it would never, ever change
the global wanted state without an order to do so from the admin.

 If you want a dependency manager with online properties, I think this
is the way to do it.

--
 Laurent

Re: thoughts on rudimentary dependency handling

[Laurent Bercot]

On 06/01/2015 22:17, Avery Payne wrote:

And there's the rub.  I'm at a crossroad with regard to this because:

1. The user wants service A to run.
2. Service A needs B (and possibly C) running, or it will fail.

Should the service fail because of B and C, even though the user wants A up,

  or

Should the service start B and C because the user requested A be running?


 My answer is: there are two layers, and what to do depends on what exactly
the user is asking and whom it is asking.

 If the user is asking *A* to run, as in "s6-svc -u /service/A", and A needs B
and B is down, then A should fail.
 If the user is asking *the global state manager*, i.e. the upper layer, to
change the current state to the current state + A, then the global state
manager should look up its dependency database, see that in order to bring up
A it also needs to bring up B and C, and do it.

 If you have a global state manager, that is the entity the user should
communicate with to change the state of services. It is the only entity
deciding which individual service goes up or down; if you type
"s6-svc -u /service/A", the state manager should notice this and go
"nope, this is not the state I've been asked to enforce" and bring down
A immediately. But if you tell it to bring A up itself, then it should do
whatever it takes to fulfill your request, including bringing up B and C.

 This is Unix: for every shared resource, there should be one daemon which
centralizes access to that resource to avoid conflicts. If you want service
dependencies, then the set of service states becomes a resource, and you
need that centralization.

 I'll get to writing such a daemon at some point, but it won't be
tomorrow, so feel free to implement whatever fulfills your needs:
whoever writes the code is right. I just wanted to make sure you
don't start an underspecified kitchen sink that will end up being a
maintenance nightmare, and I would really advise you to stick to the naive,
dumb approach until you can commit to a full centralized state manager.

--
 Laurent

Re: thoughts on rudimentary dependency handling

[Laurent Bercot]

On 07/01/2015 09:16, Luke Diamand wrote:

One small concern would be that it's not enough to simply signal a
dependency to be "up" - it needs to be actually running and working
before you can start the dependent service successfully. To take my
[least-]favourite example, you can't start autofs until ypbind has
actually contacted the NIS server.


 Readiness notification is hard: it needs support from the daemon
itself, and most daemons are not written that way.

 Of course, systemd jumped on the opportunity to encourage daemon
authors to use systemd-specific interfaces to provide readiness
notification. Embrace and extend.

 Fortunately, it's not the only way. There's a very simple way
for a daemon to notify the outside world of its readiness: write
something to a descriptor.

 See http://skarnet.org/software/s6/notifywhenup.html

--
 Laurent

Re: thoughts on rudimentary dependency handling

[Laurent Bercot]

On 07/01/2015 23:25, Avery Payne wrote:

The goal is the same but the emphasis has changed.  This will be considered
a fall-back feature for those systems that do not have such a tool
available, or have constraints that force the continued use of a shell
launcher.  It is the option of last resort, and while I think I can make it
work fairly consistently, it will come with some warnings in the wiki.  For
Laurent, he wouldn't even need to lift a finger - it fully complies with
his desires out of the box. ;-)


 Eh, you don't have to. It's not for me, it's for the users. I honestly
believe it's better this way, but you may disagree, and as long as you're
the one writing the code, you're the one having the say.



Unfortunately, the envdir tool, which I use to abstract away the daemons
and settings, only chain-loads; it would be nice if it had a persistence
mechanism, so that I could "load once" for the scope of the shell script.


 Here's an ugly hack that allows you do that using envdir:
set -a
eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort | uniq -u)
set +a

 It only works for variables you add, though, not for variables you remove.
 Also, be aware that it considerably lengthens the code path... but not
significantly more than dependency management in shell already does. :P



Sshh!  Don't talk too loudly...you'll expose my secret plan! :-)  But
thanks for saying that, and you are right.  The drive of the project is to
increase accessibility of the frameworks, and in doing so, increase
adoption.  Adoption will lead to feedback, which leads to improvements,
which drives development forward, increasing accessibility, etc. in a
continuous feedback loop.


 This is a very laudable goal, if you think the way to getting into distros
is this kind of accessibility.

 I'm of the opinion that packagers will naturally go towards what gives
them the less work, and the reason why supervision frameworks have trouble
getting in is that they require different scripting and organization, so
supporting them would give packagers a lot of work; whereas sticking to
the SysV way allows them not to change what they already have.
 Especially with systemd around, which they already kinda have to convert
services to, I don't see them bothering with another out of the way
packaging design.

 So, to me, the really important work that you are doing is the run script
collection and standardization. If you provide packagers with already made
run scripts, you are helping them tremendously by reducing the amount of
work that supervision support needs, and they'll be more likely to adopt it.
 It's a thankless job, and I understand that you want to have some fun by
playing with a dependency manager ;)

--
 Laurent

Re: thoughts on rudimentary dependency handling

[Laurent Bercot]

  Here's an ugly hack that allows you do that using envdir:
set -a
eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort | uniq -u)
set +a


 Ugh, in the morning (almost) light it's even uglier than I thought,
because it won't work for values you *change* either, which could
be important for things like PATH. Or if values contains a _=
or such.
 Try this instead:

set -a
eval $(envdir ../.env env | grep -v -e ^_= -e ^SHLVL=)
set +a

 It will reexport all the variables you already have, which is
also ugly, and it still won't remove anything, but at least it
will pick up changes to existing variables.

 Ah, the shell. Everytime I doubt whether execline was worth
writing, it only takes a few minutes of /bin/sh scripting to
renew my confidence.



It will work fine. I'm attempting to pre-load values that will remain
constant inside the scope of the script, so there isn't a need to change
them at runtime.


 My point is, if you are using this construct in run scripts, remember
the scripts will inherit some environment from init - that could contain
system-wide settings such as PATH or TZ. If for some reason you need to
remove or modify those variables in a specific script, it is important
that it works as expected. :)

--
 Laurent

[announce] s6-2.0.1.0

[Laurent Bercot]

 Greetings,

 s6-2.0.1.0 is out.

 It fixes the issue that some people encountered with parallel builds
(make -j2).

 There's a new program, s6-applyuidgid, that is a more generic
s6-setuidgid, meant to be called by programs that want to drop
privileges automatically. (The s6-networking superservers do this.)

 The support for readiness notification has been improved: if a server
supports it and has been launched under s6-notifywhenup, then
s6-svwait -U correctly waits for readiness in every case.

 http://skarnet.org/software/s6/
 git://git.skarnet.org/s6

 Enjoy,
 Bug-reports welcome.

--
 Laurent

Re: I need your advice on this web page

[Laurent Bercot]

On 16/01/2015 01:05, Steve Litt wrote:

http://www.troubleshooters.com/linux/init/features_and_benefits.htm


 (I'm lacking sleep and I'm going to talk about systemd. Not a good
combination. So, apologies in advance for the rant, for the inevitable
coarse language, and for the very opinionated post.)

 Hi Steve,

 The main comment that I have to make after reading your document is that
despite your attempt at impartiality, and avowed liking of daemontools-
inspired schemes, it is still systemd-centric and biased in its favor.
Not purposefully, of course, but simply because the systemd propaganda
machine works, and has already taught you to think in systemd terms - which,
let it be said openly, are often pure marketing bullshit.

 Let's dig into some of those.


 I. Socket activation.

 This has to be my new favorite marketing buzzword. Socket activation,
people. (My sockets are activated. I put my feet into them, and now
they move. It's awesome.)
 Last summer, I wrote a post about it - and you were in that discussion,
Steve:
 
http://forums.gentoo.org/viewtopic-t-994548-postdays-0-postorder-asc-start-25.html#7581522

 The short version of it is that "socket activation", as systemd defines
it, is a hack that mixes several different already-existing concepts in a
shaker, and what you get in the end is *worse* than if you had nothing
at all - but since everything is mixed and confused, nobody notices, and
systemd can pretend it's doing that wonderful thing that no other system
does, and people believe it.
 When you think "socket activation", the questions to ask are the
following. (I wrote answers from the s6 point of view, which mostly applies
to other supervision suites too.)

 Q1. Does the init system work as a super-server pre-opening and binding
sockets so that daemons do not have to do it themselves ?

 A1. It is NOT the freaking init system's job to pre-open sockets. Doing
so requires the init system to be aware of every single socket-listening
daemon on the machine, which translates into a central registry. And that
is how you turn Unix into Microsoft Windows.
 If you need to pre-open and bind sockets all at once, use inetd. This
is exactly what inetd does, and at least it doesn't require running as
process 1, or communicating over D-Bus.
 Better, use decent superservers, such as s6-tcpserver or tcpsvd, one
per service. For Unix domain sockets, which is what systemd focuses on
(and rightly so), there's s6-ipcserver. Starting those superservers in
parallel, as any supervision suite can, will end up being just as fast as
trying to open every possible socket early on in process 1. There is
no reason at all why a superserver should be tied to an init system.

 Q2. Does the init system allow you to start processes as soon as the
sockets are open, before the servers are ready ? This is the much touted
benefit of socket activation on
 http://0pointer.de/blog/projects/socket-activation.html

 A2. HELL NO. WHY ON EARTH WOULD I WANT THAT ?
 - Doing so has a serious reliability cost: if a service ends up having
issues, but dependent services have been started and are assuming that
it's working, hilarity will ensue. I mean, you could also play Dance
Dance Revolution on a mat made of old WWII landmines. What could possibly
go wrong ?
 - It's especially twisted with logging. Sure, start your daemons
before the logger is running, no problem. That way, if anything goes
wrong, you'll have no way of knowing what happened. Have fun debugging.
 - The speed benefits are minimal at best. As I wrote in my post, daemons
can perform their first writes in parallel, but as soon as they have to
read, they block anyway, waiting for their dependencies. The only case
where daemons write and never read, and could benefit from such a scheme,
is... when they write to their logger. And, as we just saw, it is a really
good idea to write logs before the logger is guaranteed operational.
 This is just not worth it. Simply starting all the services as soon as
possible in parallel will have the same benefits - the kernel will
schedule everything to the best of its abilities, and there will be no
risk of spectacular crashes.

 Q3. Does the init system allow you to hold a copy of a bound socket for the
daemon to retrieve if it has to restart ?

 A3. This is what I call "fd-holding", and is the *only* thing of value in
socket activation. No, supervision suites do not perform fd-holding -
except, precisely, for communication between a daemon and its logger,
which is arguably the most important fd to keep open, in order to avoid
losing logs.
 And for the rest, it's a work in progress. I'm working on a fd-holding
daemon. s6-ipcserver now supports getting its socket from another source
(instead of binding it itself); so does s6-tcpserver. We will get there.
 But again, this has nothing to do with an init system. A fd-holding daemon
can, and should, run as a separate service, maintained by the supervision
suite, and providing fd-holding service to

Re: [PATCH 0/4] Add info on why process is down to statusfile

[Laurent Bercot]

 Hi Olivier !
 Thanks for the patches. A bit of discussion below.



So I wanted to have the return code/signal that occured when a process went down
available on the statusfile, as I believe this is useful information to have for
all services.


 Yes, that makes sense.

 

Looking into it, I noticed that s6 already did gather that info from wait(), and
sends it to ./finish -- that was a nice surprise, but (unless I missed it?)
totally undocumented.


 I was certain I had documented it somewhere, but I must have been wrong, 
because
I couldn't find it. So, thanks for the report, and I'll apply that patch or
update the doc somehow.



First patch fixes the first argument to ./finish, since I assume it was meant to
be able to know whether there was a return code or not, but 255 actually is a
valid return code. Second patch adds a mention of those arguments to the doc.


 Yes, I didn't think too much about that, but in hindsight, since it's possible
to use a code outside of the valid range, it's the right thing to do indeed.
What bothers me, though, is that it breaks existing scripts, so it requires a
major (as in second number in my 4-number versioning system) number change,
loud horns and blinking lights. It can't be the sneaky patch. Totally my
fault, but compatibility breaks are never fun.



Then third patch does add that info to the statusfile


 And here I'm slowing down. I agree it makes sense for the info to be there.
However, wstat is specified as an int, and the mapping of the bits inside
a wstat is totally unspecified, even if most implementations are the same
in practice. That means that wstat cannot be stored as an uint32 in the
status file; if anything, it should be stored as an uint64, to accommodate
archs where int is 64 bits. Adding 8 bytes to the status file is heavy
(all things are relative...) I'll probably end up doing that, but I need to
think about it a bit more first.



and finally as a separate
patch I've also added the signal name to s6-svstat as I think it's a nice thing
to have.


 I'm reluctant to add a number-to-signal-name conversion to a single program.
It really should be a libc function. If anything, I'll add it to skalibs, so
other programs can use the conversion too.
 More importantly, the s6-svstat output is meant to be parsable, and changing
the numbers into signal names makes parsing harder. So there should be at
least an option to deactivate the conversion.

 Next time, please express wishes before sending patches - I like design
discussions and agreement on what to do, I kinda feel trapped or forced when
I get patches, as in "I worked hard on this so you better apply them!".
I have to say your patches are pretty much on-point, though, and almost
applicable as is, so thank you for that.

--
 Laurent

Re: [PATCH 0/4] Add info on why process is down to statusfile

[Laurent Bercot]

On 19/01/2015 01:04, Olivier Brunel wrote:


Sure; though I'm totally fine if you want to discuss them, or would like
me to fix/make changes and re-send, as you see fit. That's what I
expected really.


 I have implemented your suggestions, with minor changes.

 For instance, I removed the "felldown" flag: the only case where it would
be false is at boot time, when the service hasn't been launched yet.
Using s6-svstat at this point is very unlikely and would return a transient
result, identified with "want up" in the output, unless the service has a
down file, in which case the admin is supposed to know what he's doing -
in other words, it's a corner case not worthy of ad-hoc code IMO.

 Please try the latest skalibs and s6 git snapshots and tell me if they're
working for you.

--
 Laurent

Re: [PATCH 0/4] Add info on why process is down to statusfile

[Laurent Bercot]

On 19/01/2015 21:55, Olivier Brunel wrote:

Side question: I see you haven't added support of the file "ready" into
s6-svstat, any reason?


 I hesitated, then decided against it, but that's not a strong decision.

 The argument was that s6-svstat shows the state of the service as it is
seen by s6-supervise, i.e. purely the process up/down state. Introducing
the notion of readiness into it would change things. For instance,
s6-svstat currently prints the number of seconds that the process has
been up; users may be more interested in knowing want the number of
seconds that the process has been ready, and that needs storing another
timestamp. The supervise/ready file could be used for that, but I'm
not comfortable enough yet with tightly integrating readiness into the
whole series of tools. In the beginning, I thought I could get away
with s6-notifywhenup only, but it appears that modifications are
spreading across several binaries - I need time to convince myself that
it's not gratuitous feature creep and the utility is worth the increase
in complexity. I'm paranoid like that.

--
 Laurent

Re: [PATCH 0/4] Add info on why process is down to statusfile

[Laurent Bercot]

On 20/01/2015 19:04, Olivier Brunel wrote:

I see... Well, thing is, as you said, support for the ready file is
already more than just s6-notifywhenup, since s6-supervise takes care of
removing it when needed, or s6-svwait also has support for it.


 Yeah. I added the support to s6-svstat, to see whether it fits nicely,
and it does. So it's staying.

 Available in the latest git snapshot. Numbered release probably coming
soon, but I'd like to wait a bit just in case we need to break
compatibility more: I'd rather not break it several times in a row.

--
 Laurent

Re: [PATCH 0/4] Add info on why process is down to statusfile

[Laurent Bercot]

On 21/01/2015 00:20, Olivier Brunel wrote:

Cool; I see you've also added a "tainstamp" into the ready file, that's
good but you forgot to update the doc of s6-notifywhenup as well, which
still talks of empty file.


 Fixed in current git.

--
 Laurent

Re: [PATCH 0/4] Add info on why process is down to statusfile

[Laurent Bercot]

On 21/01/2015 18:16, Olivier Brunel wrote:

So in the doc you mention how "format and contents of this file are
subject to change"; not to be a PITA but in that case I think it would
be a good idea to have functions to read (and write?) that file in
libs6, similar to what is available for the statusfile.
So if external tool wanted to use/read the ready file, they could do it
(properly).


 Yeah, I thought about it, but right now I just wanted to have something
working. I don't think the format is going to change too much, I simply
didn't want to hard commit to not changing it; I'm not planning to add
more stuff to the ready file, unless it's really related to the
service being ready, not only up.

 If/when the ready file format changes, I'll provide a clean API around
it. For now, consider that the presence/absence of the file is the API,
and s6-svstat is the API if you want the notion of time.

--
 Laurent

Re: Could s6-svscan ignore non-servicedir folders?

[Laurent Bercot]

On 21/01/2015 18:24, Olivier Brunel wrote:

I'll have to setup some scripts for different init stages, using
s6-svscan as stage 2, as you've described elsewhere. But I also want to
have a system to start (and stop) services in order. I see this whole
idea of order/dependency is something that is being talked about, but
currently not supported.


 Yes. A dependency manager is something fundamentally different from
a process supervisor, and I definitely do not want to mix the two.
They can be tightly coupled, but I'm not going to make the process
supervisor more complex to accommodate the dependency manager: all the
necessary hooks are already there.

 nosh gets it right. I don't agree with all of nosh's design principles
- especially running the system manager as process 1, which needlessly
makes it more complex and increases the height of the supervision tree -
but I absolutely agree with its concept of system manager changing the
global state by, among others, sending commands to the supervision tree
(which nosh calls "service manager"). This is the right way to perform
dependency management with a supervision infrastructure.

 And it's also a real project, not something I can add to s6 in a week.
It needs a lot of work, and it's in my long-term plans, but not in my
short-term ones.



Furthermore, I want this system of mine to include other kinds of
services, that is one-time process/scripts that needs to be run once (on
boot), and that's it. And to make things simpler, I want to have it all
work together, mixing longrun services (s6 supervised) and oneshot
services when it comes to dependency/order definition.

So I'll have servicedir of sorts, for oneshot services. And I'm planning
of having one folder, that I tend to call runtime repository, but that
would also be the scandir for s6-svscan.

Obviously though, those aren't servicedirs in the s6 meaning, they
shoudln't be supervised, so I'd like for s6-svscan to check if a folder
does in fact have a file run, and if not to simply skip/ignore it.


 Sorry, but it's going to be a frank no.
 The scandir is the place where s6-svscan does its stuff. It belongs to
s6-svscan. Private property. Don't use it for other purposes, and don't
mess with it, or you're going to get into trouble.

 That's intentional, for several reasons.

 * s6-svscan is a very, very vital and very, very low-level piece of
infrastructure. Which means:
   - The simpler it is, the better. Every line of extra code in it is
potentially a bug in your process 1. It's not the place to handle corner
cases or convenience issues. s6-svscan and s6-supervise are the two
programs where I'm *the most* paranoid about feature creep; everything
I can take out of them and put somewhere else instead, I will.
   - You should not mix interfaces. Clear separation between what is
lower-level and what is higher-level is the key to good architecture.
s6-svscan should not have to know or care about what a one-shot is.
The facts that one-shots and supervisable services should both be
handled by the same "system manager" entity is of no concern to the
supervision tree.

 * Don't mix config-time and run-time.
   Your one-shot scripts, or directories, belong to a collection of scripts
that will be called whenever the global state changes, for instance at
boot time or shutdown time. They will only be used when the state changes,
not in normal system operation. As such, they belong in your service
repository, along with the entirety of your service directories, even
those that are not active. That's read-only configuration information.
 The scandir, on the other hand, is a snapshot of what is currently live;
it's run-time information. That's not the same thing, so it doesn't
belong in the same place. (That is also why I advise making a live copy
of the active service directories: to separate runtime data from
offline data.)

 * Clarity.
   - In a scandir, it's easy to identify the servicedirs: those are
the non-hidden subdirectories of the scandir. Period. If you change
that, it will be more complex to detect what is a servicedir and what
is a one-shot. "ls" won't tell you at a glance. Remember how much of
a pain it was, figuring out what exactly happened when entering a SysV
runlevel, which scripts were one-shots and which ones spawned a daemon ?
I don't want to make the same mistake. Especially with a supervision
infrastructure, that has the concept of live data that SysV didn't have.



So, what do you think of this? Would you be willing to have s6-svscan
ignore folders not containing a run file?


 No. Keep out of the scandir. I swear that your system, and also the
software you're developing, and its users, will thank you.

 Directories are not a scarce resource: you can always make a deeper
hierarchy to properly categorize what you're doing. In your case, I
would make at least 4 subdirectories of my main work directory:
* .../service, the live scandir, containing only symlinks
* .../services, which contains the 

Re: Could s6-scscan ignore non-servicedir folders?

[Laurent Bercot]

On 21/01/2015 19:03, Steve Litt wrote:


I do too. If you have a run-once thing that quickly returns, couldn't
you just not exec the thing in the run script, and then have the last
statement in your run script write a "down" file to the service? I'm
assuming that s6 does down files the same way as daemontools.


 That's really using a supervision infrastructure for things it was
not made for. You don't want to spawn a s6-supervise process for every
one-shot script you're running!
 There have been attempts to use a process supervisor as a service
management framework before - you can check on the gmane archives.
It can be made to work, but it's always kludgy, brittle and
unsatisfying - at this rate, you're better off staying with sysvrc or
whatever your favorite system manager is.

 No, the right way to proceed is a real system manager that handles
one-shot programs too, and starts/stops daemons by feeding them to
the supervision tree, as nosh does. We don't have it yet - at least
not in pure C and working with s6/runit/daemontools - but it's on my
TODO-list, and in the meantime Avery is working on dependency scripts,
check with him.

--
 Laurent

Re: Could s6-scscan ignore non-servicedir folders?

[Laurent Bercot]

On 21/01/2015 23:26, Steve Litt wrote:

But at least, if I *absolutely must* run a one-shot, on boot, that
depends on a daemon, I have a way of doing it which involves no change
to the current runit, s6, nosh or (as far as I know) perp programs.


 Oh, you can do it with s6 already: it requires blurring the frontier
between stage 1 and stage 2, as I explain (but not enough, because it
was not the focus of the document) in
 http://skarnet.org/software/s6/s6-svscan-1.html .

 The trick is to have s6-svscan up and running before you complete
the one-shot initialization of stage 1. It's doable, if a little
tricky; the future s6-init package will do it automatically for you
(but it's at least several months away).

 What is true is that you cannot interleave one-shots and long-running
processes if you are following the runit model, that only launches
runsvdir once all the one-shots have run.

--
 Laurent

Re: Could s6-svscan ignore non-servicedir folders?

[Laurent Bercot]

On 21/01/2015 21:47, Olivier Brunel wrote:

The thing is, that you've referred to services & oneshots, whereas I
would refer to longrun services & oneshot services resp., i.e. I see
those as two types of services.


 They are both services from a "system management", higher-level, point
of view. From s6-svscan's point of view, longruns are services, oneshots
are something it's entirely irrelevant for.



In fact, a reason I want/need one folder with everything (servicedirs
for both oneshot & longrun) is to avoid issues of two services with the
same name or such complications when resolving dependencies/ordering.


 Refer to services as oneshot/foo or longrun/foo, from your system
manager's main directory. Now users can name their oneshots and their
longruns the same thing, and your dependency manager doesn't care. :)



To maybe explain/detail a bit more how what I'm planning would work: my
"service manager" is asked to start some services, it pulls dependencies
and ordering from subfolders (of servicedirs) needs/wants/after/before.
It then starts everything in order, waiting on some services to be
started before others can be. Starting a oneshot service would mean
running the start script, and waiting for it; Starting a longrun service
would mean sending the command to its supervise process, and waiting for
the event on its fifodir.

So while the service manager does start services, when it comes to
longrun ones it always delegate to s6, as it should.


 That sounds perfectly reasonable. Make sure to also have a stop script
in your oneshot directories, for when you deactivate the service.



(To be more precise, all longrun servicedirs will include a down file
when created (into /run), to make sure they're started as/when needed.
So starting a longrun service actually also includes removing the down
file once the event has been received (or when sending the start
command, I'm not sure yet which is best) to reflect the change of state.)


 Yeah, that's ugly, but that's because down files are inherently ugly.
I'd say a better solution for the initial boot (which is the only time
where you'd need down files) would be to start with an empty, or almost
empty, scandir, and have your dependency manager auto-populate it if the
longrun service it wants to start isn't being supervised yet.



I understand what you're saying regarding the scandir though. Now I'm
thinking I would have a folder with all servicedirs, oneshots &
longruns, and use a folder .scandir that would only contain symlinks for
all the longrun servicedirs.
That way, the scandir only has the servicedirs that needs to be
supervised, and the service manager still has a single place for all
services.


 That can work, but your ls output will still be confusing, and there
*will* come a day when you'll accidentally link a oneshot directory into
the scandir. And that will be fun.
 What do you have against subdirectories ?



(I might also say, that I'm planning of having this folder of
servicedirs be created on boot into /run, for all enabled services. So
it is all runtime information.)


 Well, you only need the oneshot information during state changes, i.e.
mostly boot and shutdown, and on admin intervention in any case. There's
no live, automatic state to maintain, so why have oneshot directories eat
precious tmpfs space when they could just sleep on your rootfs ?

--
 Laurent

[announce] s6-2.1.0.0

[Laurent Bercot]

 Hello,
 s6-2.1.0.0 is out, with a lot of new features.

 The less shiny stuff:

 - Bugfixes. One of them could have hurt, but fortunately didn't.
 - API compatibility break: ./finish scripts now get 256 as their first argument
(instead of 255) when the ./run script was killed by a signal.
 - Also, s6-svstat now reports readiness, the output format has changed.
 - Also, the s6_svc_main() function has disappeared.
 - The access control utilities and Unix domain socket utilities have moved
from s6-networking to s6.

 The more shiny stuff:

 - New commands: s6-svlisten and s6-svlisten1. They do for services what
s6-ftrig-listen and s6-ftrig-listen1 do for regular fifodirs.
 - A whole new set of programs has been added: a fd-holding daemon and its
clients. As well as a client library, if the command-line clients are not
enough.
 So... s6 now officially supports socket activation. And it doesn't only work
on Linux - but at least on Free/Open/NetBSD, Solaris and MacOS X too.
Spread the hype.

 http://skarnet.org/software/s6/
 git://git.skarnet.org/s6

 New skalibs and execline versions have also been released; this version of s6
depends on them, so grab them too.

 Enjoy,
 Bug-reports always welcome.

--
 Laurent

[announce] s6-2.1.0.1

[Laurent Bercot]

 Hello,
 s6-2.1.0.1 is out.

 It simply fixes two bugs:
 - s6-2.1.0.0 would not build with clang
 - s6-fdholder-daemon would incorrectly rewrite the -T option (which
is why I'm making another release so soon: can't have a visible bug,
no matter how minor, in the flagship program for the release!)

 http://skarnet.org/software/s6/
 git://git.skarnet.org/s6

 Enjoy,
 Keep those bug-reports coming.

--
 Laurent

Re: [announce] s6-2.1.0.0

[Laurent Bercot]

On 27/01/2015 15:50, post-sysv wrote:

Were these by any chance inspired by aux/listen and aux/listen1 from Plan 9?


 They were not - for the simple reason that I don't know Plan 9 at all.
(It's on my list of "nice things to study when I get the time", which
probably won't happen for 2 or 3 decades.)

 But if it means that Plan 9 has an integrated notification framework, that's
one more item to add to the list of good things I've heard about it.

--
 Laurent

Re: sv check exit code not always helpful

[Laurent Bercot]

On 08/02/2015 09:12, Colin Booth wrote:


To get the functionality you want the dependent service should, as
part of its runscript, do something like 'sv status $otherservice |
grep "run"' to see if runsv considers the service to be running
followed by a call to 'sv check $service' to see if the service is
available.


 Note, however, that there's a race condition: the service can die
between the instant when the status is read and the instant when
readiness is checked.

 Avoiding that race condition isn't very simple - that's the whole point
of the libftrig mechanism in s6, and of the complexity of the s6-svwait
and s6-svlisten1 programs. But even without a notification mechanism or
precise state tracking, it would make sense for sv check (which is a
"readiness polling" mechanism) to exit nonzero when the service is down,
so it is never reported as ready when it isn't.

--
 Laurent

Re: runit source control?

[Laurent Bercot]

On 09/02/2015 18:50, Buck Evan wrote:

Is there truly no public access to the source control for runit?
If so, it's decidedly odd.


 Not that odd.
 Small projects don't really need an SCM. Most pre-git SCMs are
cumbersome or impractical. runit predates git. Gerrit chose to
focus on the code, not on the packaging - that's all there is
to it.

 I only started using a SCM, for s6 and other projects, a few months
ago, because people wanted me to. I have no regrets, and enjoy the
convenience of git, but it wasn't, strictly speaking, necessary.

--
 Laurent

Re: mailing list archive semi-broken

[Laurent Bercot]

 Yeah, it's a bug in ezmlm-cgi, and I haven't investigated it yet.
Not sure if it's non-ascii content or something else, but some messages
just make ezmlm-cgi crash. I'll look into it and hopefully fix it  (or
file a precise bug-report) at some point. Not a priority though, since
mail-archive.com is working well.

--
 Laurent

Re: mailing list archive semi-broken

[Laurent Bercot]

 In the meantime, I've set up a more user-friendly behaviour when a crash
happens. I should have done it sooner.

--
 Laurent

Re: s6, musl and ENOENT

[Laurent Bercot]

On 15/02/2015 18:50, Gorka Lertxundi wrote:

s6-log: warning: unable to read current time - timestamps may be wrong for a 
while: No such file or directory


 Hi Gorka,
 Please try pulling the latest git head of skalibs, and recompiling
skalibs (then s6 if you linked against the static version of skalibs).
Then please tell me if it fixed the problem.

 New releases should come out some time next week.

--
 Laurent

Re: s6, musl and ENOENT

[Laurent Bercot]

It was my fault, I thought the problem was related to standard libc vs musl but 
it wasn’t.


 No, it definitely wasn't. :)
 I thought that might be a problem in skalibs, that's why I advised you
to try with the latest build, but it apparently wasn't either. Glad
you could make it work !

--
 Laurent

[announce] skalibs-2.3.0.0, execline-2.0.2.1, s6-2.1.1.0

[Laurent Bercot]

 Hello,

 * skalibs-2.3.0.0 is out. (Needed for the s6 release.)
 Mostly bugfixes, and some additions for siovec management.
 The "major" number has been bumped because some functions
(buffer_getvall, buffer_putvall) changed interfaces.

 http://skarnet.org/software/skalibs/
 git://git.skarnet.org/skalibs


 * execline-2.0.2.1 is out. (Needed for the s6 release.)
 Bugfix release (ifthenelse wasn't working properly).
 Also, a -D option has been added to backtick, to have a
default value in case the called program fails.

 http://skarnet.org/software/execline/
 git://git.skarnet.org/execline


 * s6-2.1.1.0 is out.
 - Access control for listing has been added to s6-fdholderd.
 - s6-log has been overhauled.
   * The -t and -e options have been deprecated, as well as the "e"
directive (which is now called "2"). They still work, but s6-log
prints a warning message to stderr when it encounters them; and they
will disappear in a future version.
   * New "t" and "T" directives have been added for timestamping.
"T" prints ISO 8601 stamps, for those who insist on it.
   * A new "1" directive allows logging to stdout. This is useful,
for instance, if a service signals readiness via a precise log line:
selecting that log line and piping it to a listening program on
stdout (that then writes a 'U' event to the service's fifodir) gives
more options for readiness notification support.
   * The script now lives in the stack, not the heap. This is mostly
an artistic change, but if stars align correctly, it may save you a
page of RAM.

 http://skarnet.org/software/s6/
 git://git.skarnet.org/s6

 Enjoy,
 Bug-reports welcome as always.

--
 Laurent

Re: [announce] skalibs-2.3.0.0, execline-2.0.2.1, s6-2.1.1.0

[Laurent Bercot]

On 16/02/2015 22:43, James Powell wrote:

Thanks as always Laurent. Is there any news on any future s6/skarnet software 
to tease us with? :D


 Not for the near future, I'm afraid. But I'm working on good stuff,
and this year will be interesting. ;)

--
 Laurent

Re: from multiple sources to multiple destinations using s6-log

[Laurent Bercot]

On 18/02/2015 09:12, Gorka Lertxundi wrote:

Imagine I have a single process which generates multiple kind of logs, for 
example, nginx, or even better mysql, which could generate three types of logs: 
general, error and slow-query. Piping all of them to the same destination in 
order to be eatable by s6-log, will converge typologies into one log processing 
chain.


 Hi Gorka,
 I'm not sure what your objective is:

 1. Being able to pipe several log sources into the same pipe to a
unique s6-log process, which would then select lines on a token given
in the input, and log into a separate logdir for each original source;

 or

 2. Being able to have several supervised s6-log processes all reading
from the same service, one per log flux.

 I don't recommend going after 1. Generally, if you have several log
sources, you don't want to merge them into a single destination - that's
the syslogd design and is inefficient. The sources are different for a
reason, and it's always easier to concatenate several sets of logs for
analysis than it is to parse information in a single set to retrieve the
origin of a log line. But from this diagram:


general   -> /dev/mypipe1 (prepend “g:”)  --\ /-- if g  s6-log 
/generalpath
error -> /dev/mypipe2 (prepend “e:”)  ---|— /dev/stdout -|--- if e  s6-log 
/errorpath
slow-quey -> /dev/mypipe3 (prepend “sq:”) --/ \-- if sq s6-log 
/slow-query


it looks like you're going after 2, and feel constrained by the need to
pipe all the logs into the service's stdout for them to be picked up
by a logger.

 So my answer to 2 is: don't feel constrained. The "one logger per
service" model isn't an absolute one, it's just a special treatment
of the common case by s6-svscan to make it easier; you can have as
many log fluxes as you need, there just won't be any specific help
from s6-svscan.

 - Create a supervised service (without its own logger!) that runs a
s6-log process reading from a fifo for every flux you have: i.e.
a "mysqld general-path log" service, a "mysqld error-path log" service,
and a "mysqld slow-query log" service.
 - In your mysqld's run script, redirect the outputs of your daemon
to the appropriate fifo.
 - There, you're done. But if you want the same guarantees of not
losing any logs when you restart a s6-log process as s6-svscan gives
you with a service's "native" logger, you should perform two more
steps:
   * Make sure you have a s6-fdholder-daemon service running.
   * At some point in your initialization scripts, open each of your
fifos for reading and store the descriptor into the fd-holding daemon.

 Now your fifos will never close - unless you kill the fdholder without
transferring its contents first, so don't do that before shutdown time.

 Honestly, now that s6 can perform fd-holding, the specific logger
handling performed by s6-svscan becomes largely irrelevant. I may even
deprecate it in a very distant future. (Very distant! Please don't
freak out.)

 HTH,

--
 Laurent

Re: from multiple sources to multiple destinations using s6-log

[Laurent Bercot]

On 18/02/2015 15:31, Gorka Lertxundi wrote:

s6-fdholderd[20489]: segfault at 104fb ip 0040cd03 sp 7fff8ffca460 
error 4 in s6-fdholderd[40+17000]


 Wow. A segfault is bad indeed.
 Please send me:
 - the name and version of your OS and compiler
 - your skalibs sysdeps file
 - if possible, a system call trace output of s6-fdholderd.
To do that, add "strace -v", or whatever your tracer's invocation
command is, in front of "s6-fdholder-daemon" in your run script;
make sure to add the options to continue tracing after execve()
(it's the default for strace, not sure about the other ones).
strace will then dump its output to your fd-holder service's logger.
ktrace works differently, but you should find the ktrace.out in the
service directory. Not sure about truss.



- I'm missing something? Segfault seems that something is going really 
wrong in here.


 You're not missing anything, there's something wrong indeed and I
can't reproduce on my usual platform. I will need all the information
you can give me to debug this.



- How would be the steps to create&use a pipe and forward log messages to 
it?


 * mkfifo /service/mysqld-log-general/fifo
 * In /service/mysqld-log-general/run, redirect stdin to fifo before
running s6-log
 * In /service/mysqld/run, or in the appropriate configuration file,
give the right options to mysqld so it writes its general log to
/service/mysqld-log-general/fifo



   1.- s6-mkfifodir or mkpipe?


 Not s6-mkfifodir, a fifodir is not a fifo ;)



   2.- How to register into the fdholder store?
   obviusly this won't work: s6-ipcserver-socketbinder /my/pipe 
s6-fdholder-store /etc/s6/fdholder/socket MYPIPE?


 Nope, s6-ipcserver-socketbinder creates a socket, not a pipe.
 Try "redirfd -rnb 0 /my/pipe s6-fdholder-store /etc/s6/fdholder/socket MYPIPE",
once the segfault is fixed...



   3.- How to pipe from holder to s6-log?


 You don't need to. Just read normally from the fifo.
 s6-fdholderd is just there to keep the reading end of the fifo open
in case s6-log dies.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

 (Moving the discussion to the supervision@list.skarnet.org list.
The original message is quoted below.)

 Hi Dreamcat4,

 Thanks for your detailed message. I'm very happy that s6 found an
application in docker, and that there's such an interest for it!
skaw...@list.skarnet.org is indeed the right place to reach me and
discuss the software I write, but for s6 in particular and process
supervisors in general, supervision@list.skarnet.org is the better
place - it's full of people with process supervision experience.

 Your message gives a lot of food for thought, and I don't have time
right now to give it all the attention it deserves. Tonight or
tomorrow, though, I will; and other people on the supervisionlist
will certainly have good insights.

 Cheers!

-- Laurent


On 25/02/2015 11:55, Dreamcat4 wrote:

Hello,
Now there is someone (John Regan) who has made s6 images for docker.
And written a blog post about it. Which is a great effort - and the
reason I've come here. But it gives me a taste of wanting more.
Something a bit more foolproof, and simpler, to work specifically
inside of docker.

 From that blog post I get a general impression that s6 has many
advantages. And it may be a good candidate for docker. But I would be
remiss not to ask the developers of s6 themselves not to try to take
some kind of a personal an interest in considering how s6 might best
work inside of docker specifically. I hope that this is the right
mailing list to reach s6 developers / discuss such matters. Is this
the correct mailing list for s6 dev discussions?

I've read and read around the subject of process supervision inside
docker. Various people explain how or why they use various different
process supervisors in docker (not just s6). None of them really quite
seem ideal. I would like to be wrong about that but nothing has fully
convinced me so far. Perhaps it is a fair criticism to say that I
still have a lot more to learn in regards to process supervisors. But
I have no interest in getting bogged down by that. To me, I already
know more-or-less enough about how docker manages (or rather
mis-manages!) it's container processes to have an opinion about what
is needed, from a docker-sided perspective. And know enough that
docker project itself won't fix these issues. For one thing because of
not owning what's running on the inside of containers. And also
because of their single-process viewpoint take on things. Andy way.
That kind of political nonsense doesn't matter for our discussion. I
just want to have a technical discussion about what is needed, and how
might be the best way to solve the problem!


MY CONCERNS ABOUT USING S6 INSIDE OF DOCKER

In regards of s6 only, currently these are my currently perceived
shortcomings when using it in docker:

* it's not clear how to pass in programs arguments via CMD and
ENTRYPOINT in docker
   - in fact i have not seen ANY docker process supervisor solutions
show how to do this (except perhaps phusion base image)

* it is not clear if ENV vars are preserved. That is also something
essential for docker.

* s6 has many utilities s6-*
 - not clear which ones are actually required for making a docker
process supervisor

* s6 not available yet as .deb or .rpm package
 - official packages are helpful because on different distros:
+ standard locations where to put config files and so on may differ.
+ to install man pages too, in the right place

* s6 is not available as official single pre-compiled binary file for
download via wget or curl
- which would be the most ideal way to install it into a docker container


^^ Some of these perceived shortcomings are more important /
significant than others! Some are not in the remit of s6 development
to be concerned about. Some are mild nit-picking, or the ignorance of
not-knowning, having not actually tried out s6 before.

But my general point is that it is not clear-enough to me (from my
perspective) whether s6 can actually satisfy all of the significant
docker-specific considerations. Which I have not properly stated yet.
So here they are listed below…


DOCKER-SPECIFIC CONSIDERATIONS FOR A PROCESS SUPERVISOR

A good process supervisor for docker should ideally:

* be a single pre-compiled binary program file. That can be downloaded
by curl/wget (or can be installed from .deb or .rpm).

* can take directly command and arguments. With argv[] like this:
 "process_supervisor" "my_program_or_script" "my program or script
arguments…"

* will pass on all ENV vars to "my_program_or_script" faithfully

* will run as PID 1 inside the linux namespace

* where my_program_or_script may spawn BOTH child AND non-child
(orphaned) processes

* when "process_supervisor" (e.g. s6 or whatever) receives a TERM signal
   * it faithfully passes that signal to "my_program_or_script"
   * it also passes that signal to any orphaned non-child processes too

* when my_program_or_script dies, or exits
   * clean up ALL remaining non-child

[announce] s6-2.1.1.2

[Laurent Bercot]

 Hello,

 s6-2.1.1.2 is out. It fixes:
 - a bug in s6-log that wasn't parsing correctly multiple sets of
selections/actions;
 - a bug in s6-fdholderd that would crash when provided with certain
configurations.

 It depends on skalibs-2.3.1.0 and execline-2.1.0.0, also newly released.

 http://skarnet.org/software/skalibs/
 http://skarnet.org/software/execline/
 http://skarnet.org/software/s6/
 or
 git://git.skarnet.org/skalibs
 git://git.skarnet.org/execline
 git://git.skarnet.org/s6

 Enjoy,
 Keep those bug-reports coming.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 25/02/2015 21:23, Gorka Lertxundi wrote:

Laurent, any differences between `kill -15 1` and `s6-svscanctl -t 
/path/to/whatever`?


 Well, maybe it's not the case with a fake process 1 created by Docker,
but normally "kill -15 1" just doesn't work. The kernel is supposed
to ignore any signal sent to process 1 from another process.
(The kernel itself can raise signals in process 1, though: for instance,
on Linux, a SIGINT when you Ctrl-Alt-Del on the console, if
/proc/sys/kernel/ctrl-alt-del is 0.)

 So, to be safe, use s6-svscanctl -t.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 26/02/2015 13:00, Dreamcat4 wrote:

^^ Okay so this is what I have been trying to say but Gorka has put
more elegantly here. So you kindda have to try to support both.


* start your supervisor by default: docker run your-image
* get access to the container directly without any s6 process started:
docker run your-image /bin/bash
* run a custom script and supervise it: docker run your-image /init
/your-custom-script


 I'm still catching up on Docker and this thread, and working on other
small things, and I will prepare a general reply from s6's point of
view, but I have a suggestion for this part: I believe it can be
implemented in a simple fashion.

 - When the image starts, /init sets up and starts the supervision tree,
but not before forking a "stage 2" process that blocks until the
supervision tree is operational; the "init-stage1" script in the
examples/ subdirectory of the s6 package shows how it can be done.
 - If any arguments have been given to /init, the stage 2 process
takes them. ("background { init-stage2 $@ }")
 - The stage 2 process can perform any one-time initialization that
is needed for the image to be fully operational.
 - If stage 2's command line is empty, the stage 2 process simply exits;
the whole container keeps running, with a supervision tree inside.
 - If stage 2's command line is not empty, the stage 2 process execs
into "foreground { $@ } s6-svscanctl -t /service".
This will run the given command line, then shut down the container.

 This gives the client the following options:

 * docker run image commandline
   Runs commandline in the image without starting the supervision environment.
 * docker run image /init
   Starts the supervision environment and lets it run forever. It can be
stopped from the outside via "docker run image s6-svscanctl -t /service".
 * docker run image /init commandline
   Runs commandline in the fully operational supervision environment. When
commandline exits, the supervision environment is stopped and cleaned up.
 * In all cases, everything will be run with the right environment
transmitted from the docker invocation.

 If that is the flexibility you want, it can definitely be achieved, and
it's really easy to do - it's just a question of scripting around the
s6 primitives. (More on that later.)

 While I'm at it: I definitely recommend checking out Alpine Linux if
you're looking for a lightweight, practical distribution that makes
static linking easy. The maintainers are regular contributors to the
musl libc mailing list and seem intent on keeping things working.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 26/02/2015 14:11, John Regan wrote:

Just to clarify," docker run" spins up a new container, so that wouldn't work for 
stopping a container. It would just spin up a new container running "s6-svscanctl -t 
service"

To stop, you run "docker stop "


 Ha! Shows how much I know about Docker.
 I believe the idea is sound, though. And definitely implementable.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 26/02/2015 15:38, John Regan wrote:

When you build a Docker image, the ENTRYPOINT is what program you want
to run as PID1 by default. It can be the path to a program, along with some
arguments, or it can be null.

CMD is really just arguments to your ENTRYPOINT, unless ENTRYPOINT is
null, in which case it becomes your effective ENTRYPOINT.

At build-time, you can specify a default CMD, which is what gets run
if no arguments are passed to docker run.


 OK, got it.



I'm going to call these case 1 (command with no supervision environment), case
2 (default supervision environment), and case 3 (supervision environment with
a provided command).

Here's a breakdown of each case's invocation, given a set ENTRYPOINT and CMD:

ENTRYPOINT = null, CMD = /init

* Case 1: `docker run image commandline`
* Case 2: `docker run image
* Case 3: `docker run image /init commandline`

ENTRYPOINT = /init, CMD = null

* Case 1: `docker run --entrypoint="" image commandline`
* Case 2: `docker run image`
* Case 3: `docker run image commandline`


 Makes sense so far.



Now, something worth noting is that none of these command run interactively -
to run interactively, you run something like 'docker run -ti ubuntu /bin/sh'.

-t allocates a TTY, and -i keeps STDIN open.

So, I think the right thing to do is make /init check if there's a TTY
allocated and that STDIN is open, and if so, just exec into the passed
arguments without starting the supervision tree.


 ... but here I disagree.
 Performing different actions when a fd is a tty from when it is not
breaks the principle of least surprise. It's counter-intuitive,
generally bad practice, and simply not what you want in most cases.
It's okay for simple user interface stuff, such as programs that
pretty-print their output when stdout is a terminal, but in the
docker container case I don't think it's justified.



ENTRYPOINT = /init (with TTY/STDIN detection), CMD = null

* Case 1: `docker run -ti image commandline`
* Case 2: `docker run image`
* Case 3: `docker run image commandline`

So basically, if you want to run your command interactively with no execution
environment, you just pass '-ti' to 'docker run' like you normally do. If
you want it to run under the supervision tree, just don't pass the '-ti'
flags. This makes the image work like pretty much every image ever, and
the user doesn't ever need to type out "/init".


 The main problem with that approach is that there's no way to run an
interactive command with the supervision environment - but it's something
that users are very likely to need. What if I want my services to run,
but at the same time I want a shell inside the container to explore and
debug things ?

 I think you're better off with:

 * Case 1 : docker run --entrypoint="" image commandline
(with or without -ti depending on whether you need an interactive terminal)
 * Case 2 : docker run image
 * Case 3: docker run image commandline
(with or without -ti depending on whether you need an interactive terminal)

 docker run --entrypoint="" -ti image /bin/sh
would start a shell without the supervision tree running

 docker run -ti image /bin/sh
would start a shell with the supervision tree up.



Laurent, how hard is it to check if you're attached to a TTY or not?


 It's not hard at all:
 http://pubs.opengroup.org/onlinepubs/9699919799/functions/isatty.html
 (then you need a command line that does this, but I think busybox
provides one, and in the worst case it's trivial to write.)
 However, I don't think it's a good idea to do so in this case.

--
 Laurent

Re: Some wishes for s6-log & s6-envdir

[Laurent Bercot]

 I fully agree with Colin on both points.

 - For the s6-envdir thing: all the tools already exist to perform
what you want. Sure, adding an option isn't hard, and would make
things convenient, but it's a slippery slope. I try to provide
tools that perform basic operations and rely on users to combine
them to get the results they want. If I start adding options here
and there to minimize the number of executables you have to call,
what is convenience, what is redundancy, and what is outright
bloat ? Those are questions I'd rather not have to ask myself
all the time.

 - For the s6-log thing: prepending a line with a string is
definitely work for a processor. A trivial sed one-liner at
that. If you need processing after s6-log has forwarded a
line, pipe it into another program.
 Sure, the "p" option you're suggesting is a simple one, it
would be easy to implement, but then again, where to stop ?
Why limit ourselves to fixed strings ? Why not a more elaborate
format ? Slippery slope again - I'd rather draw the line
early and keep line modifications outside of s6-log.
(Timestamps are a special case. There's specific code to handle
them already and more line modifications would mean more
specific code.)

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 26/02/2015 21:53, John Regan wrote:

Besides, the whole idea here is to make an image that follows best
practices, and best practices state we should be using a process
supervisor that cleans up orphaned processes and stuff. You should be
encouraging people to run their programs, interactively or not, under
a supervision tree like s6.


 The distinction between "process" and "service" is key here, and I
agree with John.


 There's a lot of software out there that seems built on the assumption that
a program should do everything within a single executable, and that processes
that fail to address certain issues are incomplete and the program needs to
be patched.

 Under Unix, this assumption is incorrect. Unix is mostly defined by its
simple and efficient interprocess communication, so a Unix program is best
designed as a *set* of processes, with the right communication channels
between them, and the right control flow between those processes. Using
Unix primitives the right way allows you to accomplish a task with minimal
effort by delegating a lot to the operating system.

 This is how I design and write software: to take advantage of the design
of Unix as much as I can, to perform tasks with the lowest possible amount
of code.
 This requires isolating basic building blocks, and providing those building
blocks as binaries, with the right interface so users can glue them
together on the command line.

 Take the "syslogd" service. The "rsyslogd" way is to have one executable,
rsyslogd, that provides the syslogd functionality. The s6 way is to combine
several tools to implement syslogd; the functionality already exists, even
if it's not immediately apparent. This command line should do:

 pipeline s6-ipcserver-socketbinder /dev/log s6-envuidgid nobody s6-applyuidgid -Uz 
s6-ipcserverd ucspilogd "" s6-envuidgid syslog s6-applyuidgid -Uz s6-log 
/var/log/syslogd

 Yes, that's one unique command line. The syslogd implementation will take
the form of two long-running processes, one listening on /dev/log (the
syslogd socket) as user nobody, and spawning a short-lived ucspilogd process
for every connection to syslog; and the other writing the logs to the
/var/log/syslogd directory as user syslog and performing automatic rotation.
(You can configure how and where things are logged by writing a real s6-log
script at the end of the command line.)

 Of course, in the real world, you wouldn't write that. First, because s6
provides some shortcuts for common operations so the real command lines
would be a tad shorter, and second, because you'd want the long-running
processes to be supervised, so you'd use the supervision infrastructure
and write two short run scripts instead.

 (And so, to provide syslogd functionality to one client, you'd really have
1 s6-svscan process, 2 s6-supervise processes, 1 s6-ipcserverd process,
1 ucspilogd process and 1 s6-log process. Yes, 6 processes. This is not as
insane as it sounds. Processes are not a scarce resource on Unix; the
scarce resources are RAM and CPU. The s6 processes have been designed to
take *very* little of those, so the total amount of RAM and CPU they all
use is still smaller than the amount used by a single rsyslogd process.)

 There are good reasons to program this way. Mostly, it amounts to writing
as little code as possible. If you look at the source code for every single
command that appears on the insane command line above, you'll find that it's
pretty short, and short means maintainable - which is the most important
quality to have in a codebase, especially when there's just one guy
maintaining it.
 Using high-level languages also reduces the source code's size, but it
adds the interpreter's or run-time system's overhead, and a forest of
dependencies. What is then run on the machine is not lightweight by any
measure. (Plus, most of those languages are total crap.)

 Anyway, my point is that it often takes several processes to provide a
service, and that it's a good thing. This practice should be encouraged.
So, yes, running a service under a process supervisor is the right design,
and I'm happy that John, Gorka, Les and other people have figured it out.

 s6 itself provides the "process supervision" service not as a single
executable, but as a set of tools. s6-svscan doesn't do it all, and it's
by design. It's just another basic building block. Sure, it's a bit special
because it can run as process 1 and is the root of the supervision tree,
but that doesn't mean it's a turnkey program - the key lies in how it's
used together with other s6 and Unix tools.
 That's why starting s6-svscan directly as the entrypoint isn't such a
good idea. It's much more flexible to run a script as the entrypoint
that performs a few basic initialization steps then execs into s6-svscan.
Just like you'd do for a real init. :)


 

Heck, most people don't *care* about this kind of thing because they
don't even know. So if you just make /init the ENTRYPOINT, 99% of
people will probabl

Re: Some wishes for s6-log & s6-envdir

[Laurent Bercot]

On 26/02/2015 19:37, Olivier Brunel wrote:

saying I feel this option makes sense on its own, not just to reduce the
number of executable to call, but because s6-envdir is used to define an
environment, having a way to guarantee what said environment will be
seems a good/logical thing.


 Yes, it kinda makes sense. I see how things evolve, and if there are
a few places where I feel it's useful, I'll add the option.



But you're missing one thing here: I don't want to simply add a prefix,
I want to do that *after* the whole selection bit happened, and so
conditionally.

Otherwise I'd then need to maintain the regex to identify the lines in
both locations: service logger (s6-log) to send the line to stdout/fifo,
and then in the service reading from that fifo to determine what line
this is/prefix to use.

I would really like to avoid this. A simple sed one-liner makes sense
only if I know what the line is. In s6-log I do, because it was selected
via regex already, but with an external service I don't and need to
re-implement that whole step again...


 But you are going to perform the selection twice anyway. You're going
to select the line via regex in s6-log, and send it to a fifo with a
prefix, and the service reading from the fifo will select on the prefix.
As long as you're multiplexing data into a single fifo, you cannot
escape parsing the line again.
 Sure, adding a prefix to the line would make your second parsing very
simple, but it would still be a second parsing.



In s6-log I can select each line via regex and send both to some fifo
(via 1), but if I didn't add a prefix I don't know which event this is.
Now I could of course send all lines to that fifo and do the selection
there, but then we're saying s6-log's selection feature is useless, and
needs to be reimplemented elsewhere to be used?

The prefix directive was a simple way to benefit from the selection
feature by giving control over the output (log line, prefixed log line,
or only the prefix even).
A more complete solution would be a processor indeed, but, as in Colin's
example, that has to be an integrated part of/as a processor of s6-log
to remain a simple one-liner and benefits from the regex selection feature.


 I'm feeling more and more that what you want is a full log processing
flow, with frontends that select lines and deal them to processors that
modify them and pipe them to backends that write them to disk or network.
 And that wouldn't even be completely absurd. That would be powerful and
flexible. And that interests me.

 But s6-log itself was supposed to be mostly a backend, and the "pipe the
line to stdout" thing was added... because you wanted me to. It's on the
verge of an identity crisis; it's getting too close to the "that tool
performs a bit of everything, and isn't very good at it" state that I
absolutely want to avoid. The things s6-log is currently good at is
selecting lines and logging them to autorotated directories: a frontend
job and a backend job. And the backend part is the most important one,
because it's unique and rather tricky; the frontend part is easy to do
and can be put together with some grep and multitee magic. Now, you want
to use s6-log more as a frontend, and between the reasonably good
frontend and the good backend that it already has, you want me to add
mediocre processing. So of course I'm reluctant to do that.

 Rather than piling more stuff into s6-log (whose source is already huge
by my standards), I'd rather break it apart into several executables and
properly design a full log processing chain. Which is doable, but
requires a bit of thought to keep UI complexity under control - and I'm
working on something else atm, that I really would like to finish before
starting another serious project.



I get that, but one could say that it allows to fully benefit from the
selection feature of s6-log. One could also point out that even you saw
an interest in such a prefix, only making it hard-coded and required
when using 2, and unusable when using 1.


 2 is a special case: if anything gets sent to s6-log's stderr, it is
serious business. It was never meant to be a "normal" log flow. But I
get what you mean.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

The idea is that with a docker-targeted s6 tarball, it should
universally work on top of any / all base image.


 Just to make things perfectly clear: I am not going to make a
special version of s6 just for Docker. I like Docker, I think it's
a useful tool, and I'm willing to support it, as in make adjustments
to s6 if necessary to ease integration with Docker without impacting
other uses; but I draw the line at adding specific code for it,
because 1. it's yet another slippery slope I'm not treading on, and
2. it would defeat the purpose of Docker. AIUI, Docker images can,
and should, be standard images - you should be able to run a Linux
kernel with init=YOURENTRYPOINT and it should just run, give or take
a few details such as /dev and more generally filesystems. (I don't
know what assumption a docker entrypoint can make: are /proc and
/sys mounted ? is /dev mounted ? is / read-only ? can I create
filesystems ?)



If what Laurent says is true that the s6 packages are entirely
self-contained.


 It depends on what you mean by self-contained. If you compile and
statically link against a libc such as musl, all you need is the
execline and s6 binaries. Else, you need a libc.so. And for the
build, you also need skalibs.
 It's unclear to me what format a distributed docker image takes.
Is it source, with instructions how to build it ? Is it a binary
image ? Or is it possible to distribute both ?
 Binary images are the most practical, *but* they make assumptions
on the target architecture. Having to provide an image per target
architecture sounds contrary to the purpose of Docker.



* re-write /init from bash ---> execline
* add the argv[] support for CMD/ENTRYPOINT arguments


 I can do that with Gorka, with his permission.



   * /init (probably renamed to be 's6-init' or something like that)


 Why ? An init process is an init process is an init process, no
matter the underlying tools.
 Also, please don't use the s6-init name. I'm going to need it for
a future package and I'd like to avoid possible confusions.



* Can probably start work on the first bullet point (convert "/init"
to execline) during this weekend. Unless anyone else would rather jump
in before me and do it. But it seems not.


 Hold your horses, Ben Hur. People usually take a break during the
weekend; give at least John and Gorka some time to answer. It's
John's initial idea and Gorka's image, let them have the final say,
and work on it themselves or delegate tasks as they see fit.



* If Laurant wants to push his core s6 releases (including the docker
specific one) onto Github. Then it would be great for him to make a
"github/s6" org with Gorak, as new home for 's6', or else a git mirror
of the official skanet.org.


 I'm not moving s6's home. I can set up a mirror for s6 on github, but
I fail to see how this would be useful - it's not as if the skarnet.org
server was overloaded. (The day it's overloaded with s6 pull requests
will be a happy day for me.)
 If it's not about pulling, then what is it about ?

 (In case you can't tell: I'm not a github fan. Technically, it's a
good piece of software, git, at the core, with 2 or 3 layers of all your
standard bloated crap piled onto it - and it shows whenever you're
trying to interoperate with it. Politically, the account creation
procedure makes it very clear that github is about money first. The
only thing I like about github is the presentation, which is directly
inspired from Google. So, yeah, not much to save.)

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 28/02/2015 11:58, Laurent Bercot wrote:

  (In case you can't tell: I'm not a github fan.


 Meh. At this time, publicity is a good thing for my software,
even if 1. it's still a bit early, and 2. I have to use tools
I'm not entirely comfortable with. So I set up mirrors of
everything on github.
 https://github.com/s6 in particular.
 Pull to your heart's content and spread the word. Have fun.

--
 Laurent

Re: process supervisor - considerations for docker

[Laurent Bercot]

On 02/03/2015 14:24, Dreamcat4 wrote:

For the rough edges: Each item raised as seperate Github issue on your
repo Gorka. Laurent please check them also if you can. They are here


 Working on the overlay atm. We're aware of the issues. Please give
us some time to whip up a new version. Those things aren't instant,
especially since we're not getting paid for it (hint, hint).



* I've chosen /etc/s6/.s6-init as the destination folder for the init
scripts, would you like me to change?


 Yeah, I'll change it in the version I'll send a Gorka a pull request
about. Not because of the name, but because you really shouldn't store
anything under the scandir.

--
 Laurent

[announce] s6-2.1.2.0

[Laurent Bercot]

 Hello,
 s6-2.1.2.0 is out.

 It features new options in s6-setsid.
s6-setsid can start a process in a new process group, and even
in a new *foreground* process group. It can also be entirely
silent when it fails.

 http://skarnet.org/software/s6/
 git://git.skarnet.org/s6

 Enjoy,
 Bug-reports welcome.

--
 Laurent

[announce] s6-2.1.3.0

[Laurent Bercot]

 Hello,

 * s6-2.1.3.0 is out.
 It features new options to s6-envuidgid.
 (It needs the latest version of skalibs, grab that one too.)

 http://skarnet.org/software/s6/
 git://git.skarnet.org/s6

 Enjoy,
 Bug-reports welcome.

--
 Laurent

Re: Is it worth having shell-agnostic ./run and ./finish?

[Laurent Bercot]

On 20/03/2015 23:05, Avery Payne wrote:

question is: how difficult would it be to write a ./finish script in
execline?


 It all depends on what you want to accomplish in your finish script,
of course. But there's no reason why it should be difficult: finish
scripts are essentially cleanup duties, and should be about the same
order of complexity as run scripts. So, yes, fairly easy.

 Note, however, than unlike run scripts, you're not likely to spawn
hundreds of finish scripts at the same time. So resource consumption
isn't a real problem, and removing a /bin/sh dependency in finish
scripts is purely artistic at that point, so don't make it a priority. ;)

--
 Laurent

Re: Thoughts on S6 and Docker

[Laurent Bercot]

 Hi Aristomenis !
 Thanks for your comments. My replies in-line.



1) The S6 docs (http://skarnet.org/software/s6/
) direct me to read the INSTALL
file in a tarball, and then link to 10(!) different s6-* subprograms.
This is too much to digest. I'd love to see a quick start guide.


 I agree. The existing documentation is intended as a reference, and
it is perfectly accessible to people who have heard of supervision
before and know more or less what to expect; however, if you are new
to supervision, I understand that it can be confusing. What is lacking
is an overview, a tutorial and a step-by-step guide.

 I'm not good at writing those - and, to be perfectly honest, working
on more documentation instead of the next piece of code (which should
make all of this simpler to use) sounds like a daunting task to me.

 You're not the first one to report the difficulty of getting past the
initial bump; I know it's there, and I should address it. It's just a
question of mustering up the time and the courage to do it, because
writing a good tutorial is hard, and time-consuming.
 A simple overview should help already; I'll try and write one sooner
rather than later.



2) Mirroring the S6 code to Github is a nice idea which Laurent
mentioned in the previous thread. Laurent, may we start posting
issues to the https://github.com/skarnet/s6/
 repo? In particular, I'm having
some trouble which I describe in a later bullet point.


 I would rather centralize the discussions here, on this mailing-list.
(Or on the skaware mailing-list for technical details.) I think that
scattering information and resources would be detrimental, at least
at this stage of the project. Also, I really don't want to depend on
github for anything else than visibility.



3) I understand the reasons behind breaking S6 up into a large set of
simple utilities, but the current state of affairs is pretty extreme.
;ss /usr/bin |grep s6-* | wc -l reports 55 different utilities.


 Please check your installation. It should be 56. :)
(Plus ucspilogd, s6lockd and s6lockd-helper.)



Have you considered breaking s6 up into multiple packages, for
example "core" and extras?
 That may not be feasible, so my main feedback is that the
larger the number of utilities provided, the greater the handholding
necessary to get users up to speed.


 Yes, I understand.
 s6 had a lot less executables up until the recent releases, because,
as you said, it was more broken up: all the Unix domain socket management
stuff was in the s6-networking package.
 The addition of the s6-fdholder series of programs, which was necessary
to claim that s6 can perform "socket activation", forced me to integrate
more executables into the s6 package itself. I can't say I like it, but
it's really a lot simpler that way.
 And I agree with your conclusion: a tutorial would help. At some point,
I'll get around to writing one, I promise.


 4) I tried compiling s6 in an

Ubuntu container. ./configure froze while looking for /dev/random.
You can ctrl+c through this step, but this won't work when running
docker build, because it's noninteractive. Is there a way to bypass
this step?


 Give the --enable-force-devr option to ./configure when compiling
skalibs.


 5) For major distributions and usage mediums (Ubuntu,

docker, OS X, and perhaps Alpine/busybox), the docs should provide
clearer steps for installation and usage. John recently posted an
Alpine package, which is great. But the version available in Ubuntu
is extremely old - 0.47. Have you considered adding a Launchpad PPA?


 Sorry, but this is a can of worms I'm not going to open.
 The world of Linux distributions is a shark tank, and my very strong
opinion is that the sheer amount of them - as well as the politics and
corporate that govern most of them - is hurting software a lot, and
ultimately is hurting users. I could ramble for days about why I think
so, but the point is that I will not explicitly support or favor one
distribution over another. If I added a Ubuntu Launchpad PPA, I would
have to add equivalent support for every distribution out there, and
not only the mainstream ones. This is simply not feasible.

 Providing working tarballs and possibly a git repository is the job
of software authors, and I will do my best to ensure that the software
works, and to listen to suggestions. Providing packages is the job of
*packagers* (duh), and I have neither the time nor the inclination to
be a packager.
 If anyone wants to provide a Launchpad PPA for s6, or the equivalent
for any other distribution, I'll be very happy and grateful about it,
of course. John's work of providing s6-based container images for
various distributions is invaluable.



 6) Gorak/John: None of the s6/skalibs images are explicit about
/etc/leapseconds.dat. This confuses me. When reading through the
build scripts, the file is usually copied into the image, but I
can't find the original leapseconds

s6 overview (was: Thoughts on S6 and Docker)

[Laurent Bercot]

 I have added a documentation page to s6:
 http://skarnet.org/software/s6/overview.html

 Please read it, especially if you're still new to s6 or to
supervision in general, or if you had trouble understanding
what to do with s6 once you had installed it, or if you were
confused by the amount of executables.

 Tell me if that page helps you, or if it would have helped you
when you were trying to make sense of it all. If it's insufficient,
please explain what you would like to see, or what needs clarifying.

 You're free to hate me for not having written that page earlier, as
long as you also give constructive comments. :)

 Additional suggestions welcome!

--
 Laurent