Re: Pattern for multiple subservices and dynamic discovery i.e. VPN

2022-08-17 Thread Laurent Bercot


I'm looking for a pattern to solve a problem, where you have to
discover dynamically the services you have to start.

Examples could be VPN configurations, where you discover the
configuration files and start for every file an instance of the VPN
service.


 Hi Oliver,

 Dynamic instantiation is a real pain point - it's an often requested
feature, but it's surprisingly hard to make it work correctly and
safely in a supervision scheme. Supervision works very well in static
environments, but dynamic discovery is at odds with the architecture.

 I have a few ideas to mitigate that and help people create instanced
services. Instantiation is a planned feature of the future s6-rc v1
but it's still a ways away; I am also thinking of adding tools to help
people handle instances with regular s6, amd they may come in the near
future, but there are currently no such helpers, sorry.

--
 Laurent



Re: Be prepared for the fall of systemd

2022-08-04 Thread Laurent Bercot




What do we as a community need to do
to get S6 into a "corporate friendly" state?

What can I do to help?


 "Corporate-friendly" is not really the problem here. The problem is
more "distro-friendly".

 Distributions like integrated systems. Integrated systems make their
lives easier, because they reduce the work of gluing software pieces
together (which is what distros do). Additionally, they like stuff like
systemd or openrc because they come with predefined boot scripts that,
more or less, work out of the box.

 There are two missing pieces in the s6 ecosystem before it can be
embraced by distributions:

 1. A service manager. That's what's also missing from runit. Process
supervisors are good, but they're not service managers. You can read
why here[1].
 In case you missed it, here is the call for sponsors I wrote last year,
explaining the need for a service manager for s6: [2]. It has been
answered, and I'm now working on it. It's going very slowly, because I
have a lot of (easier, more immediately solvable) stuff to do on the
side, and the s6-rc v1 project is an order of magnitude more complex
than what I've ever attempted before, so it's a bit scary and needs me
to learn new work habits. But I'm on it.

 2. A high-level, user-friendly interface, which I call "s6-frontend".
Distros, and most users, like the file-based configuration of systemd,
and like the one-stop-shop aspect of systemctl. s6 is lacking this,
because it's made of several pieces (s6, s6-linux-init, s6-rc, ...) and
more automation-friendly than human-friendly (directory-based config
instead of file-based). I plan to write this as well, but it can only
be done once s6-rc v1 is released.

 Once these pieces are done, integration into distributions will be
*much* easier, and when a couple distros have adopted it, the rest
will, slowly but surely, follow suit. Getting in is the hard part, and
I believe in getting in by actually addressing needs and doing good
technical work more than by complaining about other systems - yes,
current systems are terrible, but they have the merit of existing, so
if I think I can do better, I'd rather stfu and do better.



Here are some ideas:
- easier access to the VCS (git, pijul, etc)


 The git repositories are public: [3]
 They even have mirrors on github.
 All the URLs are linked in the documentation. I don't see how much 
easier

I can make it.

 Note that the fact that it's not as easy to submit MRs or patches as
it is with tools like gitlab or github is intentional. I don't want to
be dealing with an influx of context-free MRs. Instead, if people want
to change something, I'd like *design discussions* to happen on the ML,
between human beings, and when we've reached an agreement, I can either
implement the change or accept a patch that I then trust will be
correctly written. It may sound dictatorial, but I've learned that
authoritarian maintainership is essential to keeping both a project's
vision and its code readability.



- Issue tracking system


 The supervision ML has been working well so far. When bigger parts
of the project (s6-rc v1 and s6-frontend) are done, there may be a
higher volume of issues, if only because of a higher volume of users, so
a real BTS may become an asset more than a hindrance at some point.
We'll cross that bridge when we get to it.



- CI/CD build chain (being careful not to make it too painful to use)


 Would that really be useful? The current development model is sound,
I think: the latest numbered release is stable, the latest git head
is development. The s6 ecosystem can be built with a basic
configure/make/make install invocation, is it really an obstacle to
adoption?

 I understand the need for CI/CD where huge projects are concerned,
people don't have the time or resources to build these. I don't think
the s6 ecosystem qualifies as a huge project. It won't even be "huge"
by any reasonable metric when everything is done. It needs to be
buildable on a potato-powered system!



- "idiot proof" website
- quick start / getting started guide
- easier access (better?) Documentation


 I file these three under the same entry, which is: the need for
community tutorials. And I agree: the s6 documentation is more of a
reference manual, it's good for people who already know how it all works
but has a very steep learning curve, and beginner-to-intermediate
tutorials are severely lacking. If the community could find the time
to write these, it would be a huge help. Several people, myself 
included,

have been asking for them for years. (For obvious reasons, I can't be
the one writing them.)

 Turns out it's easier to point out a need than to fulfill it.

 It's the exact same thing as the s6 man pages. People can whine and 
bitch

and moan for ages saying that some work needs to be done, but when
asked whether they'll do it, suddenly the room is deathly silent.
For the man pages, one person eventually stepped up and did the work[4]
and I'm forever grateful to them; I 

Re: [DNG] Be prepared for the fall of systemd

2022-08-04 Thread Laurent Bercot




I find it symptomatic of the fact that a guy wrote some Rube Goldberg code and a
corporation decided it would be a great idea to spend millions getting the Rube
Goldberg code into many major distros. As far as us running our of road with the
Unix API, systemd solves no problem and offers no improvement that couldn't have
been solved or improved in a dozen different ways, almost all of which would 
have
been more modular and less prone to vendor lock in.


 We know all of this.
 Please keep these posts out of the supervision ML. It's not that we
don't like systemd-bashing, it's that it takes a lot of space and makes
a lot of noise, and I'd like the supervision ML to be laser-focused on
solving technical issues with supervision systems, not be yet another
place where we complain about systemd day in day out. Thanks.

--
 Laurent



Re: runsvdir does not run runsv recursively, as is documented.

2022-08-03 Thread Laurent Bercot

I'm trying to set up services, which are in subdirectories of other services. 
This is supported, according to the second paragraph of the runsvdir man page:

runsvdir starts a runsv(8) process for each subdirectory,  or
symlink  to a directory, in the services directory dir, up to
a limit of  1000  subdirectories,

In my directory service/ I have:
Foo
Foo/run
Foo/bar
Foo/bar/run


 That's not what the man page means. The "subdirectories" are
subdirectories *of dir*.
 runsvdir will understand service/Foo, and service/bar, but it will not
understand service/Foo/bar.

 The exception is if you have a service/Foo/log directory, but that's
handled by runsv, not runsvdir: service/Foo/log is a logger for
service/Foo.

--
 Laurent



[announce] skarnet.org Summer 2022 release

2022-06-14 Thread Laurent Bercot



 Hello,

 New versions of some skarnet.org packages are available.

 skalibs has undergone a major update, mostly to yet again revamp
librandom. This time I am happy with the API and implementation: I 
believe

it finally addresses all the cases in a satisfying way, providing cross-
platform failure-free pseudorandom number generation with options to 
choose

between waiting until the entropy pool has been initialized and possibly
getting less cryptographically secure data if the entropy pool is too
shallow. It wasn't easy to design; it's here at last.

 Compatibility with previous skalibs version is not assured, but apart
from librandom, and one additional function, no other interface has been
modified, so the compatibility breaks are minimal and a lot of software
will still build with this version without needing any modification.

 Most of the rest of the skarnet.org software stack has undergone at 
least
a release bump, in order to build with the new skalibs; a large part of 
it

has also received some changes and fixes. Some packages did not need
changing at all: no release is provided for these, they should keep 
building

with the new stack.

 execline comes with a quality-of-life parser change: backslashes at the
end of lines are now ignored, which makes it possible to directly copy
some multiline commands from shell scripts.

 s6-linux-utils comes with a new utility, rngseed, which is an original
implementation of Jason Donenfeld's seedrng[1]. This is the work that
made it necessary to get librandom right once and for all. With rngseed,
no Linux system should ever have uninitialized entropy pool problems 
ever

again.

 The new versions are the following:

skalibs-2.12.0.0  (major)
utmps-0.1.2.0 (minor)
execline-2.9.0.0  (major)
s6-2.11.1.1   (release)
s6-rc-0.5.3.2 (release)
s6-linux-init-1.0.8.0 (minor)
s6-portable-utils-2.2.5.0 (minor)
s6-linux-utils-2.6.0.0(major)
s6-dns-2.3.5.4(release)
s6-networking-2.5.1.1 (release)
mdevd-0.1.5.2 (release)
dnsfunnel-0.0.1.4 (release)

 Details of some of these package changes follow.


* skalibs-2.12.0.0
  

 - librandom rewritten. random_init and random_finish functions removed.
The new random_buf function, which replaces random_strin), never fails.
It blocks if the entropy pool is not initialized; the new 
random_buf_early

function is the same, but does not block. random_devurandom is now
exported, but should not be needed except in very specific cases 
(rngseed).

 - New functions added: waitn_posix and waitn_reap_posix, openc*_at.
 - readnclose is now exported.
 - openreadnclose_at() now returns an ssize_t, aligning with 
openreadnclose().
You should check your code for any use of openreadnclose_at(), and adapt 
it
to the new API. (Previously it returned a size_t and the user was 
supposed
to assume an error if it didn't fill the entire length of the buffer. 
Now

errors are reported with -1.)
 - Endianness conversion primitives reworked. The nonportability of 
endian.h
and bswap has always been a pain point; the new portable functions in 
skalibs
should now be just as efficient as the system-dependent endian.h 
functions.

 - Added an implementation of the blake2s hash.

 https://skarnet.org/software/skalibs/
 git://git.skarnet.org/skalibs


* utmps-0.1.2.0
  -

 - Nothing to do with the new skalibs; utmps-0.1.2.0 has been available 
for

a while, but was never properly announced. The main feature is that
utmps-wtmpd can now take an argument naming its database file. This is
useful for implementing btmp, one of the numerous idiosyncrasies of 
historic

Linux software.

 https://skarnet.org/software/utmps/
 git://git.skarnet.org/utmps


* execline-2.9.0.0
  

 - Bugfixes.
 - The execlineb parser has been rewritten and its transition table is 
now

documented.
 - The wait command can now wait for *one* of the listed processes, in
addition to its original capability of waiting for *all* of them. It can
also stop waiting after a timeout. The new features can be used even 
when

wait is used in posix mode.

 https://skarnet.org/software/execline/
 git://git.skarnet.org/execline


* s6-linux-init-1.0.8.0
  -

 - The system scandir is now configurable at compile-time via the
--scandir configure option. It is a relative path under the tmpfsdir.
The default is still "service", for a /run/service default scandir.

 https://skarnet.org/software/s6-linux-init/
 git://git.skarnet.org/s6-linux-init


* s6-portable-utils-2.2.5.0
  -

 - s6-test now understands the =~ operator, matching its left argument
against an extended regular expression given as its right argument (this
is originally a GNU bash extension to test).

 https://skarnet.org/software/s6-portable-utils/
 git://git.skarnet.org/s6-portable-utils


* s6-linux-utils-2.6.0.0
  --

 - New command: 

Re: Unprivileged Shutdown?

2022-05-28 Thread Laurent Bercot




I have been using simple privilege escalation to poweroff the machine,
but looking through the source code for s6-linux-init-shutdownd and
friends, it appears the only constraint on interacting with the daemon
is the permissions on run-image/service/s6-linux-init-shutdownd/fifo.

The default appears to be:
600 root root
I've changed it on my system to be:
620 root power
and added my user to the power group.

This seems like the cleanest way to implement unprivileged
poweroff/reboot, but I'm concerned that it's not possible by default.
Is there a better way, or is it just meant to be done manually?


 No, you are correct that it is the right mechanism.

 Allowing unprivileged shutdown is a dangerous operation and should
only be done is very specific circumstances (i.e. when a normal user
has complete seat and console access), so it's not the default and the
mechanism is fairly hidden.
 If there's demand, I can probably write a s6-linux-init-shutdown-perms
program in a future version that would let you specify the user/group
allowed to shutdown, rather than having you manually tinker with the
fifo.

--
 Laurent



Re: s6 xinit replacement?

2022-05-14 Thread Laurent Bercot

Is the purpose of executing setsid() in s6-supervise to allow for the
services to continue beyond the termination of the supervision tree?


 It's actually the opposite: it's to protect the supervision tree
against misbehaved services. :) setsid() makes sure the service is
isolated, and a killpg() or equivalent won't affect anything outside
of it. Of course, it also protects *other* services running under the
same supervision tree.



If that's the case, could there possibly be a flag to disable that,
with the understanding that something like nohup or even s6-setsid would
be necessary to replicate that behavior?  That would enable a non-root
Xorg to be managed by s6.


 The direction s6 has taken is really the opposite: there was such a
flag in earlier versions, but it was causing a lot of special cases
and problems I definitely did not want to deal with.
 The real issue is that a supervision tree should not be run with a
controlling terminal. It's not meant to be run from a logged-in user
process, but as a background infrastructure that's always there. The
whole point of s6 is to make your services *more reliable*; and there
are few things less reliable than a whole tree of processes that can
die on an accidental ^C.

 Because users insisted a lot, there are still accommodations for
killing a whole supervision tree with ^C when s6-svscan has been 
launched
in a terminal. It is a nice feature to have (and although it was by 
design

that services persisted beyond the ^C, it was unintuitive to most users,
so from a pure UI standpoint, killing the entire tree in one go was
better).
 However, I'm not going back on the "each service runs in its own 
session"

thing, because if there's a case for allowing the user who controls
s6-svscan to kill the whole tree at once, there is just no case for
allowing a service running under the tree to impact other services and
the tree itself.

 Despite this, you're right that the pattern of xinit is similar to what
s6 does, and it *is* possible to run Xorg under s6; several users are
doing so, and I hope they will post their setup. (You may have more
luck by asking in the IRC channel, but it would be nice for the setup
to be documented on the mailing-list.) It does not involve running
s6-svscan from your own VT; it involves having a supervision tree
already running (as your user), and starting the Xorg service, e.g.
with a s6-svc -u command, on a given VT, possibly passed to the run
script via a file that your xinit emulation script would fill in
with the output of `tty`.

 If you insist on doing hacky things, you could even probably get away
with something that looks like:

xinit emulation:
#!/bin/sh
tty | sed s:/dev/tty:: > /home/me/.screen-number
s6-supervise /home/me/Xorg &
# small race here that disappears when s6-supervise has already run once
s6-svwait -U /home/me/Xorg
your-X-client
s6-svc -dx /home/me/Xorg

/home/me/Xorg/notification-fd:
3

/home/me/Xorg/run:
#!/bin/execlineb -P
backtick -E screen { cat /home/me/.screen-number }
export DISPLAY :$screen
X -displayfd 3 :$screen vt$screen

-displayfd is used as a notification mechanism, unblocking s6-svwait
when the X server is ready.

 Hope this helps (and I hope users who actually have done something
similar will share their experience),

--
 Laurent



Re: Supervision on the BSD's

2022-04-09 Thread Laurent Bercot

In searching, I found some messages on the Skaware lists about
running s6 as PID 1 on FreeBSD; has that work been published anywhere?
I'm not sure if I want to go so far as replacing PID 1 right out
of the gate, but having some existing service directories would be
nice.

 I have done some experiments and my conclusion was that to replace
pid 1 on FreeBSD, a real s6-freebsd-init package was needed, because
the way the BSDs organize their init and shutdown is radically
different from the way Linux does it, and the conversion is far from
obvious.

 However, you don't need to replace pid 1 to run s6 on a BSD. As
mentioned in https://skarnet.org/software/s6/s6-svscan-not-1.html , you
can start a supervision tree from /etc/ttys, and run your services
under it. It will work like on any other system.

 Quite a few people on the #s6 channel on IRC (OFTC network) are using
s6 on a BSD, so if you're looking for example service directories, and
various tips and tricks, I suggest you join the channel and ask them. ;)



Have I correctly understood how daemons/services work on the BSD's?
If not, what am I missing? Are the daemons included with the
distributions so incredibly stable that they don't need supervision
in order to keep the system functional?


 The BSDs are tightly integrated systems, more than "distributions", and
especially with OpenBSD, daemons are carefully audited and patched so
they are indeed super stable. Which is a very good thing - but because
of that, the BSD community tends to look down on supervision, without
understanding that it has other benefits than auto-restarting crashed
daemons.



Finally, if you wanted to create a router that you could (metaphorically)
put in a closet and forget about for 5 years, what approach would
you take? My initial thought was OpenBSD + s6, but I worry now that
there could be an impedance mismatch between these systems.


 OpenBSD + s6 will definitely work. Just make sure not to get in the
way of how OpenBSD does things; run an s6 supervision tree at boot
time and start your services under it as you see fit, independently from
OpenBSD's rc.

 Since the BSDs don't care for supervision, though, depending on
upstreams it may be difficult to find options for your packaged daemons
that stop autobackgrounding and that are not debugging options. Just a
small practical hurdle, but when it happens it can be infuriating.

--
 Laurent



Re: s6-linux-init and ttys in lxc containers

2022-03-08 Thread Laurent Bercot

s6-linux-init: warning: unable to ttyname stdout: No such device

I suspect this is due to the mechanism described on 
https://github.com/lxc/lxd/issues/1724#issuecomment-194412831, although I’m not 
using LXD, only lxc (which does not have a daemon running as root).


 You're right, it's the exact same issue. If /proc/1/fd/1 does not
point to a valid pts in the container, then ttyname() will be unable to
resolve it and you will get that warning message - and you also will be
unable to transmit your controlling terminal to stage 2 (which is not a
problem in your case).

 The only solution, I'm afraid, is to make sure your pseudo-filesystems
are correctly prepared before you launch your container, which you may
not be able to do without root privileges. Docker can do it; if you
were running lxd there would probably be a way to do it as well. But if
you have no use for a controlling terminal, you can forget about all 
this

and just ignore the warning.

--
 Laurent



[announce] Small skarnet.org update

2022-03-07 Thread Laurent Bercot



 Hello,

 New versions of some skarnet.org packages are available.

 The changes are minor, mostly quality-of-life and small additions 
driven

by the new version of s6-overlay.
 There are bugfixes all around, so users are encouraged to upgrade even
if they're not using s6-overlay.

 The new versions are the following:

skalibs-2.11.2.0(minor)
execline-2.8.3.0(minor)
s6-2.11.1.0 (minor)
s6-portable-utils-2.2.4.0   (minor)
s6-linux-init-1.0.7.3   (release)

- skalibs features better sysdep autodetection when custom compilation 
flags

are provided, and adds an option to envdir_internal() for unlimited-size
variable reading.
- execline adds the -P and -p options to getpid, to get the parent pid 
or

force the default behaviour.
- s6 features world-usable s6-applyuidgid and s6-setuidgid, as well as
a new -L option to s6-envdir for unlimited-size variable reading.
- s6-portable-utils adds a -N option to s6-dumpenv (add a newline after
dumping a variable) for easier reading back via s6-envdir.
- s6-linux-init fixes corner cases when used in containers.

 Enjoy,
 Bug-reports welcome.

--
 Laurent



Re: s6-svscan shutdown notification

2022-02-23 Thread Laurent Bercot




What's the cleanest way to wait on s6-svscan to shut down after issuing of
a SIGTERM (say s6 via-svscanctl -t)?


 Be its parent, and wait for it. :)
 On SIGTERM, s6-svscan will not exit until the supervision tree is
entirely down, so that will work.
 If you're not the parent, then you'll have to wait for a notification
somehow, but that's easy:

 When s6-svscan wants to exit, it doesn't exit right away, but
tries to exec into the .s6-svscan/finish script. So you have a clear
indicator here: when .s6-svscan/finish runs, it means the supervision
tree is down.
 So, for instance, make a finish script that writes a byte in a fifo,
and have your jail shutdown script read on that fifo. Something like:

.s6-svscan/finish:
#!/bin/sh
exec echo > /run/blah/fifo

shutdown script:
#!/bin/sh
...
rm -f /run/blah/fifo
mkfifo /run/blah/fifo
read < /run/blah/fifo &
s6-svscanctl -t /run/service
wait
...

(read on the fifo before running s6-svscanctl, to avoid the small
race condition.)



Looking at the documentation, my only option appears to be to check if the
return code of s6-svscanctl is 100, or maybe to monitor for the existence
of .s6-svscan/control (not sure if it's removed on exit). Are there any
other ways to monitor s6-svscan?


 Ew. Don't poll.
 Use .s6-svscan/finish to do anything you want to do at s6-svscan
death time.

--
 Laurent



Re: s6-log weirdness

2022-02-04 Thread Laurent Bercot

I noticed that in some cases s6-log exits cleanly but does not log
anything. What's worse, it depends on the message content.


 Hi Vallo,

 That's the difference between '!zstd -q' and '!zstd' -q ;)

 When -q isn't a part of your processor command, but a part of the
s6-log command line, it is interpreted as a selection directive,
and will filter anything that contains a 'q' character.

--
 Laurent



Re: [announce] skarnet.org Winter 2021-2022 release

2021-12-22 Thread Laurent Bercot

I think trying to explain s6-linux-init + s6-rc through the lens of
runit's stages isn't a good idea.


 Carlos is correct - both here and in his explanation of s6-linux-init
stages.

 When designing s6-linux-init, I kept runit's "stage" terminology
because at the time it was a useful framework to see where things
should go; but in retrospect, it was probably a mistake, prone to
confusion, as exemplified by Steve's message. Even though, functionally,
the "stages" in runit and s6-l-i have similarities, the way they're
implemented and work under the hood is fundamentally different.

runit's stage 1 and stage 2 are similar, their difference is only
conventional: traditionally, runit runs all the oneshots in stage 1,
and stage 2 is just an execution of runsvdir - but a setup where
/etc/runit/1 is empty and all the oneshots are performed in
/etc/runit/2 prior to the execution of runsvdir would work as well.

s6-linux-init's stage 1 and stage 2 do not have the same similarity.
Stage 1 is the early init, running as pid 1; this is only the
/sbin/init script produced by s6-linux-init-maker, which executes
into the s6-linux-init binary (which sets up the system and execs
into s6-svscan); and users are not supposed to do anything with it.
 Stage 2, on the other hand, is the real boot sequence, running as
not-pid-1; it is only run when stage 1 has completed, which means
the system has an adequate long-running pid 1 process, a supervision
tree, and a catch-all logger - all the basic infrastructure is in
place and the services can be started. With s6-linux-init, stage 2
is where all the real work happens; and when the system's boot
sequence is complete, the stage 2 script simply exits and the
system keeps running until a shutdown command is issued.

 I want to keep the "stages" framing for s6-linux-init, because I think
it is useful: these are qualitatively different parts of the init
process. (There's a "stage 3" and a "stage 4" as well, at shutdown
time: they're handled by the s6-linux-init-shutdownd daemon.) The
framing is actually *more* useful here than in runit, where "stages"
are only sequence points and the only real hardcoded meaning is that
the system shuts down when /etc/runit/3 exits.



 The preceding is the best interpretation I could put together from
https://skarnet.org/software/s6-rc/overview.html,
https://skarnet.org/software/s6-rc/s6-rc.html, and discussions with
 you. What do I need to do to make the preceding sequence accurate?


 I don't know, honestly, Steve. At this point I cannot tell whether
you're acting in good or bad faith.

 You seem to be talking about s6 and s6-linux-init, yet only mention
the s6-rc documentation. You do not seem to have read the
https://skarnet.org/software/s6-linux-init/quickstart.html page,
which explains that s6-linux-init-maker is run offline, or the
https://skarnet.org/software/s6-linux-init/s6-linux-init.html page,
where the "Early preparation" part explains how stage 1 works. You
do not seem to have watched my FOSDEM 2017 video at
https://archive.fosdem.org/2017/schedule/event/s6_supervision/ where
I describe the various duties of an init system and how the components
in the s6 suite fill the boxes.

 Despite your claims to be interested, you have not put in s6 the
tenth of the effort you put in runit. It's been going on for ages.
You say you haven't paid much attention to the progress of s6, but
over the years the fundamentals have not changed, they've been the
same for a while now; the truth is that you have never paid much
attention to s6 at all. You come to the list once in a while and
ask a question that shows you are still lacking a basic understanding
of s6, an understanding that comes from two hours of experimenting
while having a browser open with 3-4 tabs to the documentation.
And then you seem to ignore the answers, and go away until the
next time when you come back just as helpless.

 You are clearly not dumb. So either you are a troll, or you need to
get a grip and realize that if you're really interested, you can do
the work of looking up and finding the relevant documentation,
experimenting, and finally getting the reward of feeling the pieces
of the puzzle fall into place, and acquiring the understanding that
has eluded you for so long and that you seem to crave. As a technical
writer, that is *your job*, so you can make the process easier for
other people.

 s6 is not as complex as you seem to think it is, far from it. There
are real, living people who understand how it works, and they're not
all acne-ridden nerds living in a basement. The documentation may
not be perfect, but it seems to be adequate. It lacks tutorials, yes,
but I expect tutorials to be written by *people like you*, who could
do a much better job of it than I ever would, if only they'd stop
acting like damsels in distress at the first mention of a Unix pipe.

 And if you're not interested, or simply not enough to really get
into it, that's okay too; you just need to own it and 

[announce] skarnet.org Winter 2021-2022 release

2021-12-21 Thread Laurent Bercot



 Hello,

 New versions of all the skarnet.org packages are available.

 The changes are, for the most part, minimal: essentially, the new
versions  fix a bug in the build system that made cross-building under
slashpackage more difficult than intended. Very few people should
have been impacted by this bug.
 Some packages had a few more bugfixes; and some packages have
additional functionality. No major update; no compatibility break.

 The new versions are the following:

skalibs-2.11.1.0 (minor)
nsss-0.2.0.1 (release)
utmps-0.1.1.0(minor)
execline-2.8.2.0 (minor)
s6-2.11.0.1  (release)
s6-rc-0.5.3.0(minor)
s6-portable-utils-2.2.3.4(release)
s6-linux-utils-2.5.1.7   (release)
s6-linux-init-1.0.7.0(minor)
s6-dns-2.3.5.3   (release)
s6-networking-2.5.1.0(minor)
mdevd-0.1.5.1(release)
bcnm-0.0.1.5 (release)
dnsfunnel-0.0.1.3(release)
smtpd-starttls-proxy-0.0.1.1 (release)

 Dependencies have all been updated to the latest versions. They are not
strict: libraries and binaries may build with older releases of their
dependencies, although this is not guaranteed.

 You do not need to recompile your s6-rc service databases. To make use
of the new s6-linux-init functionality, however, you will have to
recreate your run-image.
 You do not need to restart your supervision tree, unless you're 
deleting

your old s6 binaries.

 Details of minor package changes follow.

* skalibs-2.11.1.0
  

 - New function: opendir_at()


* utmps-0.1.1.0
  

 - New binary: utmps-write, a generic utmp client that can write
user-crafted records to the utmp and/or wtmp databases.


* execline-2.8.2.0
  

 - New -s option to the case binary, enabling fnmatch() (shell)
expression matching instead of regular expression matching.


* s6-rc-0.5.3.0
  -

 - Bundle contents are now read in a "contents.d/" subdirectory, one
file per content, instead of one per line in a "contents" file. In
the same way, service dependencies are now read in a "dependencies.d/"
subdirectory, one file per dependency. Old "contents" and "dependencies"
files are still supported, but deprecated. This change allows better
integration of s6-rc service definitions with package managers.


* s6-linux-init-1.0.7.0
  -

 - New -S option to s6-linux-init-maker, forcing a sync on halt even
in a container.


* s6-networking-2.5.1.0
  -

 - SNI wildcarding is implemented, as well as a workaround for a
bearssl bug causing errors on certificate signatures in certain cases.


 Enjoy,
 Bug-reports welcome.
 And happy holidays to you all!

--
 Laurent



Re: A program that can get exactly the log of a supervised process?

2021-10-25 Thread Laurent Bercot




Why not have the grepper listen on the log file directly? You'll need to have a 
timestamp in the log and know where the log is, but those can be known at the 
time of writing the service script.


 There's no such thing as "the log file". There's the log backendS,
which can be one or more automatically rotated log files in a directory,
status files, network backends, etc.
 The only way to access the log stream in a reliable and generic way
is to read it on the way to the backends. You have no control over
what happens later on.

--
 Laurent



Re: A program that can get exactly the log of a supervised process?

2021-10-25 Thread Laurent Bercot

Well, I do realise the lifespan issue of the loggrep program, which is
why I asked the question in the first place.  But I really never thought
of directly inserting loggrep into the logging chain as a new node;
instead, what I have thought is making loggrep a program "attachable" to
the logger.  That is, the logger is extended with a listener which at
least one client can connect to, and which upon connection tees the log
to the client.  I do not know whether you have similar ideas.


 Well in theory you could have something like skabus-dyntee
( https://skarnet.org/software/skabus/skabus-dyntee.html ) as your
logger, and have your "real" logger run as a skabus-dyntee client.
Then you could add a loggrep as a second skabus-dyntee client, and it
would just disappear when it has finished its work.

 It would be a little unreliable as is, because skabus-dyntee doesn't
care if it has no client at all, so if the real logger dies, it won't
block the log stream until the real logger has restarted, so you may
lose logs. But apart from that, it would work.

 A really reliable solution would be having a skabus-dyntee equivalent
that has one permanent output and blocks as long as that output isn't
there. As you say, it would be a logger extended with a listener.

 Another question is how to piggyback loggrep into the notification
mechanism: if loggrep is tied to the logger and not to the service,
it doesn't have native access to the notification pipe. That means a
specific mechanism is needed to give it cross-service access.

 That's definitely a lot of code and a lot of architectural
convolutions to accommodate what is ultimately a daemon misdesign.
But it's probably the least bad way to do it, so I might think about it
more and add something like that to s6 in the distant future.

--
 Laurent



Re: A program that can get exactly the log of a supervised process?

2021-10-24 Thread Laurent Bercot

Any idea on how the log "teeing" may be done cleanly (and portably
if possible; something akin to `tail -f' seems unsuitable because of
potential log rotation), and perhaps any flaw or redundancy in the
design above?


 The obstacle I have always bumped against when trying to do similar
things is that the teeing program always has to remain there, even
after it has done its job (it has read the readiness line and informed
the supervisor). And so, instead of "service | logger", your data
flow permanently becomes "service | loggrep | logger" before
readiness, and "service | cat | logger" after readiness (the best
loggrep can do is exec into cat, or reimplement its functionality,
once it has read its readiness line).

 That wouldn't be a huge performance problem, especially if "cat" can
do zero-copy data, but it is definitely a reliability problem:
 - loggrep must die when the service dies, so a new loggrep can be run
when the service runs again. So loggrep cannot be run as a separate
supervised service in the same service pipeline. (If loggrep were to
restart independently from service, it would need to check whether
service is ready, and run as 'cat' if it is. This is doable, but more
complex.)
 - That means that either the pipe between service and loggrep cannot
be held, or loggrep must have an additional notification that service
died. This is, again, doable, but more complex.
 - If loggrep isn't supervised, and the pipe isn't being held, then
killing loggrep will incur a broken pipe, which means a service restart
with a lost line of logs, which supervision aims to avoid.

 So basically, either loggrep is a simple tee-like program but you
weaken the supervision properties of the service, or the functionality
needs to be embedded in the supervision architecture, with loggrep
being a consumer for service and a producer for logger (which is
easy with s6-rc but not so much with pure s6) and loggrep always
watching the state of service (which is not so easy with s6-rc, where
you don't know the full path to another service directory).

 In short: practical issues. It's impossible to do that in a clean,
satisfying way.

 And it entirely makes sense that it's so difficult, because the very
idea is to use the *data flow* to inform the *control flow*, and that
is inherently dangerous and not planned for in supervision 
architectures.

Making your control flow depend on your data flow is not a good pattern
at all, and I really wish daemons would stop doing that.

 I'm sorry I don't have a better answer.

--
 Laurent



Re: logging services with shell interaction

2021-10-19 Thread Laurent Bercot

we have a fair number of services which allow (and occasionally require) user 
interaction via a (built-in) shell. All the shell interaction is supposed to be 
logged, in addition to all the messages that are issued spontaneously by the 
process. So we cannot directly use a logger attached to the stdout/stderr of 
the process.


 I don't understand the consequence relationship here.

 - If you control your services / builtin shells, the services could
have an option to log the IO of their shells to stderr, as well as
their own messages.
 - Even if you cannot make the services log the shell IO, you can add
a small data dumper in front of the service's shell, which transmits
full-duplex everything it gets but also writes it to its own stdout or
stderr; if that stdout/err is the same pipe as the stdout/err of your
service, then all the IO from the shell will be logged to the same place
(and log lines won't be mixed unless they're more than PIPE_BUF bytes
long, which shouldn't happen in practice). So with that solution you
could definitely make your services log to multilog.



procServ is a process supervisor adapted to such situations. It allows an 
external process (conserver in our case) to attach to the service's shell via a 
TCP or UNIX domain socket. procServ supports logging everything it sees (input 
and output) to a file or stdout.


 That works too.



IOC=$1

/usr/bin/procServ -f -L- --logstamp --timefmt="$TIMEFMT" \
 -q -n %i --ignore=^D^C^] -P "unix:$RUNDIR/$IOC" -c "$BOOTDIR" "./$STCMD" \
 | /usr/bin/multilog "s$LOGSIZE" "n$LOGNUM" "$LOGDIR/$IOC"
```

So far this seems to do the job, but I have two questions:

1. Is there anything "bad" about this approach? Most supervision tools have 
this sort of thing as a built-in feature and I suspect there may be a reason for that 
other than mere convenience.


 It's not *bad*, it's just not as airtight as supervision suites make
it. The reasons why it's a built-in feature in 
daemontools/runit/s6/others

are:
 - it allows the logger process to be supervised as well
 - it maintains open the pipe to the logger, so service and logger can
be restarted independently at will, without risk of losing logs.

 As is, you can't send signals to multilog (useful if you want to force
a rotation) without knowing its pid. And if multilog dies, it broken
pipes procServ, and it (and your service) is probably forced to restart,
and you lose the data that it wanted to write.
 A supervision architecture with integrated logging protects from this.



2. Do any of the existing process supervision tools support what procServ gives 
us wrt interactive shell access from outside?


 Not that I know of, because that need is pretty specific to your
service architecture.
 However, unless there are more details you have omitted, I still
believe you could obtain the same functionality with a daemontools/etc.
infrastructure and a program recording the IO from/to the shell. Since
you don't seem opposed to using old djb programs, you could probably
even directly reuse "recordio" from ucspi-tcp for this. :)

--
 Laurent



Re: Service watchdog

2021-10-19 Thread Laurent Bercot

Yes, in my usecase this would be used at the place where sd_notify()
is used if the service runs under systemd. Then periodically executed
watchdog could check the service makes progress and react if it
doesn't.

The question is how to implement the watchdog then - it could be either
a global service or another executable in service directory, which
would be started periodically by runsv.


 If a single notification step is enough for you, i.e. the service
goes from a "preparing" state to a "ready" state and remains ready
until the process dies, then what you want is implemented in the s6
process supervisor: https://skarnet.org/software/s6/notifywhenup.html

 Then you can synchronously wait for service readiness
(s6-svwait $service) or, if you have a watchdog service, periodically
poll for readiness (s6-svstat -r $service).

 But that's only valid if your service can only change states once
(from "not ready" to "ready"). If you need anything more complex, s6
won't support it intrinsically.

 The reason why there isn't more advanced support for this in any
supervision suite (save systemd but even there it's pretty minimal)
is that service states other than "not ready yet" and "ready" are
very much service-dependent and it's impossible for a generic process
supervisor to support enough states for every possible existing service.
Daemons that need complex states usually come with their own
monitoring software that handles their specific states, with integrated
health checks etc.

 So my advice would be:
 - if what you need is just readiness notification, switch to s6.
It's very similar to runit and I think you'll find it has other
benefits as well. The drawback, obviously, is that it's not in busybox
and the required effort to switch may not be worth it.
 - if you need anything more complex, you can stick to runit, but you
will kinda need to write your own monitor for your daemon, because
that's what everyone does.

 Depending on the details of the monitoring you need, the monitoring
software can be implemented as another service (e.g. to receive
heartbeats from your daemon), or as a polling client (e.g. to do
periodic health checks). Both approaches are valid.

 Don't hack on runit, especially the control pipe thing. It will not
end well.
 (runit's control pipe feature is super dangerous, because it allows a
service to hijack the control flow of its supervisor, which endangers
the supervisor's safety. That's why s6 does not implement it; it
provides similar - albeit slightly less powerful - control features
via ways that never give the service any power over the supervisor.)

--
 Laurent



Re: Readiness notification using signals

2021-09-29 Thread Laurent Bercot



 Hi Carlos,


I'm supervising an instance of an X Server using s6.

X.Org has a built-in readiness notification mechanism: it sends a USR1
to its parent process once it's ready. From what I know, it would be
s6-supervise.


 This...  this is appalling.

 This mechanism is a terrible way to notify readiness.
 1. It forces the program reading readiness information to be the
direct parent of the daemon.
 2. This is not a concern for X, but if the daemon has dropped
privileges, it is very possible that it does not have the rights to
signal its parent anymore.
 3. It forces the parent to use the siginfo_t structure, and more
complex signal management, in order to detect what process is sending
the signal. This can work, but is by no means a common use of signals.
 4. Signals can be lost if several signals of the same value are
received by a process. So if two processes are notifying readiness at
the same time to a unique supervisor via SIGUSR1, it is possible that
the supervisor only gets one of them.



However, after reading the documentation, there doesn't seem to be a
way to set up custom USR1 handlers for s6-supervise.


 Indeed, there isn't. Signals are meant to be a control mechanism from
above, not from below: the administrator is supposed to be able to send
signals to s6-supervise, but the supervised process *definitely is not*.



 As far as I know,
this leaves me with two, not quite ideal solutions: make the run
script spawn X and do readiness notification on its behalf (which
lengthens the supervision chain), or poll it (which is what I've been
doing). Is there a way to not let the information from this USR1 not
"go to waste"?


 Let this "information" go to waste: it should never be sent in the
first place, and if patching were to happen, it should be to delete
that chunk of code from the Xorg server. We should consider ourselves
lucky that s6-supervise is not using SIGUSR1 for its own purposes and
is happily ignoring it as is.

 Fortunately, there is a different solution for you: the -displayfd 
option

to the X command will print the display number, followed by a newline,
to the (argument of -displayfd) file descriptor, when X has found a
display. This only happens when X is ready - unless it only starts
listening to its socket later, but the window should be super small; and
it has the correct format for an s6 notification.
 So, using displayfd as your notification fd will work.
 For instance, if you have 3 in your notification-fd file, starting
X -displayfd 3 should properly (or *almost* properly) notify readiness.
Some people already use that mechanism, and are happy with it. :)

 Good luck,

--
 Laurent



Re: [announce] skarnet.org Fall 2021 release

2021-09-26 Thread Laurent Bercot

just a minor thing.  You probably didn't push to s6-rc repository.
I don't see the new v0.5.2.3 version commit and tag.


 Weird. I must have missed a case with my script.
 Thanks for the report; it should be fixed now. :)

--
 Laurent



[announce] skarnet.org Fall 2021 release

2021-09-26 Thread Laurent Bercot



 Hello,

 New versions of all the skarnet.org packages are available.

 skalibs has undergone a major update, with a few APIs having 
disappeared,
and others having changed. Compatibility with previous versions is  
*not*

assured.
 Consequently, all the rest of the skarnet.org software has undergone
at least a release bump, in order to build with the new skalibs. But
some packages also have new functionality added (hence, a minor bump),
and others also have their own incompatible changes (hence, a major 
bump).


 The new versions are the following:

skalibs-2.11.0.0  (major)
nsss-0.2.0.0  (major)
utmps-0.1.0.3 (release)
execline-2.8.1.0  (minor)
s6-2.11.0.0   (major)
s6-rc-0.5.2.3 (release)
s6-portable-utils-2.2.3.3 (release)
s6-linux-utils-2.5.1.6(release)
s6-linux-init-1.0.6.4 (release)
s6-dns-2.3.5.2(release)
s6-networking-2.5.0.0 (major)
mdevd-0.1.5.0 (minor)
bcnm-0.0.1.4  (release)
dnsfunnel-0.0.1.2 (release)

Additionally, a new package has been released:
smtpd-starttls-proxy-0.0.1.0

 Dependencies have all been updated to the latest versions. They are,
this time, partially strict: libraries and binaries may build with older
releases of their dependencies, but not across major version bumps. The
safest approach is to upgrade everything at the same time.

 You do not need to recompile your s6-rc service databases or recreate
your s6-linux-init run-images.
 You should restart your supervision tree after upgrading skalibs and 
s6,

as soon as is convenient for you.

 Details of major and minor package changes follow.


* skalibs-2.11.0.0
  

 - A lot of obsolete or useless functionality has been removed:
libbiguint, rc4, md5, iobuffer, skasigaction, environ.h and
getpeereid.h headers, various functions that have not proven their
value in a while.
 - Some functions changed signatures or changed names, or both.
 - All custom types ending in _t have been renamed, to avoid treading on
POSIX  namespace. (The same change has not been done yet in other
packages,  but skalibs was the biggest offender by far.)
 - Signal functions have been deeply reworked.
 - cdb has been reworked, the API is now more user-friendly.
 - New functions have been added.

 The deletion of significant portions of code has made skalibs leaner.
libskarnet.so has dropped under 190 kB on x86_64.
 The cdb rewrite on its own has helped reduce an important amount of
boilerplate in cdb-using code.
 All in all, code linked against the new  skalibs should be slightly
smaller and use a tiny bit less RAM.

 https://skarnet.org/software/skalibs/
 git://git.skarnet.org/skalibs


* nsss-0.2.0.0
  

 - Bugfixes.
 - nsss-switch wire protocol slightly modified, which is enough to
warrant a major version bump.
 - _r functions are now entirely thread-safe.
 - Spawned nsssd programs are now persistent and only expire after a
timeout on non-enumeration queries. This saves a lot of forking with
applications that can call  primitives such as getpwnam() repeatedly, as
e.g. mdevd does when  initially parsing its configuration file.
 - New nsssd-switch program, implementing real nsswitch functionality
by dispatching queries to various backends according to a script.
It does not dlopen a single library or read a single config file.

 https://skarnet.org/software/nsss/
 git://git.skarnet.org/nsss


* execline-2.8.1.0
  

 - Bugfixes.
 - New binary: case. It compares a value against a series of regular
expressions, executing into another command line on the first match.

 https://skarnet.org/software/execline/
 git://git.skarnet.org/execline


* s6-2.11.0.0
  ---

 - Bugfixes.
 - Some libs6 header names have been simplified.
 - s6-svwait now accepts -r and -R options.
 - s6-supervise now reads an optional lock-fd file in the service
directory; if it finds one, the first action of the service is to take
a blocking lock. This prevents confusion when a controller process dies
while still leaving workers holding resources; it also prevents log
spamming on user mistakes (autobackgrounding services, notably).
 - New binaries: s6-socklog, s6-svlink, s6-svunlink. The former is a
rewrite of smarden.org's socklog program, in order to implement a fully
functional syslogd with only s6 programs. The latter are tools that 
start

and stop services by symlinking/unlinking service directories from a
scan directory, in order to make it easier to integrate s6-style 
services

in boot scripts for sequential service managers such as OpenRC.

 https://skarnet.org/software/s6/
 git://git.skarnet.org/s6


* s6-networking-2.5.0.0
  -

 - Bugfixes.
 - minidentd has been removed. It was an old and somehow still buggy
piece of  code that was only hanging around for nostalgia reasons.
 - Full support for client certificates. Details of the client
certificate are transmitted to the application via 

Re: First time caller to the show - am I understanding the fifo trick correctly?

2021-08-25 Thread Laurent Bercot

Forgiving privilege separation failures and minor grammatical mistakes, does it 
look as if I understand the fifo trick's application in practice?


 Hi Ellenor,
 Yes, I think you have the right idea.

 The goal here is to redirect s6-svscan's own stdout and stderr to
the stdin of the catch-all logger process, so that the supervision
tree's messages, and the messages from every service that lacks a
dedicated logger, go to the catch-all logger instead of /dev/console.
(Because /dev/console is a terrible default place to send logs and
should only be used for very critical messages such as kernel panics,
or, in our userland case, for catch-all logger failures.)

 The problem is that we want the catch-all logger to run as a service
under the supervision tree, so the s6-log process does not exist yet
when we exec into s6-svscan: it will be spawned later as a grandchild
of s6-svscan (with an s6-supervise intermediary). So we cannot use an
anonymous pipe for this.

 We use a fifo instead: we can redirect init's stdout and stderr to
a fifo, and later on, when the catch-all logger starts, we can
instruct it (in its run script) to read from the fifo.

 But the Unix fifo semantics say that we *cannot* open a fifo for
writing while there is no reader: open() would either block (default
flags) or return -1 ENXIO (with O_NONBLOCK). So the "fifo trick" is:
1. open the fifo for reading
2. open it for writing, which now works
3. close the reading end

At this point, any write() to the fifo will fail with -1 EPIPE. That is
not a problem per se, except it will also generate a SIGPIPE, so in
order to avoid crashing and burning, it is important to ignore SIGPIPE
at the very least - or, better, to make sure that no process writes to
the fifo until the catch-all logger is up. This is the case for 
s6-svscan

and s6-supervise, so our system structure is safe; but we need to make
sure that no other process starts before the catch-all logger is up,
else they will just eat a SIGPIPE and die.

 In the s6-l-i model, s6-svscan is executed as soon as possible, on a
very minimal supervision tree that only contains the catch-all logger
and a few other essential "early services" (such as the shutdown daemon
and an early getty). All the rest of the initialization is done in
"stage 2 init", which is a script run as a child of s6-l-i's.
So the end of the "fifo trick" uses the Unix fifo semantics as a
synchronization mechanism:
4. fork
5. In the child, close our fd to the fifo
6. In the child, open the fifo for writing once again,
   *without* O_NONBLOCK.

 This last open() will block until the fifo has a reader. That
ensures the child will only resume once the parent has completed
its work and executed into s6-svscan, and the supervision tree has
started and the catch-all logger is running. Then the child can exec
into stage 2 init and perform the rest of the work with the guarantee
that the supervision tree is operational and all the stdout and stderr
messages go to the catch-all logger by default.

 To see exactly how to implement stage 1 init and the fifo trick as
an execline script, you can checkout (or download) any version of
s6-l-i *prior to* 1.0.0.0; try version 0.4.0.1, downloadable from
skarnet.org if you type the URL by hand, and accessible via the
v0.4.0.1 tag in git. It is very different from what it is now, as in
there is no sysv compatibility at all, but stage 1 should be
understandable.

 A few months ago, I tried adding a few conditional compilation options
to s6-l-i to make it work under FreeBSD, but unfortunately the
organization of the FreeBSD init is so different from Linux's,
especially shutdown-wise, that my attempt only succeeded in turning
the package into an unholy plate of spaghetti. At some point in the
future, however, a similar-but-separate s6-freebsd-init package may
make sense.

--
 Laurent



Re: s6-rc-init verbose equivalent messages?

2021-08-19 Thread Laurent Bercot

# /usr/local/bin/s6-rc-init -c /s/comp -l /s/run /s/scan
s6-rc-init: fatal: unable to supervise service directories in
/s/run/servicedirs: No such file or directory
I've completed a disk-disk copy, as I need to integrate s6 into
hardenedbsd.


 Do you have a s6-svscan process running on /s/scan ? You need one
before you can run s6-rc-init.

 If you have one, please get a strace of s6-rc-init, if you can:
that will tell exactly what's happening. But this error generally
points to a supervision tree that's not set up properly.

--
 Laurent



Re: S6 Queries

2021-08-11 Thread Laurent Bercot




Thanks Laurent for the detailed explanations. We did a bootup speed
comparison between S6 and systemd. S6 is able to boot up slightly faster
than systemd. Actual result is 4-4.5% faster but we were expecting
something near to 20%.
Ours is a bit complex setup with more than 140 services (includes a lot of
long run services and a lot of dependencies). The main advantage in systemd
is, it starts many critical processes very quickly since it has no
dependency to logging services. We collect the logs from journalctl and
store it in log files. Whereas in S6, the critical services start up is a
bit delayed since it has to depend on logging services which in turn
depends on other services (responsible for backing up the previous logs).


 Thank you for these numbers! Indeed they confirm my intuition: booting
via s6 is a little faster than systemd, but nothing extraordinary,
because s6-rc emphasizes reliability over speed.

 Your critical services may be slightly delayed, but you have the
guarantee that you will not lose logs - whereas with systemd, if a piece
of your logging system fails to start, your critical services may 
already

have produced some data that will vanish into the aether. Whether or not
that's an acceptable risk is up to you.

--
 Laurent



Re: S6 Queries

2021-08-02 Thread Laurent Bercot

Do you think this is any better?

=
#!/bin/sh
test_for_myrequirement || exit 1
exec mydaemon -myarg1 -myarg2
=


 This does not accomplish the same thing at all: it does not ensure
that myrequirement is at least attempted before mydaemon runs. Instead,
it conditions the readiness of mydaemon to that of myrequirement. So,
it is "better" in the sense that it does not control another service
from a run script, but it is even further from what the OP wants.

 Any reference to another service in a  run script is going to be quirky
at best. Managing all kinds of dependencies between services is really
best done *outside* of run scripts, which is why s6-rc exists. It does
not currently have all the expressive power of systemd for dependencies,
but in the future, it will.

--
 Laurent



Re: S6 Queries

2021-08-02 Thread Laurent Bercot

I thought the way to do what the OP asked is:

=
#!/bin/sh
s6-svc -u myrequirement || exit 1
exec mydaemon -myarg1 -myarg2
=


 This is not a good idea in a s6-rc installation, because it sends
raw s6 commands, which may mess with the service state as viewed by
s6-rc. Also, it sends control commands to a service from another
service's run script, which is bad form in general: it is
unintuitive that starting the mydaemon service also causes the
myrequirement service to be started. Dependencies should be handled at
the service manager level, not at the process supervision level.

 Of course, you can do that in a pinch for small s6-only installations,
where you have no proper dependency engine, but that does not seem to be
what the OP is asking.

 And even then, this does not implement After= because s6-svc -u
returns instantly. This only implements Wants=. To implement After=,
you would need something like s6-svc -uwU instead, which is not good
because it adds the myrequirement readiness delay to the mydaemon
readiness delay, so if mydaemon has strict readiness timeouts, it
can make it fail.

 All in all, it's better to avoid controlling another service in a run
script: there's always an annoying corner case somewhere. Dependency
management is best handled by the external tool that is explicitly
supposed to do it: the service manager.

--
 Laurent



Re: S6 Queries

2021-08-02 Thread Laurent Bercot

1. In systemd, the services are grouped as targets and each target depends
on another target as well. They start as targets. [ex: Reached
local-fs.target, Reached network.target, Reached UI target,...]. Is there
any way in S6 to start the init system based on bundles?


 Yes, that is what bundles are for. In your stage 2 boot script
(typically /etc/s6-linux-init/current/scripts/rc.init), you should
invoke s6-rc as:
  s6-rc change top
if "top" is the name of your default bundle, i.e. the bundle that
contains all the services you want to start at boot time. You can
basically convert the contents of your systemd targets directly into
contents of your s6-rc bundles; and you decide which one will be
brought up at boot time via the s6-rc invocation in your stage 2
init script.



2. Are there any ways to have loosely coupling dependencies? In systemd, we
have After=. After option will help the current service to start after the
mentioned service (in after). And the current service will anyway start
even if the mentioned service in After fails to start. Do we have such
loosely coupled dependency facility in S6?


 Not at the moment, no. The next version of s6-rc will allow more types
of dependencies, with clearer semantics than the systemd ones (After=, 
Requires= and Wants= are not orthogonal, which is unintuitive and causes

misuse); but it is still in early development.

 For now, s6-rc only provides one type of dependency, which is the
equivalent of Requires+After. I realize this is not flexible enough
for a lot of real use cases, which is one of the reasons why another
version is in development. :)



3. Is there any tool available in S6 to measure the time taken by each
service to start? We can manually measure it from the logs, but still
looking for a tool which can provide accurate data.


 Honestly, if you use the -v2 option to your s6-rc invocation, as in
  s6-rc -v2 change top
and you ask the catch-all logger to timestamp its lines (which should
be the default, but you can change the timestamp style via the -t
option to s6-linux-init-maker)
then the difference of timestamps between the lines:
  s6-rc: info: service foo: starting
and
  s6-rc: info: service foo successfully started
will give you a pretty accurate measurement of the time it took service
foo to start. These lines are written by s6-rc exactly as the
"starting" or "completed" event occurs, and they are timestamped by
s6-log immediately; the code path is the same for both events, so the
delays cancel out, and the only inaccuracy left is randomness due to
scheduling, which should not be statistically significant.

 At the moment, the s6-rc log is the easiest place to get this data
from. You could probably hack something with the "time" shell command
and s6-svwait, such as
  s6-svwait -u /run/service/foo ; time s6-svwait -U /run/service/foo
which would give you the time it took for foo to become ready; but
I doubt it would be any more accurate than using the timestamps in the
s6-rc logs, and it's really not convenient to set up.



4. Does the S6 init system provide better boot up performance compared to
systemd ?  One of our main motives is to attain better bootup performance.
Is our expectation correct?


 The boot up performance should be more or less *similar* to systemd.
The code paths used by the s6 ecosystem are much shorter than the ones
used by systemd, so _in theory_ you should get faster boots with s6.

 However, systemd cheats, by starting services before their dependencies
are ready. For instance, it will start services before any kind of
logging is ready, which is pretty dangerous for several reasons. As a
part of its "socket activation" thing, it will also claim readiness
on some sockets before even attempting to run the actual serving
processes (which may fail, in which case readiness was a lie.)
Because of that, when everything goes well, systemd cuts corners that
s6 does not, and may gain some advantage.

 So all in all, I expect that depending on your system, the difference
in speed will not be remarkable. On very simple setups (just a few
services), systemd's overhead may be noticeable and you may see real
improvements with s6. On complex setups with lots of dependencies, s6
might still have the speed advantage but I don't think it will be
anything amazing. The real benefit of s6 is that it achieves
roughly the same speed as systemd *while being more reliable and
predictable*.

 If you actually make bootup speed comparisons between systemd and s6,
please share them! I am interested in that kind of benchmarks, and
I'm sure the community would like to see numbers as well.

--
 Laurent



Re: Query on S6 system shutdown

2021-07-29 Thread Laurent Bercot

I believe the finish script is not being called by s6-svc. When I run it
manually , the finish script runs and kills the process and graceful
shutdown is happening as expected.

What may be the cause for not triggering the finish script of critical
service.


 The finish script, which is entirely optional, is not supposed to
kill the process. It is not called by s6-svc, and does not run when
the process is still alive. It is *not* equivalent to the "down"
script of a oneshot. (There is a subtle hint in the fact that the
scripts are named "run" and "finish" instead of "up" and "down".)

 If it exists, the finish script is run by s6-supervise *after* the
service dies. The point is to perform any potentially needed cleanups
that are not automatically performed at service death.

 Your current finish script is entirely incorrect and you should
delete it.

 In order to kill your service, you need to send it a signal. Daemons
normally have a signal that tells them to gracefully exit; for most
daemons, it is SIGTERM, which is what svc -d sends by default.

 If your critical.service does not die on s6-svc -d, it means that
it ignores SIGTERM. Then you need to find out *what* signal it
interprets as a request for a graceful shutdown, and put the name
of this signal in the "down-signal" file in your service definition
directory. Then, when you send s6-svc -d, critical.service will
receive the signal you indicated.

 down-signal can be SIGKILL - this is what you're using at the moment -
but if your service is that critical, chances are it's not the best way
to kill it and you should look for a gentler signal to send it first.
If there are cases where a graceful exit request is ignored or takes
too long, and you want to make sure the service dies at some point,
you can create a "timeout-kill" file, containing a number of 
milliseconds

- if the process is still alive after that duration after receiving
its down-signal, it will be sent a SIGKILL.

--
 Laurent



Re: Query on s6-log and s6-supervise

2021-06-09 Thread Laurent Bercot

I have checked the Private_Dirty memory in "smaps" of a s6-supervise
process and I don't see any consuming above 8kB. Just posting it here
for reference.


 Indeed, each mapping is small, but you have *a lot* of them. The
sum of all the Private_Dirty in your mappings, that should be shown
in smaps_rollup, is 96 kB. 24 pages! That is _huge_.

 In this list, the mappings that are really used by s6-supervise (i.e.
the incompressible amount of unshareable memory) are the following:

 - the /bin/s6-supervise section: this is static data, s6-supervise
needs a little, but it should not take more than one page.

 - the [heap] section: this is dynamically allocated memory, and for
s6-supervise it should not be bigger than 4 kB. s6-supervise does not
allocate dynamic memory itself, the presence of a heap section is due
to opendir() which needs dynamic buffers; the size of the buffer is
determined by the libc, and anything more than one page is wasteful.

( - anonymous mappings are really memory dynamically allocated for
internal  libc purposes; they do not show up in [heap] because they're
not obtained via malloc(). No function used by s6-supervise should
ever need those; any anonymous mapping you see is libc shenanigans
and counts as overhead. )

 - the [stack] section: this is difficult to control because the
amount of stack a process uses depends a lot on the compiler, the
compilation flags, etc. When built with -O2, s6-supervise should not
use more than 2-3 pages of stack. This includes a one-page buffer to
read from notification-fd; I can probably reduce the size of this
buffer and make sure the amount of needed stack pages never goes
above 2.

 So in total, the incompressible amount of private mappings is 4 to 5
pages (16 to 20 kB). All the other mappings are libc overhead.

 - the libpthread-2.31.so mapping uses 8 kB
 - the librt-2.31.so mapping uses 8 kB
 - the libc-2.31.so mapping uses 16 kB
 - the libskarnet.so mapping uses 12 kB
 - ld.so, the dynamic linker itself, uses 16 kB
 - there are 16 kB of anonymous mappings

 This is some serious waste; unfortunately, it's pretty much to be
expected from glibc, which suffers from decades of misdesign and
tunnel vision especially where dynamic linking is concerned. We are,
unfortunately, experiencing the consequences of technical debt.

 Linking against the static version of skalibs (--enable-allstatic)
should save you at least 12 kB (and probably 16) per instance of
s6-supervise. You should have noticed the improvement; your amount of
private memory should have dropped by at least 1.5MB when you switched
to --enable-allstatic.
 But I understand it is not enough.

 Unfortunately, once you have removed the libskarnet.so mappings,
it's basically down to the libc, and to achieve further improvements
I have no other suggestions than to change libcs.


If possible, can you please share us a reference smap and ps_mem data on
s6-supervise. That would really help.


 I don't use ps_mem, but here are the details of a s6-supervise process
on the skarnet.org server. s6 is linked statically against the musl
libc, which means:
 - the text segments are bigger (drawback of static linking)
 - there are fewer mappings (advantage of static linking, but even when
you're linking dynamically against musl it maps as little as it can)
 - the mappings have little libc overhead (advantage of musl)

# cat smaps_rollup

0040-7ffd53096000 ---p  00:00 0  [rollup]
Rss:  64 kB
Pss:  36 kB
Pss_Anon: 20 kB
Pss_File: 16 kB
Pss_Shmem: 0 kB
Shared_Clean: 40 kB
Shared_Dirty:  0 kB
Private_Clean: 8 kB
Private_Dirty:16 kB
Referenced:   64 kB
Anonymous:20 kB
LazyFree:  0 kB
AnonHugePages: 0 kB
ShmemPmdMapped:0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb:0 kB
Private_Hugetlb:   0 kB
Swap:  0 kB
SwapPss:   0 kB
Locked:0 kB

 You can see 40kB of shared, 16kB of Private_Dirty, and 8kB of
Private_Clean - apparently there's one Private_Clean page of static
data and one of stack; I have no idea what this corresponds to in the
code, I will need to investigate and see if it can be trimmed down.

# grep -E '[[:space:]](-|r)(-|w)(-|x)(-|p)[[:space:]]|^Private_Dirty:' 
smaps


0040-00409000 r-xp  ca:00 659178  /command/s6-supervise
Private_Dirty: 0 kB
00609000-0060b000 rw-p 9000 ca:00 659178  /command/s6-supervise
Private_Dirty: 4 kB
02462000-02463000 ---p  00:00 0  [heap]
Private_Dirty: 0 kB
02463000-02464000 rw-p  00:00 0  [heap]
Private_Dirty: 4 kB
7ffd53036000-7ffd53057000 rw-p  00:00 0  [stack]
Private_Dirty: 8 kB
7ffd5309-7ffd53094000 r--p  00:00 0  [vvar]
Private_Dirty: 0 kB
7ffd53094000-7ffd53096000 r-xp  00:00 0  [vdso]
Private_Dirty: 0 kB

 One page of static data, one page 

Re: Suspend s6-log writing to log file and resume it back after sometime

2021-06-08 Thread Laurent Bercot

Any pointers on how I can go about this? Is there any hack or tricks that could 
be done in s6-log to achieve this?


 Sorry, but no, nothing comes to mind - s6-log was not designed for 
this.


 I don't think expecting services to keep running while not logging to
disk, whether or not in standby/sleep mode, is reasonable: if logs keep
coming up, memory fills up. What do you do if the machine doesn't wake
up before the memory is full? The logger will die and you will lose all
your carefully accumulated logs.

 Ideally, you would have dynamic verbosity in the service, and switch
it to zero when going into standby/sleep mode, so it would stop
producing logs, so you'd never wake up the disk. Of course, unless it's
the wakeup event listener, the concept of still having a service running
when in standby mode is weird: it defeats the very purpose of standby
mode, which is saving energy. The best way to not have your disk spin is
to have nothing to make it spin in the first place. :P

 s6-svc -p all your services when entering standby mode! (Except the
wakeup event listener.) :D

 Sorry for being unhelpful, and good luck,

--
 Laurent



Re: Query on s6-log and s6-supervise

2021-06-08 Thread Laurent Bercot

   1. Why do we need to have separate supervisors for producer and consumer
   long run services? Is it possible to have one supervisor for both producer
   and consumer, because anyhow the consumer service need not to run when the
   producer is down.  I can understand that s6 supervisor is meant to monitor
   only one service, but why not monitor a couple of services when it is
   logically valid if I am not wrong.


 Hi Arjun,

 The logic of the supervisor is already complex enough when it has
to monitor one process. It would be quadratically as complex if it
had to monitor two. In all likeliness, the first impact of such a
change would be more bugs, because the logic would be a lot more
difficult to understand and maintain.

 The amount of memory used by the s6 logic itself would not change
(or would *increase* somewhat) if the code was organized in a
different way in order to reduce the amount of processes, and you
would see an overall decrease in code quality.

 Worsening the design to offset operational costs is not a good
trade-off - it is not "logically valid", as you put it. I would not
do it even if the high amount of memory consumed by your processes
was due to s6 itself.

 But it is not the case: your operational costs are due to something
else. See below.




   2. Is it possible to have a single supervisor for a bundle of services?
   Like, one supervisor for a bundle (consisting of few services)?


 Again, there would be no engineering benefit to that. You would likely
see operational benefits, yes, but s6 is the wrong place to try and get
those benefits, because it is not the cause of your operational costs.



   3. Generally how many instances of s6-supervise can run? We are running
   into a problem where we have 129 instances of s6-supervise that leads to
   higher memory consumption. We are migrating from systemd to s6 init system
   considering the light weight, but we have a lot of s6-log and s6-supervise
   instances that results in higher memory usage compared to systemd.  Is it
   fine to have this many number of s6-supervise instances?ps_mem data -
  5.5 MiB   s6-log (46) ,  14.3 MiB   s6-supervise (129)


 It is normally totally fine to have this many number of s6-supervise
instances (and of s6-log instances), and it is the intended usage.
The skarnet.org server only has 256 MB of RAM, and currently sports 93
instances of s6-supervise (and 44 instances of s6-log) without any
trouble. It could triple that amount without breaking a sweat.

 The real problem here is that your instances appear to use so much
memory: *that* is not normal.
Every s6-supervise process should use at most 4 pages (16k) of private
dirty memory, so for 129 processes I would expect the memory usage to
be around 2.1 MB. Your reported total shows 7 times as much, which
sounds totally out of bounds to me, and even accounting for normal
operational overhead, a factor of 7 is *completely bonkers*.

 There are two possible explanations here:
 - Either ps_mem is not accurately tallying the memory used by a given
set of processes;
 - Or you are using a libc with an incredible amount of overhead, and
your libc (and in particular, I suspect, dynamic linking management in
your libc) is the culprit for the insane amount of memory that the
s6-supervise processes seem to be eating.

 The easiest way to understand what's going on is to find a
s6-supervise process's pid, and to perform
# cat /proc/$pid/smaps_rollup

 That will tell you what's going on for the chosen s6-supervise process
(they're all similar, so the number for the other s6-supervise processes
won't be far off). In particular, look at the Private_Dirty line: that
is the "real" amount of uncompressible memory used by that process.
 It should be around 16k, tops. Anything over that is overhead from
your libc.
 If the value is not too much over 16k, then ps_mem is simply lying to
you and there is nothing to worry about, except that you should use
another tool to tally memory usage.
 But if the value is much higher, then it is time to diagnose deeper:

# cat /proc/$pid/smaps

 That will show you all the mappings performed by your libc, and
the amount of memory that each of these mappings uses. Again, the
most important lines are the Private_Dirty ones - these are the
values that add up for every s6-supervise instance.

 My hunch is that you will see *a lot* of mappings, each using
4k or 8k, or even in some cases 12k, of Private_Dirty memory.
If it is the case, unfortunately there is nothing I can do about it,
because that overhead is entirely caused by your libc.

 However, there is something *you* can do about it:

 - If "ldd /bin/s6-supervise" gives you a line mentioning libs6.so
or libskarnet.so, try recompiling s6 with --enable-allstatic. This
will link against the static version of libs6 and libskarnet, which
will alleviate the costs of dynamic linking. (The price is that the
*text* of s6-supervise will be a little bigger, but it doesn't 

Re: [PATCH] s6-rc-compile: Fix setting of flag-essential

2021-06-03 Thread Laurent Bercot




Don't set the bitposition (which is 0 for 'flag-essential')
to the flags, but the bit at the position.


 Ha, nice catch. Applied, thanks!

--
 Laurent



[announce] #s6 is moving from Freenode to OFTC

2021-05-24 Thread Laurent Bercot



 Hello,
 As some of you are aware of, last week, the Freenode IRC network was
subjected to a forceful change of its operational control. The history
and details of the change are deeply political in nature and very much
off-topic for our lists, and I really do not wish this event to be
discussed here; there is plenty of information and discussion to be had
in other corners of the Internet.
 The fact remains, however, that Freenode is no longer a safe IRC
network for free and open source software communities.
 As of now, the #s6 channel (and its associated off-topic channel) has
moved to the OFTC network: https://www.oftc.net/
 The skarnet.org documentation has been updated accordingly.
 Please update your pointers, and if you're currently hanging on
Freenode, please come to OFTC instead.
 The #s6 channel on Freenode will cease to be maintained, except for
a /topic  pointing to the new hosting network. At some point, it will
be moderated, i.e. you will not be able to write in it anymore.
 In the same way, there will be an empty #s6 channel on the libera.chat
network, pointing to the OFTC network as well.
 I know that some of you would have preferred to move to libera.chat,
the "official" successor of freenode, instead of OFTC. It was not an
easy choice to make, and no matter the decision, some people would
have ended up unhappy anyway. Please fight the undoubtedly overwhelming
urge to explain why the decision was wrong and the other one would have
been better; instead, feel free to address all complaints to Andrew Lee,
Crown Prince of Korea and originator of all this shitshow.
 Thank you. I hope to meet you on the other side.
--
 Laurent



[announce] New hypermail archives of skarnet.org mailing lists

2021-05-09 Thread Laurent Bercot



 Hello,

 ezmlm-cgi, the web interface to the archives of the skarnet.org
mailing-lists, has been broken for... forever, resulting in an
inability to display certain messages. I tried debugging it, but
could not find out what was happening within a reasonable amount
of time dedicated to it.

 A new web interface to these archives is now available, and this
one appears to work better (there are still a few quirks with
utf-8 display but we'll sort them out).

 https://skarnet.org/lists/skaware/
 https://skarnet.org/lists/supervision/

 If you are using web links to ML archived messages, please update
your links to point to the hypermail archive instead of the ezmlm
one. At some point in the (distant) future, the ezmlm archive will be
deprecated. Thanks.

 Enjoy.

--
 Laurent



Re: S6 logging

2021-03-09 Thread Laurent Bercot

I am trying to log the prints coming from daemons like dropbear using
s6-log, but couldn't make it.
Am I missing something?


 You really need to improve on your way of asking for help.

"couldn't make it" is not an actionable report. You need to say:
- exactly what you did (you did that, good, but you did not mention
that you were using s6-rc, which may or may not be relevant)
- exactly what you expected (you didn't do that, but it is easy to
figure out)
- exactly what happened (you didn't do that at all and we're left
to guess).

 Going out on a limb, and reading the manual page (I haven't used
dropbear in a little while), I'm guessing that the problem is this:



/usr/sbin/dropbear -F


 You also need the -E option in order to tell dropbear to send its
error messages to stderr instead of syslog.

 But if it still doesn't work with -F -E, then we'll need more precise
information.

--
 Laurent



Re: Path monitoring support in s6-services

2021-02-17 Thread Laurent Bercot

Since it requires individual instances of inotifyd for each service(s)
[which depends on  multiple files/paths modifications) to get started]


 Have you tried it and noticed an impact on your boot times?
(AKA: "Profile, don't speculate.")

--
 Laurent



Re: Path monitoring support in s6-services

2021-02-17 Thread Laurent Bercot

 inotifyd (or something similar) + s6-svc (or s6-rc)?

Thought of the same but I have many such services;Just thinking of cpu
overhead during the initial boot up.


 What makes you think this would have a noticeable impact on your CPU
load?

--
 Laurent



Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-02-15 Thread Laurent Bercot

(Apologies for the broken threading, I originally sent my answer with
the incorrect From: and it was rightfully rejected.)



Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-02-15 Thread Laurent Bercot




I do not really understand their excuse here.  CLI incompatibility is
trivially solvable by creating links (or so) for `halt' / `poweroff' /
`reboot', and even the `shutdown' command can be a wrapper for an `atd'
based mechanism.


 The options! The options need to be all compatible. :) And for
"shutdown", they would never implement a wrapper themselves, I would
have to do it for them - which is exactly what I did, although it's
a C program that actually implements shutdown, not a wrapper around an
atd program I can't assume will be present on the system.

 I'm not defending distros here, but it *is* true that a drop-in
replacement, in general, is a lot easier to deal with than a drop-in-
most-of-the-time-maybe-but-not-with-that-option replacement. Anyone
who has tried to replace GNU coreutils with busybox can relate.



  In case they complain about the implementation of the
CLI, the actual interface to `shutdownd' is not that similar to the
`telinit' interface (at least to the one I think it is) either.


 Which is why s6-l-i also comes with a runleveld service, for people
who need the telinit interface. shutdownd is only for the actual
stages 3 and 4, not service management (which telinit is a now obsolete
forerunner of).



If I understand it correctly, letting `s6-svscan' exec() stage 3 also
achieves immunity to `kill -KILL -1'.  I also find this "old-fashioned"
approach conceptually and implementationally simpler than an army of
`s6-supervise' restarting only to be killed again


 What army? By the time the final kill happens, the service manager
has brought everything down, and shutdownd has cleaned up the scandir,
only leaving it with what *should* be restarted. You seem to think
I haven't given these basic things the two minutes of attention they
deserve.

 Conceptually, the "old-fashioned" approach may be simpler, yes.
Implementationally, I disagree that it is, and I'll give you a very
simple example to illustrate it, but it's not the only thing that
implementations must pay attention to, there are a few other quirks
that I've stumbled upon and that disappear when s6-svscan remains
pid 1 until the very end.

 You're going to kill every process. The zombies need to be reapt,
else you won't be able to unmount the filesystems. So your pid 1
needs to be able to wait for children it doesn't know it has
(foreground does not) and guarantee that it doesn't try unmounting
the filesystems before having reapt everything (a shell does not give
ordering guarantees when it gets a SIGCHLD, even though it works in
practice). So for this specific use I had to add a special case to
execline's wait command, "wait { }", that waits on *everything*, and
also make sure that wait doesn't die because it's going to run as pid 1,
even very briefly.
 And after that, you need to make sure to unmount filesystems
immediately, because if you spawn other processes, you would first have
to wait on them as well.

 For every process that may run as pid 1, you need extra special care.
Using an interpreter program as pid 1 means your interpreter needs to
have been designed for it. Using execline means every execline binary
that may run as pid 1 needs to be vetted for it. If your shutdown
sequence is e.g. written in Lisp, and your Lisp interpreter handles
pid 1 duties correctly, okay, that's fair, but that's *two* programs
that need to do it, when one would be enough.
 s6-svscan has already been designed for that and provides all the
guarantees you need. When s6-svscan is running as pid 1, it takes away
a lot of mental burden off the shutdown sequence.



 and a `shutdownd'
restarting to execute the halting procedure (see some kind of "state"
here?  Functional programmers do not hate it for nothing).


 Yes, there is one bit of state involved. I think our feeble human minds,
and a fortiori computers, can handle one bit of state.



  I know this
seems less recoverable than the `shutdownd' approach, but does that
count as a reason strong enough to warrant the latter approach, if the
halting procedure has already been distilled to its bare essentials
and is virtually immune to all non-fatal problems (that is, excluding
something as severe as the absence of a `reboot -f' implementation)?


 My point is that making the halting procedure virtually immune to all
non-fatal problems is *more difficult* when you tear down the
supervision tree early. I am more confident in the shutdownd approach,
because it is less fragile, more forgiving. If there's a bug in it, it
will be easy to fix.

 I understand that the barebones approach is intellectually more
satisfying - it's more minimalistic, more symmetrical, etc. But shutting
down a machine is *not* symmetrical to booting it. When you boot, you
start with nothing and need a precise sequence of instructions in order
to build up to a functional system. When you shutdown, you have a fully
functional system already, that has proven to be working, and you just
need to clean up and make sure you 

Re: [s6-svperms] Handling service permissions at creation time.

2021-02-15 Thread Laurent Bercot

Services can fix their own permissions so if s6-rc is going to grow that
functionality it should be in the generated run, not in some rarely used
outboard helper service.


 As answered on IRC, for ML completeness: no, because permissions should
be fixed when the supervisor starts, not when the service starts. So a
oneshot that runs right after the supervisors are started is the
correct solution.

--
 Laurent



Re: [s6-svperms] Handling service permissions at creation time.

2021-02-15 Thread Laurent Bercot

The s6-svperms is a great feature but it only handle permissions control of a 
service at runtime. That means that we need to change the permissions of the 
service everytime that a reboot occurs.
For a server, this is not really a big deal but for a desktop machine this can 
be really hard to handle as far as the runtime services can be different at 
each boot (user can activate or disactivate service for his purpose).


 Right. The problem here is that the files holding the permissions all
exist in a tmpfs (typically they're all under /run), and are recreated
at every boot, with the default attributes.

 If you run a supervision tree on a non-tmpfs, then the attributes will
be stored on disk, and kept from one boot to the next.

 For a s6-linux-init + s6 + s6-rc installation, the service directories
are always on a tmpfs, so yes, the problem will manifest.



Obviously, a script launched at some point of the boot (or after) can change 
the permissions on the necessary services. However, i think this is not easier 
and not flexible.


 I disagree, I think it's the right way to address it; see below.



S6-supervise create the control, status and event directory with the uid:gid of 
the owner of the process (correct me if i'm wrong).


 That's correct - and the owner of the s6-supervise process is the owner
of the whole supervision tree.



So, If we have a e.g /data/perms/rules/uid//allow file and if 
s6-supervise check this directory at the creation time and create the necessary 
file/directory with the respective uid/gid found at that directory, we can configure a 
service permissions permanently.


 The problem with this approach is the following:

 - The whole service directory is stored in RAM, so you cannot store
svperms attributes anywhere under the service directory - else you'll
have the exact same problem as you do now, the attributes will not
survive a reboot. :)

 - s6-supervise does not and will not look at anything outside of a
service directory. The service directory is the place for everything
related to s6-supervise. (And even then, s6-supervise stays out of
data/ and env/.) So if you need configuration that cannot be stored
in a service directory because the service directory is all in tmpfs,
s6-supervise is not the program that can handle that configuration.

 So, the best way to apply attributes to a set of service directories
is to have another process do it once the service directories have
been copied, because only an external process will be able to access
information that is stored on disk.

 Typically, if you're using s6-rc, this can be done via a s6-rc
service running early, before the longruns are started. The "up"
script can read attributes from a file and set them; the "down"
script can save all the attributes to a file.

 Ideally, though, the user would be able to declare the attributes
in service definition directories, and s6-rc would set them
automatically at start. That wouldn't help with early services, but
early services should be few and far between and their permissions
shouldn't be trifled with.

 I can add that functionality to the next version of s6-rc. What do
you think?

--
 Laurent



Re: timer support for s6 services

2021-02-10 Thread Laurent Bercot

Is there a way to set a timer option on a particular service 'X', so that
'X' gets restarted for every timer seconds ?


 You can achieve that with another service that just sleeps for
'timer' seconds then sends a s6-svc -r command to the service you want
restarted.

--
 Laurent



Re: stage2 as a service

2021-02-01 Thread Laurent Bercot

For the normal case you are absolutly right. But with stage 2 as a service
you have a race condition between stage 2 and s6-svscan-log. The usual
trick for stage 2 solves this problem.


 Ah, now I get it: stage 2 must not start before the catch-all logger
is ready, so you open the fifo for writing with the intent to block
until the reader has started. Yes, that makes sense and is needed.

 However, I will list it as another drawback of your approach :P
The internal sauce is visible in your stage 2 script. Ideally, stage 2
would only run once everything is already in place and wouldn't have
to bother with any mechanism details. (s6-linux-init achieves this.)



Well, running once is a part of supervise from the start on, by djb. It's
invented for oneshots.


 I disagree. svc -o was made for short-lived processes that you may want
to run again. Or for testing a crashing daemon without having to deal
with automatic restarts and failure loops. Anything that you
potentially run more than once. It's "run once", but still in the
context of supervision; you would only use it on services that have
at least the potential to make use of supervision.

 For things that you will only ever run once, why even supervise them in
the first place? Supervision is a wonderful tool, but like any tool, it
should only be used when appropriate, and I don't think the one-time
initialization script is the place for it.



Well, s6-rc is using ./down, too. The shutdown is a very special case for
supervision.


 There is an important difference.
 s6-rc is using ./down files for services that it wants down,
independently from the machine's lifetime. Typically it's using them
at boot time, in order to have the supervisors start, but not the
services themselves (they'll be brought up later on according to the
dependency graph). s6-rc *wants* the services to be supervised, even
if it's not starting them at the same time as the supervisors.

 You're only using a ./down file at shutdown time because you have a
service that you know must not restart when the supervisor is killed,
and will never restart, and has not been restarted for the entire
machine lifetime. The presence of the supervisor here is not a feature,
it brings you no value, on the contrary - it's only making your life
more difficult.



Stage 2 as a service allows us to restart it, if - accidentally - it is
necessary. Obviously, that should be really seldom the case.


 Honestly, I can't think of a single case where you'd need to restart
the initialization sequence of your machine. Anything you'd want to
restart once the system has booted should be handled by the service
manager.



I'm still migrating from systemd to s6{,-rc} with /fs/* step by step.
Therfore, I need more flexibility than s6-linux-init.


 The migration from systemd's service manager (to s6-rc or anything
else) is totally independent from the init system change. You can
make a systemd oneshot that launches s6-rc-init then s6-rc, and
convert all your systemd services one by one to s6-rc services; then,
once you don't depend on systemd for anything else than the early
boot and the s6-rc service, you can switch inits, and then you should
be able to use s6-linux-init.

 I generally recommend doing the opposite: switching to s6-linux-init
first then converting services to s6-rc, because the latter is a lot
more work: for instance Adélie uses s6-linux-init but still has
OpenRC as its service manager, because I haven't done the conversion
work yet. However, it's different with systemd, because systemd cannot
be run as not-pid-1 - its service manager cannot be separated from its
early boot functionality. So you have to keep it as init until it's
not needed for anything else. They call it modular software ;)

--
 Laurent



Re: stage2 as a service [was: Some suggestions on old-fashioned usage with s6 2.10.x]

2021-01-31 Thread Laurent Bercot


 Hi Stefan,
 Long time no see!

 A few comments:



# optional:  -- Question: Is this necessary?
  redirfd -w 0 ${SCANDIR}/service/s6-svscan-log/fifo
  # now the catch all logger runs
  fdclose 0


 I'm not sure what you're trying to do here. The catch-all logger
should be automatically unblocked when
${SCANDIR}/service/s6-svscan-log/run starts.
 The fifo trick should not be visible at all in stage 2: by the time
stage 2 is running, everything is clean and no trickery should take
place. The point of the fifo trick is to make the supervision tree
log to a service that is part of the same supervision tree; but once
the tree has started, no sleight of hand is required.



foreground { s6-svc -O . } # don't restart me


 If you have to do this, it is the first sign that you're abusing
the supervision pattern; see below.



foreground { s6-rc -l ${LIVEDIR}/live -t 1 change ${RCDEFAULT} }
# notify s6-supervise:
fdmove 1 3
foreground { echo "s6-rc ready, stage 2 is up." }
fdclose 1  # -- Question: Is this necessary?


 It's not strictly necessary to close the fd after notifying readiness,
but it's a good idea nonetheless since the fd is unusable afterwards.
However, readiness notification is only useful when your service is
actually providing a... service once it's ready; here, your "service"
dies immediately, and is not restarted.
 That's because it's really a oneshot that you're treating as a
longrun, which is abusing the pattern.



# NB: shutdown should create ./down here, to avoid race conditions


 And here is the final proof: in order to make your architecture work,
you have to *fight* supervision features, because they are getting in
your way instead of helping you.
 This shows that it's really not a good idea to run stage 2 as a
supervised service. Stage 2 is really a one-time initialization script
that should be run after the supervision tree is started, but *not*
supervised.



  { # fallback login
sulogin --force -t 600 # timeout 600 seconds, i.e. 10 minutes.
# kernel panic
  }


 Your need for sulogin here comes from the fact that you're doing quite
complex operations in stage 1: a user-defined set of hooks, then
several filesystem mounts, then another user-defined set of hooks.
And even then, you're running those in foreground blocks, so you're
not catching the errors; the only time your fallback activates is if
the cp -a from ${REPO} fails. Was that intended?

 In any case, that's a lot of error-prone work that could be done in
stage 2 instead. If you keep stage 1 as barebones as possible (and
only mount one single writable filesystem for the service directories)
you should be able to do away with sulogin entirely. sulogin is a
horrible hack that was only written because sysvinit is complex enough
that it needs a special debugging tool if something breaks in the
middle.
 With an s6-based init, it's not the case. Ideally, any failure that
happens before your early getty is running can only be serious enough
that you have to init=/bin/sh anyway. And for everything else, you have
your early getty. No need for special tools.



Also I may switch to s6-linux-init finally.


 It should definitely spare you a lot of work. That's what it's for :)

--
 Laurent


Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-01-29 Thread Laurent Bercot

But even `s6-reboot' from older s6-linux-init, or `busybox reboot'
with slew can already do that...


 Yes. And as your sharp mind undoubtedly noticed, those commands are
not the same as "reboot".

 Which means burden on users.

 Yes, I also thought it was a small burden at first, but it's not.
It means that all sysvinit-compatible automation does not work, so
there is some porting work to do. And the gap between "a little work"
and "zero work" is HUGE. It's much bigger than the gap between
"a little work" and "a lot of work".

 Bear in mind that my eventual goal for s6 is distro adoption. And
distro maintainers will find any and every excuse to reject it.
Having a "shutdown" command that works exactly like sysvinit's
shutdown is essential, because it deals with a major objection, which
is incompatibility and user-unfriendliness.



There is some non-trivial trade-off: in short, the existence of the
supervision tree after stage 2 is by itself a kind of "special case"
(eg. search for "careful handling" in [1]).


 I feel like you misinterpreted my meaning.
 The *absence* of a supervision tree after stage 2 is precisely what
requires careful handling, and runit only works because Linux has
that peculiarity that kill -9 -1 does not kill the emitter!
 Having a supervision tree in stage 3 actually *helps* with the
late shutdown procedure: shutdownd dies right after the kill (which
would make it usable even on a system without the Linux specialcase)
and is restarted by the supervisor for stage 4.



  I am also thinking about
an application scenario, where a supervision tree with a new s6 version
replaces the active tree with an old version.  This is somewhat silly:
it can be a little useful in case of major version bump, but is probably
better solved by complete reboot to completely get rid of all old things
(s6 or not, updated together) in the memory.


 Yes, upgrading your init without rebooting is generally not worth
it. Note that s6-svscan could still be configured to do that with
clever use of SIG scripts; but restarting the s6-supervise processes
is a pain to do without restarting your whole supervision tree, so
it's probably better to just reboot.
 This is the case with every single init out there, so you can't paint
that as a drawback of s6. You can wish it were easier, and I agree
that it would be nice, but the necessary trade-offs to make rebootless
init upgrades viable are very much not worth it.



 all-in-all has just less of a "screwdriver and duct tape" feel than
 a bunch of execline (or rc ;)) scripts.

I am very sorry, but I do feel a strong smell of systemd mindset here :(


 A systemd mindset in an attempt to be a drop-in replacement for
sysvinit. Yeah, right.

 More seriously, you're being unfair, because you're not locked in
at all. You can use the new s6-linux-init and *still* do everything
you were doing before:
 - you can manually edit your run-image
 - you can remove the runleveld service (which is only used for
telinit emulation) and even the shutdownd service
 - you can write SIG scripts to do shutdowns the preferred way
 - I absolutely recommend against doing this, but you *still* have
a place in stage 1 where you can fiddle with things: in the
init script before the call to the s6-linux-init binary.

 So basically, all you're complaining about is that s6-linux-init-maker
is not generating your preferred run-image layout out-of-the-box
anymore. Well, you're an advanced user, you know what you are doing;
the knobs and levers are *still all there*. The only binary that
kinda hardcodes things is s6-linux-init itself, and if you give it a
try, I'm pretty sure you'll like it, because there was never any reason
to modify the core of stage 1 in the first place and what it does is
what any kind of stage 1 needs to do, no matter what language it's
written in.
 And if you don't like it, you're still free to ditch the s6-linux-init
package entirely and keep using your own stage 1.

 Besides, when systemd advocates paint sysv-rc shell scripts as
"duct tape", they're *right*. sysv-rc (and OpenRC) scripts are loaded
with boilerplate that only exists to compensate for the lack of a
supervision infrastructure, and systemd, like any supervision system,
does away with that. systemd has 99 problems, but rightly calling out
oversized script scaffoldings ain't one. Its disingenuousness lies in
pretending that an overengineered, opaque, all-encompassing, unescapable
framework is better than the duct tape; and I think you'll find that
s6-linux-init isn't quite the monster you seem to believe it is.

--
 Laurent



Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-01-29 Thread Laurent Bercot

Currently I do not understand the `s6-linux-init-shutdown(d)' way
well, so the old-fashioned way is retained at least for now, given its
simplicity in implementation and seemingly better flexibility.  Frankly
it is my intuition that the new way costs more than the old way, but
does not provide that much in return.  (Feel free to prove me wrong.)


 It may cost more *to you*, but there is real and significant value
in following existing interfaces that people are familiar with. Being
able to just use "reboot" instead of the, uh, slightly less intuitive
"s6-svscanctl -6 /run/service" to reboot your machine, is one fewer
obstacle on the way to mainstream s6 adoption.

 Additionally, and maybe more to your liking, there are also technical
benefits to never killing s6-svscan. Being able to assume that a
supervision tree will be operational at all times, including during
shutdown (and even in stage 4!), is really comfortable, it cuts down
on a lot of specialcasing, it makes shutdown procedures recoverable,
integration into various configurations easier (I'm thinking
containers with or without a catch-all logger, for instance), and
all-in-all has just less of a "screwdriver and duct tape" feel than
a bunch of execline (or rc ;)) scripts.

--
 Laurent



Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-01-29 Thread Laurent Bercot

Actually I do visit the CGit web interface fairly often


 Oh, my bad, the links in the skaware documents actually point to
https://git.skarnet.org/something. Fair enough then, I have made
git.skarnet.org an explicit alias to skarnet.org.



 Perhaps I need to batch change all
 references in the UP2020 document to
...


 No need - I'll own that one, and keep the alias explicitly working.
It's not like subdomains are a scarce resource.

--
 Laurent



Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-01-28 Thread Laurent Bercot

BTW,  seems to be returning empty HTTP replies
now; both  and  work as
expected though.


 That is a side effect of a recent s6-networking addition, where
s6-tlsd passes the SNI server name to the application via an
environment variable. Which allows me to serve virtual hosts even with
a HTTP/1.0 server, but only under TLS. Fun experiment. :)

 I may change it back, but I don't think the current state is broken,
because you're not supposed to access git.skarnet.org via HTTP(S)! :P

--
 Laurent



Re: Some suggestions on old-fashioned usage with s6 2.10.x

2021-01-28 Thread Laurent Bercot




I did not actively follow the recent evolution of s6, and have just been
bitten badly by s6 2.10.x on my Alpine servers (where slew [1] is used
of course) when it comes along with other updates.


 Sorry. This bears repeating: major version upgrades may break things.

 Compatibility is a good thing, that's why I try to keep major version
changes few and far between; but the other side of the coin is that
when I'm doing one, I want to make use of it and cram all the
incompatible changes that may be needed in the foreseeable future.
 So, you have to pay attention less often, but when it happens, you do
have to pay attention. Previous major version changes may have gone
smoothly - I try to keep it as smooth as possible when there's no need
to break UX - but it's no guarantee that it will always be smooth
sailing. This time, there were very visible user changes; sorry for
the inconvenience, but I reserve the right to do this, and I try to
document the breaking changes in the release notes.

 It is, admittedly, a drawback of distributions that they make major
version upgrades very silent - so, if you have local software that
relies on an old API, and the distro updates it under your feet,
you're caught unaware. I don't have a satisfying solution to this;
maybe I should have added a post-upgrade file printing red blinking
bold text, but that doesn't address automated or afk updates.



better if we kept the option supported for a transition period, and that
only removed it from the manual pages while urging users to get rid of
it.  After all, in this case, silently ignoring `-s' is behaviourly
similar to (if not perfectly compatible with) old `s6-svscan'.


 It's always a delicate balance, because "better" is not 
one-dimensional.

It would be better UX, yes, definitely. But also legacy code to maintain
until the next major update (which can take a while), and I tend to
assign a *very* high cost to legacy code in s6-svscan and s6-supervise,
for obvious reasons. And in my experience, few people (and you,
Casper, certainly belong to them!) actually bother changing their
scripts as long as they keep working - most only spring into action when
something breaks. A compromise I've found relatively efficient was to
add nagging warnings on deprecated option use, but 1. that's even more
code that will be removed, and 2. I hate nagware, with a passion, in
all its forms.
 There is no really good solution, and I prefer a short, sharp pain
(when things break) followed by relief (when they're fixed) to a long
dull ache (maintaining compat code). Especially when I'm not the one
experiencing the sharp pain ;)



Second, `s6-svscan' now waits for its `s6-supervise' children to exit
before exec()ing `.s6-svscan/finish'


 You seem to have found the proper way of managing this with SIG files,
but just in case: "s6-svscanctl -tb" will net you the old behaviour.

--
 Laurent



[announce] skalibs-2.10.0.1, execline-2.7.0.1, s6-2.10.0.1

2021-01-25 Thread Laurent Bercot



 Hello,

 New skarnet.org packages are available:

 skalibs-2.10.0.1
 execline-2.7.0.1
 s6-2.10.0.1

 Those are bugfix releases.
 I normally don't announce bugfix releases, but the bugs that have
been fixed here are pretty visible (sorry about that!), so all users
are encouraged to upgrade ASAP.

 https://skarnet.org/software/skalibs/
 https://skarnet.org/software/execline/
 https://skarnet.org/software/s6/

 Enjoy,
 More bug-reports always welcome.

--
 Laurent



Re: s6-man-pages updated to 2.10.0.0

2021-01-22 Thread Laurent Bercot

Very nice. Not to nitpick though, the standard way of handling
out-of-version versioning is with an underscore, not an additional dot.
So your releases would be v2.10.0.0_1 (and so on).


 The additional dot was on my suggestion, to avoid conflicting with
distros wanting to package s6-man-pages. Distros often use underscores
in their package naming schemes, and distro-version isn't always the
exact same as upstream-version.

--
 Laurent



Re: The multisubstitute commands in the run script generated by s6-usertree-maker are in a wrong order

2021-01-22 Thread Laurent Bercot

As shown above, the multisubstitute command that contains XDG_RUNTIME_DIR is 
put after the one that contains USER, HOME, UID, GID, and GIDLIST. If for 
example XDG_RUNTIME_DIR=/run/user/$UID, the $UID here will not be substituted 
with the user's UID since by the time $UID is substituted, $XDG_RUNTIME_DIR 
hasn't been substituted yet. So perhaps the order of these two multisubstitute 
should be inverted.


 You're right, of course. I remember testing it, and it *working*, so
I did not think any further, but in retrospect it appears my test was
incorrect.
 Thanks for the report! Fixed in git head.

 I'll give it the week-end, in case more bug-reports come in, then
I'll release 2.10.0.1.

 Note that skalibs-2.10.0.1 is out already, it fixes a bug that
manifests in execline's 'emptyenv -c' command.

--
 Laurent



[announce] skarnet.org New Year 2021 release

2021-01-10 Thread Laurent Bercot



 Hello,

 Happy New Year to everyone!
 New versions of the skarnet.org packages are available.

 This is a major release. The skalibs major version number has been
bumped, which means that compatibility with previous versions is not
ensured. Other packages have been updated to build against the new
skalibs. If they only had their patch number increased, that's all
the modifications they had (save for possible bugfixes); but some
packages also received significant changes and underwent either a major
(compatibility not ensured) or minor (simple additions) release.

 Support for the 2.9.* branch of skalibs, and associated versions of
the other packages, is still ensured for a while, but users are always
strongly encouraged to upgrade.


 * General
   ---

 - Some rarely-triggered build bugs have been fixed.
 - -fno-stack-protector is not part of the default CFLAGS anymore;
stack protector policy now defaults to the compiler's settings.


* skalibs-2.10.0.0
  

 - Bugfixes.
 - Significant code cleanup.
 - New sysdep: chroot.
 - Lots of new functions, mostly to optimize the number of needed
fcntl() calls at open() time. Traces should generally be marginally
shorter than they were before.
 - Removal of the DJBUNIX_FLAG_NB and DJBUNIX_FLAG_COE macros, replaced
by the POSIX O_NONBLOCK and O_CLOEXEC macros wherever they were used.
 - Removal of the skalibs/webipc.h header, and better header separation.
 - Complete revamping of the pathexec functions, now separated into
exec_* (simple execution) and mexec_* (execution with merging of the
environment first). In true skalibs fashion, there is a little code,
and 3 pages of convenience macros (the exec.h header).
 - Complete rewrite of the locking functions, with a change of
underlying mechanisms. The skalibs locking primitives are now named
fd_lock(), fd_unlock() and fd_islocked().
 The Unix locks primitive space is a horror show. flock() is not
POSIX and does not have a way to test for a lock without taking it.
The POSIX lockf() only has exclusive locks, not shared ones. The least
bad option is fcntl(), which has shared and exclusive locks *and* a way
to check for a lock without taking it, but does not allow taking a
shared lock via a O_WRONLY file descriptor. Of all inconveniences this
is the most minor one, so now skalibs uses fcntl().

 https://skarnet.org/software/skalibs/
 git://git.skarnet.org/skalibs


 * nsss-0.1.0.0
   

 - New --enable-libc-includes configure option. Without this option,
the pwd.h, grp.h and shadow.h headers are not installed anymore, so
by default installing nsss on a FHS system does not overwrite the
libc headers.

 https://skarnet.org/software/nsss/
 git://git.skarnet.org/nsss


 * utmps-0.1.0.0
   -

 - New --enable-libc-includes configure option. Without this option,
the utmpx.h header is not installed anymore, so by default installing
utmps on a FHS system does not overwrite the libc headers.

 https://skarnet.org/software/utmps/
 git://git.skarnet.org/utmps


 * execline-2.7.0.0
   

 - Bugfixes.
 - The trap program has changed. The "timeout" directive has been
removed; a "default" directive has been added, to handle all signals
for which a specific directive has not been given. Subprograms are
now run with the SIGNAL environment variable set to the signal number
(in addition to ! always being set to the application's pid).
 - The forstdin program has changed. It now exits 0 if it has read at
least one line, and 1 otherwise.
 - The default list of delimiters for backtick, withstdinas, forstdin
and forbacktickx has been set to "\n", so by default those programs
will read and/or split on lines and only lines.
 - The backtick, withstdinas, forstdin, forbacktickx, forx, getpid
and getcwd programs now have a -E option to activate autoimport.
(This saves the user from manually adding "importas var var" after
every use of these programs.)

 https://skarnet.org/software/execline/
 git://git.skarnet.org/execline


 * s6-2.10.0.0
   ---

 It is imperative to restart your supervision trees, by rebooting if
necessary, after upgrading s6 to the new version. Otherwise, new s6
binaries interacting with service directories maintained by old
s6-supervise binaries may not work.
 If you are using s6-linux-init, it is necessary to upgrade to the
latest version of s6-linux-init at the same time as s6.

 - Bugfixes.
 - Significant code refactoring.
 - The internal locking system of service directories has changed,
allowing for a cleaner permissions model and official support of
relaxed permissions.
 - New binary to implement those relaxed permissions: s6-svperms.
 - The "nosetsid" file is not supported anymore in service directories.
Services are now always started in a new session.
 - s6-supervise now traps SIGINT: before dying, it sends a SIGINT to its
service's process group. This allows correct transmission of ^C when a
supervision tree is running in a terminal, even though every 

Re: Answers to some GitHub questions

2021-01-09 Thread Laurent Bercot

Is there a migration guide from runit to s6?


 I added a very basic list of things to do as a FAQ in
 https://skarnet.org/software/s6-linux-init/quickstart.html

 Please comment if there are things that need change or clarification.
(The prose is bad, I know; please stick to content suggestions.)
I linked to Colin's post, with his permission, for a more in-depth
explanation.

--
 Laurent



Re: Why are there 3 programs in s6 that do the same thing?

2020-12-29 Thread Laurent Bercot

s6 has three programs to set UID / GID of a process: s6-setuidgid, 
s6-applyuidgid and s6-envuidgid. All three of those seem to do the same thing, 
with some *slight* differences. Why are all three included in s6?


 Those three programs don't do the same thing at all. They do *related*
things; and one of them is a wrapper around the other two.

 There are two different operations at play here:

 1. get the uid, gid and additional groups from the pwd/grp database
for a given username. It is a data retrieval operation with no impact
on the process state (save the storage of the information in the
environment, for later use by other programs in the chainloaded
command line). This is what s6-envuidgid does.

 2. actually change the current process' uid, gid and additional groups.
There is no data retrieval here, but there is a process state change.
This is what s6-applyuidgid does.

 Those two operations are distinct because you may want to set the
uid, gid and additional groups of a process to something that does not
exist in the current user database: think containers, or 
cross-compiling.

Roughly speaking, s6-envuidgid reads data from the "host", but
s6-applyuidgid performs a process state change on the "target".

 That said, in common cases you want to perform both operations at
once: and this is what s6-setuidgid does.

 Originally, there were only s6-setuidgid and s6-envuidgid, as direct
adaptations from daemontools' setuidgid and envuidgid. However, it
soon appeared that those two programs did not encapsulate the set of
operations that s6 needs to do, and a program was needed that would
only perform the process state change. Which is why s6-applyuidgid
was added; and then, in order to save code, s6-setuidgid was rewritten
as a wrapper around s6-envuidgid and s6-applyuidgid.

--
 Laurent



Answers to some GitHub questions

2020-12-25 Thread Laurent Bercot



 Hi,

 Someone on GitHub asked a series of questions about s6. I decided to
answer them here, in order to have the post in the ML archives and a
place (outside of GitHub) to refer people to if similar questions arise
in the future.


 * Benefits of s6 over runit?

 s6 has multiple benefits over runit. I'd say that its biggest advantage
is that it is actually maintained and actively developed (albeit more
slowly than I would like). Also, it has been written after runit, so it
learned from it and expanded on it, and considerable effort has been
put into it.

 From a developer's point of view, there is *a lot* that can be said in
favor of s6: technical details that are mostly invisible to the user but
that add up to significantly cleaner code. An example of this is that
s6's supervisor program, s6-supervise, is implemented as a fully
asynchronous state machine, which means it will always be responsive to
user commands no matter what happens; whereas there are situations in
which runsv, runit's supervisor program, will be unresponsive for up to
5 seconds. These details matter.

 From a user's point of view, the most visible difference is that s6
comes with a lot of additional tools that help make it a powerful
toolbox for the admin, in supervision-related matters and beyond. For
instance, s6 keeps a tally of the latest deaths of a process, and allows
you to declare permanent failure on a configurable pattern of deaths.
Or, s6 provides you with an Unix domain super-server that allows you to
quickly write local services, with access control done by uid/gid.

 A comparison between s6 and other supervision suites, including runit,
is available here: https://skarnet.org/software/s6/why.html
 It's an old page, and now very incomplete, but what is written is still
accurate.


 * Why use a database instead of symlinks?

 This question, I think, comes from a misunderstanding. s6 "uses
symlinks" just like every other supervision suite. s6-svscan has a scan
directory, just like runsvdir; s6-supervise has a service directory,
just like runsv. s6 and runit are very similar in that respect.
 Where the database comes into play is when you're using s6-rc, which is
a related, but distinct, piece of software:
 https://skarnet.org/software/s6-rc/

 s6-rc is not a supervision suite, but a service manager that runs *on
top of* the s6 supervision suite. Those are complementary tools. And
yes, s6-rc uses a database (but not a relational one, and no, there is
no SQL backend!), mostly because dependencies between services need to
be stored somewhere. Note that other service managers, like OpenRC or
systemd (the latter being a supervision suite and an init/shutdown
system as well), *also* use a database, they just do not tell you
upfront, and trying to pretend to the user that there's no database
actually makes them more complex and brittle than necessary.


 * Why don't some command-line options have a long form?

 Because s6 is not a GNU package. Long options need the getopt_long(3)
function in order to be parsed; this is a GNU extension to POSIX, and s6
tries to avoid those and stick to POSIX features as much as possible.
 The upcoming high-level interface to s6, s6-frontend, will accept long
options, but it's still in the distant future.


 * You said that the concept of runlevels in runit didn't work in case
of udevd, which had to be run unsupervised in runlevel 1. How does s6
address this problem?

 I certainly did not say that, because runit does not use runlevels,
which are a legacy concept from sysvinit, mixing two completely
different things - halt/reboot procedures (0 and 6) and machine states
(1 to 5, and 1 is a special case requiring particularly ugly hacks).
 But I did say something similar with *stages of init*.

 With the runit model, runsvdir, which starts the supervision tree, is
run in stage 2. Before stage 2, there is no supervision tree available.
And once runsvdir runs, there is no possibility of running oneshots.
So all the oneshots have to be run in stage 1. When a oneshot depends
on a longrun (a daemon), it means that that longrun also has to be run
in stage 1, which prevents it from being supervised since the
supervision tree is not running yet.

 s6 addresses this problem in two ways. Rather, more accurately, it
*does not* address this problem, because it's not for a supervision
system to solve: it's about ordering services, which is the job of a
*service manager*, not a supervision suite. (And that is why runit,
which tries to be a supervision suite and also a /sbin/init and pid 1
but not a service manager, does not solve it.)

 Instead, s6 delegates the problem to other packages that were
specifically written for this:
 - s6-linux-init: https://skarnet.org/software/s6-linux-init/
is an early boot package for Linux, that provides sysvinit
compatibility, and that starts a supervision tree as early as
possible, before running basically *any* service. By using this
package, admins can have a s6-svscan 

Re: How to recover from s6-rc broken pipe?

2020-12-16 Thread Laurent Bercot

sudo rsync -avr --delete compiled/ /etc/s6-rc/compiled/
sudo s6-rc-update /etc/s6-rc/compiled/


 There's your problem. You're overwriting the live database; that will
throw a wrench into everything.
 s6-rc-update needs to run while the live database is still the old
one. Only after s6-rc-update has completed can you delete the old
database.

 Do something like this instead:

stamp=`echo | s6-tai64n`
s6-rc-compile /etc/s6-rc/compiled-$stamp sources...
s6-rc-update /etc/s6-rc/compiled-$stamp
old=`readlink /etc/s6-rc/compiled`
ln -sf compiled-$stamp /etc/s6-rc/compiled.new
rename /etc/s6-rc/compiled.new /etc/s6-rc/compiled
rm -rf /etc/s6-rc/$old

--
 Laurent



Re: How to recover from s6-rc broken pipe?

2020-12-16 Thread Laurent Bercot




I'm using s6-rc to manage services and have been changing databases.

For some unknown reason sometimes the update fails with the error:

s6-rc-update: fatal: unable to read /run/s6-rc/state: Broken pipe


 That should definitely not happen.
 Have your databases been built with the same version of s6-rc as
the one you're using? (Normally they're compatible, but there has
been an incompatible change between 0.3 and 0.4)



When that happens, I cannot use s6-rc anymore:

/run/s6-rc does not exists, but s6-rc declares it as if it does:


 s6-rc-update removes the /run/s6-rc symlinks when it fails? If it
does, it's a bug that I will fix for the next release.



s6-rc-init: fatal: unable to supervise service directories in
/run/s6-rc/servicedirs: File exists

Creating the s6-rc symlink does not improve the situation.

How should I recover from this error?


 It is possible that you have a bunch of dangling symlinks in
/run/service, that were pointing to your old live directory, are not
valid anymore, but are still preventing s6-rc from doing its job.

 Generally speaking, s6-rc-update failing is bad news, because it is
difficult to do the proper cleanups (either automatically when
s6-rc-update fails, because some operations cannot be rolled back, or
manually afterwards), so yeah, your scandir may be in an ugly state
and you may need to remove all the symlinks there, delete all the
/run/s6-rc* directories and/or symlinks, and restart from scratch.
Depending on the changes your oneshots made on your system, you may
get error when running them again, too, so in the worst case, the
only good option might be to reboot. Sorry.

 But really, the original cause of the problem should not happen, and
s6-rc-update should not be failing like this. If it does not happen
all the time, is something overwriting your state file? or removing
the /run/s6-rc symlink? If your databases are compatible, then there
is definitely some external interference here.

--
 Laurent



Re: service creation in s6

2020-12-16 Thread Laurent Bercot

Is there a way of setting capability binding in S6?
like "CapabilityBoundingSet=CAP_NET_ADMIN"


 https://www.man7.org/linux/man-pages/man8/setcap.8.html

--
 Laurent



Re: s6-supervise: use of nosetsid

2020-12-03 Thread Laurent Bercot

It seems I missed the addition of s6-svscan's -X option; I saw it one
day in the output of 'ps' on a VM that was using s6-linux-init, didn't
recognize it, but then forgot to look up what it did :P But I would
have to 'tag' messages in some way, so that the catch-all logger
recognizes them and sends them to the terminal, right?


 Yes. That's why it's the "advanced" solution over just leaking a fd
for /dev/tty.



 Anyway, nosetsid + /dev/tty is still the simplest implementation, I
have to admit.


 The more I think about it, the more I see any use of nosetsid as
incorrect. Implementing SIGINT handling for s6-supervise requires
jumping through serious hoops if the nosetsid function still exists, and
the point of my current work on s6 is to get things simpler, not more
complex.
 I'm leaning more and more in the direction of removing nosetsid, which
is why I want to help people find other ways to accomplish what they're
currently doing with it.

--
 Laurent



Re: s6-supervise: use of nosetsid

2020-12-03 Thread Laurent Bercot

I remember occasionally using 'nosetsid' files for the convenience of
being able to redirect the output of 'run' and 'finish' to /dev/tty
(the controlling terminal of the shell I used to run s6-svscan) so
that I could see certain messages during tests in which s6-svscan had
its stdout & stderr redirected to a logger (typically, for emulating
an s6-linux-init-like s6-svscan-as-process-1 scenario). I think it was
for doing something like

foreground {
  redirfd -w 1 /dev/tty
  echo "Some message I want displayed on my terminal"
}


 Interesting. Would you be able to do the same thing with saving
/dev/tty as an extra file descriptor instead?

s6-svscan 3>&1 &

...

foreground {
  fdmove 1 3
  echo "Some message I want displayed on my terminal"
}

 Granted, this leaks a fd into all your services, but if it's for
terminal testing it shouldn't matter much. Otherwise, for maximum
hacking value, you could use the -X option to s6-svscan, which would
not leak the tty fd into all the services, but only transmit it as
s6-svscan-log's stderr - so in order to print something to the
terminal you'd have to send it to the catch-all logger, and configure
that logger with a filter that sends those messages to stderr. :)

 So, can you juggle fds to avoid nosetsid, or can't you do the same
thing without it?

--
 Laurent



Re: s6-supervise: use of nosetsid

2020-12-03 Thread Laurent Bercot

dcron crashes without it, which is why [1] exists.  The log says
`setpgid: Operation not permitted'.  I have not investigated further.


 The problem happens here:
 https://github.com/dubiousjim/dcron/blob/master/main.c#L272

 dcron assumes that it's not a session leader when run in the 
foreground.

This is a programming error that should be reported and fixed; if
setpgid(0, 0) fails with EPERM, it just means that the process is
already a session leader, and the program should simply keep running
instead of exiting.

 So, not a valid reason. ;)

 As a workaround, you can avoid nosetsid by just running dcrond under
execline's 'trap', or a shell, or basically any program that will
just forward signals to it.

--
 Laurent



s6-supervise: use of nosetsid

2020-12-03 Thread Laurent Bercot



 Hello,

 The next version of s6 will be a major bump, with a few long-awaited
QoL changes - mainly a thorough cleanup of how s6-svscan handles
signals and the various commands sent by s6-svscanctl, but also some
goodies that you should like. :)

 One issue that has been often reported by users is that when they
try running s6-svscan in a terminal, and then ^C to kill it, the
services remain running. This is intentional, because supervision
suites are designed to isolate processes from anything accidental that
could bring them down, and in particular services should normally
survive supervisor death - but so far there has been many more
instances of people having trouble with that behaviour than instances
of s6-supervise accidentally dying.

 I have previously added the "nosetsid" feature to s6-supervise, to
address the issue: having a "nosetsid" file in a service directory
prevents the service from being started as a session leader, it starts
in the same session as the supervision tree (and, if the nosetsid file
is empty, in the same process group). So when people want to manually
test a supervision tree, they can have nosetsid files in their test
service directories, and ^C will send a SIGINT to all the processes
including the services, so everything will die, which is what they
want.

 There are two problems with the nosetsid approach:

 - Oftentimes, users are not aware of the existence of nosetsid, and
still experience the issue. It's almost an s6 FAQ at this point.
 - The nosetsid functionality is inherently a risk: it puts the
whole supervision tree at the mercy of a misbehaved service that would
send a signal to its whole process group. There is a reason why
s6-supervise normally starts services in a different session, and
nosetsid bypasses that safety measure.

 So I am thinking of another approach to make s6 friendlier to users
who would - despite it not being recommended behaviour - test a
supervision tree in a terminal: have s6-supervise handle SIGINT and
make it kill its service before exiting. That would ensure that ^C
cleans up everything.

 This approach has the drawback of making services a little less
resilient, but s6-supervise getting a SIGINT should *only* happen in
the case of someone running a supervision tree in a terminal, which
is absolutely not something that should exist in production, so it's
probably not a big concern. However, it comes with a major advantage:
it removes the original reason for the addition of nosetsid.
 So, with the change to ^C handling, I am considering removing the
dangerous nosetsid functionality entirely.

 Hence, my question to users: do you have a *valid* reason to use
nosetsid files in your service directories? Are there use cases for
nosetsid that I have not thought about, and that would make using s6
impractical if the functionality were to be removed?

 Thanks in advance for your input.

--
 Laurent



Re: Expected behaviour zombie processes

2020-12-01 Thread Laurent Bercot

Hi everybody

I've observed the following situation:

root   490  0.0  0.0  13004  1596 ?SNov05   0:00   s6-supervise 
lxe
root  2738  0.6  0.0  0 0 ?Zsl  Nov05 251:13 [lxe] 


So you see a zombie process. What is the expected behaviour of
s6-supervise here? I would expect, that this case is solved immediatly
by s6-supervise and a new process is spawned.

The observed behaviour was, that no new process was spawned and even
s6-svc -r didn't respawn a process.


 Hi Oli,
 There is no indication that the zombie lxe process is a direct child
of s6-supervise. What does 's6-svstat /run/service/lxe' say?

 If that lxe process was s6-supervise's child, s6-supervise should
definitely reap the zombie immediately and spawn a new lxe process;
failing to do so would be a bug that has, to my knowledge, never
happened. There is definitely something else at play here.

--
 Laurent



Re: How can I make the stdin of a longrun process available to other processes by writing to a file?

2020-11-20 Thread Laurent Bercot

-  Using a named pipe and have `skabus-dyntee` read from it (and it dies as
soon as the writer program ends).


 Yes, that's normal. If you want to avoid this, you need to keep the
named pipe open, and that can be done via having another process opening
the named pipe for writing, and not dying. That process can even be the
reader itself, see below.



-  Configuring up s6-fdholder for my user to use (but it seems I don't
understand how to give it the stdin to read).


 You don't need a fdholder just for this: you already have a living
process that can maintain the named pipe - skabus-dyntee itself.
Try the following:

exec named-pipe
exec skabus-dyntee args...

(or in execline:
 redirfd -r 0 named-pipe
 redirfd -w 42 named-pipe
 skabus-dyntee args...
)

--
 Laurent



Re: s6-rc: timeout questions

2020-11-18 Thread Laurent Bercot

Could you elaborate a little more about the state transition failures
of oneshots caused by timeouts?

Let's say for example the oneshot's up script times out, so the
transition fails. From s6-rc's point of view the oneshot is still
down. What actually happens to the process running the up script? Is
it left running in the background? If yes, is it correct to assume
that since s6-rc considers it down, another invocation of the s6-rc -u
change command on the same oneshot will spawn another instance of the
up script? If not, is it killed, and how?


 It is correct to assume that another instance will be spawned, yes.
It was a difficult decision to make, and I'm still not sure it is the
right one. There are advantages and drawbacks to both approaches, but
at the end of the day it all comes down to: what set of actions will
leave the system in the *least* unknown state?

 s6-rc's design assumes that timeouts, if they exist, are properly
calibrated; if a service times out, then it's not that the timeout is
too short, it's that something is really going wrong. So it considers
the transition failed. Now what should it do about the existing
process? kill it or not?
 If the process is allowed to live on, it may succeed, in which case
s6-rc's vision of the service will be wrong, but 1. it doesn't matter
because services should always be written as idempotent, and 2. it means
that the timeout was badly calibrated in the first place. Or it may
fail and s6-rc's vision will be correct.
 If the process is killed, chances are that it will add to the problem
instead of solving it. For instance, if the process is hanging in D
state, killing it won't do anything except make the system more 
unstable.

If the process is doing some complex operation and not properly
sequencing its operations, sending it a signal may trigger a bug. etc.
 In the end I weighed that sending a signal would potentially cause more
harm than good, but I don't think using the opposite approach would be
wrong either.



Test 2:
s1 is down
s2 is down
s6-rc -u change s2
s6-rc: fatal: timed out
s6-svlisten1: fatal: timed out

Timeout failure. Unexpected. I thought timeout-up and timeout-down
applied to each atomic service individually, not to the entire
dependency chain to bring it up or down.


 Yes, it should be behaving as you say, and I suspect you have
uncovered a bug - not in the timeout management for a dependency chain,
but in the management of s6-rc's *global* timeout, which is the one
that is triggering here. I suspect I'm taking incorrect shortcuts wrt
timeout management, and will take a look. Thanks!

--
 Laurent



Re: s6-permafailon not acting as expected

2020-11-18 Thread Laurent Bercot

Sounds good. So what kind of interface changes should we expect? I
mean will they require updating the service definitions, run/finish
scripts, that sort of thing?


 (Sorry, I missed it indeed.)

 Nothing that drastic; service definitions will not change. They are the
core of s6, and such a change would warrant a lifetime version bump,
I think.

 No, the next release will be about a long overdue cleanup of s6-svscan
and s6-svscanctl invocations: command-line options will change, and
default semantics of signals received by s6-svscan will also change.

 The point is to remove some clutter that is either:
 - legacy behaviours from daemontools's svscan that have not shown they
were useful
 - ad-hoc stuff that was added to support early iterations of running
s6-svscan as pid 1, and that isn't used anymore (in part thanks to the
existence of s6-linux-init 1.x)
 - options that experience has shown to be traps (i.e. ideas that seem
good at first but really encourage bad patterns)

and make the s6-svscan/ctl system less confusing to new users by only
keeping a few relevant and useful configuration switches.
 There will also be QoL improvements to s6-svscan for people who like
to nest supervision trees; also, no promises, but I may come up with
some way to officially support user trees, which are a relatively
common request.

--
 Laurent



Re: s6-rc: timeout questions

2020-11-17 Thread Laurent Bercot

- Am I correct in thinking that if a service has properly configured
timeout-kill and timeout-finish, timeout-down becomes unnecessary and
even undesirable as it can leave services in an undefined state limbo?
I know the documentation pretty much says so, but I'm still a bit
confused by the existence of timeout-down to begin with, if it's
redundant and unhelpful.


 timeout-kill and timeout-finish are a s6 thing: if present, they're
just copied as is to the service directory that will be managed by
the s6 supervision tree.
 timeout-up and timeout-down are specific to s6-rc: they will be
embedded into the compile database. They do not interact with s6 at
all, they're just a rule for the s6-rc state machine: if the
service does not report being up (resp. down) by the timeout, then
s6-rc marks the transition as failed and stops looking at what
happens with the service.

 For longruns, yes, timeout-kill ensures that the service will
eventually be brought down no matter what. But there are cases where
you *do not want* to kill -9 a daemon (and need a timeout-kill of 0).
timeout-down is useful here, even if it's a pretty niche case.

 And then, of course, the point is that it's needed for oneshots,
which do not have the s6 mechanisms.



- Can you confirm that timeout-up and timeout-down are also used with
oneshots? They are defined in the s6-rc-compile documentation, but the
s6-rc documentation doesn't specifically mention them for oneshots
state transitions.


 Yes, I confirm that they're also (and primarily) used with oneshots.
They're defined in the "atomic services" section, which comprises
longruns *and* oneshots.

--
 Laurent



Re: s6-permafailon not acting as expected

2020-11-17 Thread Laurent Bercot

Back to s6-permafailon.c: the sigset_t sigs is not initialized
either... Looks like it needs a call to sigemptyset().


 Gah! -_- Fixed. Thanks.

--
 Laurent



Re: s6-permafailon not acting as expected

2020-11-16 Thread Laurent Bercot

The service did die with exit code 0, but s6-permafail is wrong, since
exit code 0 is not specified in the filter.

Looking at the source code in s6-permafailon.c, it seems to me that
the codes[32] bitarray is not initialized with zeroes before calling
bitarray_set(n).


 You're probably right that it is the issue! (codes was a static
variable in the first iteration; static variables do not need to be
0-initialized. But then I reworked the code, made codes an automatic
variable, and failed to initialize it to zero. And that is why relying
on magic features is a bad thing, and I should stop doing this out of
laziness.)

 I fixed it in the latest git of s6; if you prefer numbered releases,
I'm working on a bigger release (with some interface changes and a
major version bump) that should happen before the end of the year, but
not right now ^^'

 Please try the fixed version. If you don't want to use the git head,
you can apply the very simple diff:
 
https://git.skarnet.org/cgi-bin/cgit.cgi/s6/diff/src/supervision/s6-permafailon.c


 If that version doesn't solve your problem, then I'll investigate
further.

 Thanks for the report!

--
 Laurent



Re: Missing "$@" at the end of the s6-linux-init invocation in basedir/bin/init script

2020-11-10 Thread Laurent Bercot

So, is that "$@" supposed to be there or not?


 It absolutely is! You found a bug. Thanks for the report!
 Now fixed in release 1.0.5.1, which I cut immediately because it is
pretty critical.

--
 Laurent



Re: Handling environment in s6

2020-11-10 Thread Laurent Bercot

Is there anything in s6 equivalent to below of systemd?

/bin/systemctl set-environment SERVICEFILE=$SERVICEFILE


 There is not.
 The process supervisor itself (s6-supervise) and the service scanner
(s6-svscan) do not use the environment. Services can define their
own environment in their run script.

 There are several tools that help you define an environment in a run
script. The most idiomatic one is s6-envdir:
 https://skarnet.org/software/s6/s6-envdir.html

 If you have EnvironmentFiles that come from systemd, or overall
prefer to define your environment variables in a file, you may find
the envfile program useful - it's in the execline package:
 https://skarnet.org/software/execline/envfile.html

 Or you can simply write your run script in shell, and define your
environment variables via the shell syntax, or via sourcing a file,
as a lot of distributions do with /etc/default/*.conf or similar
mechanisms.

 Providing a command that changes the supervisor's environment at
run time is one of the many useless - and in this case, harmful -
features of systemd. It is harmful because it introduces changes
between what's written on disk (which is what will happen at boot time,
and what the user can see when studying files) and the current state
of the supervisor, so future service operations will violate the
principle of least surprise. It destroys the "if it's working now then
it will work after I reboot" property. And of course having the
supervisor rely on externally modifiable environment variables makes
it more vulnerable.

 If I'm judging from your recent batch of questions these last few
weeks, you are trying to convert a set of systemd services to a s6
installation. This is a very good idea (;)) and I commend you for
taking up this task.
 However, listing all the systemd features and asking 'how do I
accomplish this in s6?' is not the right way to go about this. The
systemd features have been designed with the systemd mindset, and
individually they make sense (or not) in the context and with the
architecture of systemd. Trying to exactly map the features one-to-one
would result in writing another systemd behemoth, as the author of
uselessd discovered to their dismay.

 The s6 mindset is very different from the systemd one, and there is
no direct mapping from one to the other. In order to convert a set of
services, you need to take a step back and look at the big picture:
what high-level functionality do I need? how am I expressing that
functionality with systemd? what is the way to express it with s6?
It requires a deeper rework than just a syntactic one, which is why it
is really difficult to write an automatic converter, and why it is also
generally difficult to answer questions like yours without going into
the details of what exactly you're trying to do.

--
 Laurent



Re: s6 in production on Ubuntu - yeah!

2020-11-04 Thread Laurent Bercot

we're proud to announce, that we have s6 in production in context of
platform as a service for our customers.


 That's awesome! Glad it's working, and thanks for your return!



Thanks Laurent for supporting us to develop the integration of s6 in
ubuntu 16 and 20. We can definitly recommend to engage Laurent for
integration questions.


 The pleasure was mine :)

--
 Laurent



Re: socket based service bringup

2020-11-03 Thread Laurent Bercot

Is it s6-ipcserver which can be used to make an s6 socket based service
equivalent .socket service in systemd?


 For unix-domain sockets, yes, it is.
 For inet-domain sockets, the equivalent is s6-tcpserver, available in
the s6-networking package.

--
 Laurent



Re: writing s6 services

2020-10-28 Thread Laurent Bercot

1) Is there a possibility to add an ENV variable dynamically?
I have a shell script which fills out some variables like
IPADDRESS, SERVER  etc .. which I need to use in starting a process later
as part of starting a service/stopping a service.


 You can do whatever you want in a run script, as long as by the end
of it, the pid of the long-running daemon is the pid your run script
was started as.
 If your run script is a shell script, you can absolutely source the
script that fills out your environment variables. It is not the most
idiomatic or safe way to do it, but it will absolutely work.



2) Does s6-supervise has the intelligence of findingout $MAINPID as the
systemd does?(
https://systemd-devel.freedesktop.narkive.com/dpY7US7K/a-little-help-with-mainpid-please


 Yes, that is the point of a process supervisor.
 You don't need a variable such as $MAINPID because the supervisor 
always

remembers the pid of its child. You don't need a pidfile or anything of
the sort.

--
 Laurent



Re: Can you use s6-rc with the sysvinit PID1?

2020-10-23 Thread Laurent Bercot

If I use sysvinit's PID1, is it possible for me to use s6-rc by
declaring an s6-rc program in /etc/inittab?


 Yes. You have to tell /etc/inittab to launch a s6 supervision tree
first, then s6-rc-init, then s6-rc -u change top (if "top" is the bundle
containing all the services you need), as one-time services.

 There is an example with OpenRC in Adélie: a sysvinit init, that
controls a s6 supervision tree then runs an openrc service manager.
 
https://code.foxkit.us/adelie/packages/-/blob/master/system/sysvinit/inittab-2.88

(The s6-svscanboot code is in the same directory in the repository.)
 If you remove the 'openrc sysinit' and 'openrc boot' lines,
and change the 'openrc $runlevel' lines below to 's6-rc change 
$runlevel'

invocations, then you will have a s6-rc managed system running on top
of sysvinit.

 However, it's much more work to convert a set of init scripts to
s6-rc than it is to switch from sysvinit to s6-linux-init, so you may
just as well switch to s6-linux-init first and spare yourself some
unnecessary work and some ugly init code. ;)



Also, and this is offtopic, is there a way to tell sysvinit not to run
the programs in /etc/rc.d/rc3.d or whatever? This would make it trivial
to switch between an sysvinit initted system and an s6-rc/s6 supervised
system just by commenting in or out the inittab entry and switching
sysvinit to looop /etc/rc.d/rc3.d?


 You can just comment out the lines you don't want in /etc/inittab.
 However, controlling init via a file is awkward and fragile, and you
may find yourself performing a lot of scripting (and making your
system unbootable and having to recover via init=/bin/sh once or
twice), which is another reason why s6-svscan makes a better pid 1.

--
 Laurent



Re: External health Check Process

2020-10-23 Thread Laurent Bercot




I just miss the elegance of the solution: I personally want to model
one service with one s6 service. For me it would mean thinking about a
wrapper around s6 to get that. Maybe I get now the slew thing.


 The thing is, s6 is a *process supervision* suite, so one s6 "service"
is really one long-running process. When you want health checks, you
have two long-running processes: your daemon, and your health checker.
So two s6 "services" is really the most elegant, most idiomatic and
most natural solution.

 What you could have, on the other hand, is a s6-rc bundle, that 
contains

both your daemon and your health checker: so you would be able to
handle both the daemon and the health checker (2 longruns) with a
single s6-rc/svctl command, using the name of the bundle.

 It's probably something that I can add to the next version of s6-rc:
a command or an option to automatically add a health checker service to
a longrun that is declared in the database, so you wouldn't have to
write the health checker longrun manually. How does that sound?



And it's ok to need a wrapper to get useability, but the
advertisement of that should be better on the website that you SHOULD
use that wrapper (and for me this wrapper should be part of the s6
project).


 This is indeed a UI problem and I'm still working on it. ;)

--
 Laurent



Re: External health Check Process

2020-10-22 Thread Laurent Bercot

I know you can model a service in s6, which watches another service and
kills it, so in fact the problem is solved outside of s6. But I wanted
to ask to develop a feature to get a simple way to model that within s6.
Usually it's good enough to call a external command with a timeout and
watches exit code.


 Hi Oliver,

 The s6-idiomatic way of doing it would be, as you say, to have a
separate service that calls an external command (the health checker,
which is daemon-specific) with a timeout and watches the exit code.
It is trivial to do in shell, which is why I haven't written any
particular binary for that.

 I could add a program that does it for you so you don't have to write
a 3-line shell script, and a command that creates a s6 service directory
(or even a s6-rc source definition directory) that watches another
service using the aforementioned program, it would not be hard.
However, I am concerned about scope creep, and a common criticism I
hear from distros is that s6 is "too big" - which is unfair considering
that integrated init systems providing the same level of functionality
are 5x-10x bigger, but is really a way of saying that there are a lot of
exposed binaries with miscellaneous functionality and it's difficult to
wrap one's head around it. So I'm trying not to add to the problem, and
the direction I'm going these days is more towards integration and
high-level management than towards adding building blocks to help with
various tasks, so if something is doable with a bit of scripting, then
I'd rather let users do it that way.

 I'm pretty sure that people in the community already have run script
models for healthchecker services, if they could contribute them it
would be awesome ;)

--
 Laurent



Re: Dependencies in S6

2020-10-21 Thread Laurent Bercot



 Hi Amaresh,

 As long as service B depends on service A, A and B will never start
in parallel; B will always start when A is ready.

 "A is ready" means:
  - if A is a oneshot: when the script has completed
  - if A is a longrun:
* if there is a notification-fd file in the service definition
directory: when the daemon has notified readiness
* if there is no such file: as soon as the run script has started
(which is not a good readiness indicator; it is akin to starting B
_right after A_, essentially in parallel, so you may have race
conditions here - which is why you should always use readiness
notifications.)



Secnario 1:
1. A & B are one shot services in a s6-rc bundle. Service B is dependent on
A. ( added in dependency file).
2. I think Service B will start after the completion of service A ( A then
B, in serial manner ) is it correct ? or
3. Service B will start immediately after the service A as started (in a
parallel manner) ?


 Serial manner, A then B.



Secnario 2:
1. A is one shot service and B is a long run service. Service B is
dependent on Service A.
2. Service B will start, once th service A as completed (in serial manner)
? or
3. Service B will start immediately after the service A as started (in a
parallel manner) ?


 Serial manner, A then B.



Secnario 3:
1. A is a long run service and B is a one shot service. Service B is
dependent on Service A.
2. Service B will start immediately after the service A as started (in a
parallel manner) ?
3. Do I need to use notification mechanism if I want to start service B
after A service as reached particular point in execution


 Yes, you need notification; if you don't, then B will start
immediately after A.



Secnario 4:
1. A & B both are long run services. Service B is dependent on Service A.
2. Service B will start immediately after the service A as started (in a
parallel manner) ?


 Same as above: almost-parallel manner if A does not use notification,
serial manner if it does.



 If one shot service are dependent on other one shot services. They will be
started in serial manner one after the other ?


 That is your Scenario 1 above.

 If the case is: "C depends on A and B, but A and B are independent",
then:
 * A and B are started in parallel
 * when both A and B are ready, then C is started.

 Hope this helps,

--
 Laurent



Re: Module-load binary for S6

2020-10-09 Thread Laurent Bercot

Is the module-load binary is available for S6 to load all the kernel module
?

Is there any support for volatile binds in S6 ( S6 tool for volatile bind )


 Hi Amaresh,

 Those are operations that are totally irrelevant to s6; you should use
the tools that your distribution provides you.

 To see what s6 is about, what it does, read the following overview:
 https://skarnet.org/software/s6/overview.html

 Other operations on your computer will be handled by other software.

--
 Laurent



Re: s6 v2.9.2.0 compiles only disabling execline

2020-10-08 Thread Laurent Bercot

Which advantages does this design bring to the table?


 - Header files are execline/*.h, it's logical to put static library
files in execline/*.a

 - /lib and /usr/lib are places where the dynamic linker looks for
shared libraries. Despite sharing the name "library", static and shared
libraries are not at all the same kind of object, they're not used for
the same purpose (shared libraries are run-time, static libraries are
build-time), so they should not be put in the same places. It is a
design mistake of a linker to look for static libraries in /lib and
/usr/lib, and a historical mistake of FHS not to have specified a
different default place for static libraries.

 - /usr/lib/foobar is traditionally used to host other files that are
related to the foobar package, such as unexported executable files
("libexec") or other read-only data. It is a naturally appropriate place
to put static libraries from the foobar package.

 Using /usr/lib/foobar as the default place to put libfoobar.a was my
attempt to organize files more rationally than FHS does and ld
encourages, but unfortunately people don't read INSTALL files or
./configure --help carefully (:P) and they're surprised when things
deviate ever so slightly from the lemming herd of all the autoconf-using
software - you're neither the first one to have trouble with this, nor
the last one.

 I guess it's my bad for pushing policy instead of sticking to 
mechanism.

I'll change the default in future versions of skarnet.org packages.



Why do some versions can compile without changes and some can't?


 It's not the case.
 If you previously managed to build skarnet.org software without
specifying --libdir=/usr/lib or --with-lib=/usr/lib/foobar, then
chances are you already had an old version of libfoobar.a installed in 
/usr/lib/foobar (and linked against it), or you specified

--disable-allstatic and linked against libfoobar.so (which is in /lib
or /usr/lib). In the latter case, your build is correct; but in the
former, it accidentally succeeded and you didn't use the version of
libfoobar that you were aiming for.

 Which is, to me, a much bigger problem than hitting a build-time error.
I don't know how many incorrect builds have accidentally passed and are
currently running out there; this is reason enough to change the 
default.


--
 Laurent


Re: s6 v2.9.2.0 compiles only disabling execline

2020-10-07 Thread Laurent Bercot

I'm building s6 from source on latest git tags and it seems to fail because
it cannot link execline:

./configure --disable-execline; make # works

./configure; make # fails with message
exec gcc -o s6-ftrig-listen -pipe -Wall -std=c99 -fno-exceptions
-fno-unwind-tables -fno-asynchronous-unwind-tables -Wa,--noexecstack
-ffunction-sections -fdata-sections -O2 -fomit-frame-pointer
-fno-stack-protector  -Wl,--sort-section=alignment -Wl,--sort-common
-Wl,--gc-sections src/pipe-tools/s6-ftrig-listen.o libs6.a.xyzzy -lexecline
-lskarnet
/home/linuxbrew/.linuxbrew/bin/ld: cannot find -lexecline
collect2: error: ld returned 1 exit status
make: *** [Makefile:138: s6-ftrig-listen] Error 1

Execline is installed and available.

$ ls /usr/lib/execline/
libexecline.a

Any clues about how to solve this?


 ./configure --with-lib=/usr/lib/execline

 Alternatively, build execline with --libdir=/usr/lib so the
static libraries can be found in the default linker path.

 Yeah, it's annoying. It's the result of an initial design decision that
was theoretically logical but that has brought nothing but headaches
since. I don't want to change the default libdir, but I will probably
have to do it at some point because a lot of users are bumping on this.

--
 Laurent



Re: s6-rc : Anomalies or normal behaviour

2020-10-06 Thread Laurent Bercot



 Glad it's working for you!



A significant reduction in complexity.  However, and the reason for my
delay in replying.  Magic happened!  I was now transmitting data which
crossed jail barriers (from b3 "named" to b2 "named logging").  I needed
to consult with one of the FreeBSD developers to ensure that a security
hole wasn't occurring. :)


 Well, that's also what you were doing with your former
b3:named2 and b3:named-log2, except you were transmitting the data via
a named pipe created in your run script explicitly instead of an
anonymous pipe created by s6-rc implicitly. The integrated pipe
feature does not touch your security model at all; if you were to
consult with a FreeBSD developer, you needed to do it before making
the change. :)



It appears (and I'm assuming) that s6 uses pseudo terminal sub-system to
communicate. In this specific case below, per pts/3


 No, s6 does not use pseudo-terminals at all; all it does is let
processes inherit fds from their parent. In your case, /dev/pts/3 seems
to be s6-svscan's stdout and stderr; if you don't want to have
pseudo-terminals, you should check the script that launches your
supervision tree, and redirect s6-svscan's outputs accordingly.

--
 Laurent



Re: Network-online.target

2020-10-05 Thread Laurent Bercot

How to port network-online.target of systemd to S6 ?

Some of the services requires network-online.target to be executed


 Hi Amaresh,
 It's basically impossible to answer this question without understanding
the bigger picture. What are you trying to do?

 systemd relies on the existence of a network manager on a machine
(typically systemd-networkd or NetworkManager); if you're trying to
port a set of services to s6, you will first need to decide what model
you will use for your network, and more generally what set of base
services you need to assume are there, and how you're going to bring
them up. That's the big picture we need to understand.

--
 Laurent



[announce] skarnet.org October 2020 release

2020-10-04 Thread Laurent Bercot



 Hello,

 New versions of some of the skarnet.org packages are available.
This is mostly a bugfix release: bugs have been fixed in every released
package. A .0 release number means that the minor has been bumped
because there are also new features, in which case details are provided
below.

 The new versions are the following:

skalibs-2.9.3.0
execline-2.6.1.1
s6-rc-0.5.2.0
s6-dns-2.3.3.0
s6-networking-2.3.2.0
s6-portable-utils-2.2.3.0
s6-linux-utils-2.5.1.3
mdevd-0.1.2.0
bcnm-0.0.1.1


 New features:

* skalibs-2.9.3.0
  ---

 - New function: stralloc_readyplus_tuned(). The stralloc_readyplus()
macro now rewrites to a call to stralloc_readyplus_tuned(), which
fixes a previously missing overflow check.

 https://skarnet.org/software/skalibs/
 git://git.skarnet.org/skalibs


 * s6-rc-0.5.2.0
   -

 - You can now mark a service as essential, by having a 'flag-essential'
file in the (source) service definition directory. Then, after bringing
that service up, s6-rc will refuse to bring it down, unless it is given
a special -D option; that option should only be used at machine shutdown
time.

 https://skarnet.org/software/s6-rc/
 git://git.skarnet.org/s6-rc


 * s6-dns-2.3.3.0
   --

 - New binary: s6-dnsip. It gives both IPv4 and IPv6 addresses for the
domain given on the command line.

 https://skarnet.org/software/s6-dns/
 git://git.skarnet.org/s6-dns


 * s6-networking-2.3.2.0
   -

 - New -e option to s6-tlsserver. This option ensures that
s6-tcpserver-access is always invoked, even if no other option requires
it, even in the absence of an access control ruleset. This gives the
underlying server program unconditional access to all TCP environment
variables such as TCPLOCALPORT.

 https://skarnet.org/software/s6-networking/
 git://git.skarnet.org/s6-networking


 * s6-portable-utils-2.2.3.0
   -

 - New binary: s6-tai64ndiff. It prints the time differences between
successive timestamped lines, which allows the reader to tell at a
glance how much time has elapsed between two log lines.

 https://skarnet.org/software/s6-portable-utils/
 git://git.skarnet.org/s6-portable-utils


 * mdevd-0.1.2.0
   -

 - New -o option to the mdevd program. This option takes an outputfd
argument, which is a file descriptor number that must already be
open (just like the -D notif option). If this option is given, mdevd
will print to outputfd, verbatim (with possibly a MDEV= property added),
the uevents it receives, *after* processing them. This allows a program
to be notified when the userspace handling of the event has completed.
The libudev-nih (https://git.shadowice.org/mixi/libudev-nih)
implementation of libudev, for instance, can use that mechanism.

 https://skarnet.org/software/mdevd/
 git://git.skarnet.org/mdevd


 Enjoy,
 Bug-reports welcome.

--
 Laurent



Re: s6-rc : Anomalies or normal behaviour

2020-10-03 Thread Laurent Bercot

Apologies, my earlier email, item 2, pointed to emptyenv as the cause of
zombie processes on FreeBSD 12.2S, actually it is due to background.


 Ah, then everything is working as intended and there's no anomaly.
 background spawns a process as a direct child, so if the parent execs
into a long-lived program that never reaps bastards (children it doesn't
know it has), then the zombie will hang around.
 "background -d" was made for this situation, and will avoid the
zombie.

--
 Laurent



Re: s6-rc : Anomalies or normal behaviour

2020-10-03 Thread Laurent Bercot

1. I expected to see the date in seconds since time epoch, but result is
variable name
# execlineb -Pc 'backtick D { date "+%s" } echo $D'
$D


 Normal behaviour, since there's no shell to interpret $D as the
contents of variable D. Try using "importas D D" before the echo:
it will read the value of D and substitute $D with this value, so
echo will print the value. Yeah, execline is annoying like that, it's
just a habit to take.
 Also, you generally want "backtick -n", to chomp the newline at
the end of your input.



---
2. When I use emptyenv within an execlineb script, I have a "defunct"
zombie process
89685  3  S<   0:00.01   |-- s6-supervise base:time-srv
 3020  -  S

The time server script is
#!/usr/local/bin/execlineb -P
emptyenv
multidefine -d " " "base time ntpd /usr/local/sbin/ntpd" { JAIL SERVICE
USER PROGRAM }
background { echo Starting service $SERVICE using $PROGRAM on $JAIL
under user $USER }
fdmove 2 1
redirfd -w 1 /m/base:time/fifo
$PROGRAM -c /etc/ntp.conf -N -g -u $USER --nofork

removing emptyenv, prevents the zombie from being created.  Is this normal?


 The zombie is the echo program in your background block, since it's a
direct child of your run script and there's nothing that reaps it
after it's forked (fdmove, redirfd, ntpd - those programs don't expect
to inherit a child). So the zombie is expected. To prevent that, use
"background -d", which will doublefork your echo program, so it will
be reparented to pid 1 which will reap it properly.

 The anomaly is that you *don't* have that zombie without emptyenv;
my first guess is that there's something in your environment that 
changes

the behaviour of ntpd and makes it reap the zombie somehow.



---
3. Is it normal/standard/good practice to include a dependency in a
bundle.  For example, I have a "time" bundle whose contents are
time-srv.  time-srv starts the ntpd service, and has as a dependency
time-log.

Using "s6-rc -u change time", everything behaves as documented, ie
starts "time" which starts time-log, then time-srv.  However

# s6-rc -v 9 -d change base:time
s6-rc: info: bringing selected services down
s6-rc: info: processing service base:time-srv: stopping
s6-rc: info: service base:time-srv stopped successfully
# Starting logging service time for base with user s6log folder
/var/log/time

and the time-log continues running.


 If you only have time-srv in your 'time' bundle, then time-srv and
time are equivalent. Telling s6-rc to bring down time will do the
exact same thing as telling it to bring down time-srv. time-log is
not impacted. So the behaviour is expected.

 If you want "s6-rc -d change time" to also bring down time-log, then
yes, you should add time-log to the time bundle. Then 'time' will
address both time-srv and time-log.



y 6 seconds  # This is time-srv
up (pid 85131) 6 seconds  # This is time-log,so it
has been restarted


 If you're using a manually created named pipe to transmit data
from time-srv to time-log, that pipe will close when time-srv exits,
and your logger will get EOF and probably exit, which is why it
stopped; but time-log's supervisor has received no instruction that
it should stop, so it will restart it. This is also expected.

 The simplest way of achieving the behaviour you want is s6-rc's
integrated pipeline feature. Get rid of your named pipe and of your
stdout (for time-srv) and stdin (for time-log) redirections; get rid
of your time bundle definition. Then declare time-log as a consumer
for time-srv and time-srv as a producer for time-log. In the
time-log source definition directory, write 'time' into the
pipeline-name file. Then recompile your database.

 This will automatically create a pipe between time-srv and time-log;
the pipe will be held open so it won't close even if one of the
processes exits; and it will automatically create a 'time' bundle
that contains both time-srv and time-log.

 You're on the right track. :)

--
 Laurent



Re: Disable autostart of service

2020-10-01 Thread Laurent Bercot

How to disable the autostart of service during boot up in S6.


 touch ./down

 https://skarnet.org/software/s6/servicedir.html

--
 Laurent



Re: Understanding the syslogd-linux Service Script

2020-09-12 Thread Laurent Bercot

My particular environment uses glibc, so the old version of the
example seems to work well. I omit the syslog service's log directory
and the log lines appear on my container's stdout, which was my
goal.


 If your goal is to have syslog messages appear on your container's
stdout, then you should actually be able to get away with not having
any syslogd service at all, which is always the preferred solution!

 When syslog() can't find a server that listens on /dev/log, it writes
its message to /dev/console. In a container environment, /dev/console
by definition should be your container's stdout. So, not handling
syslog messages in your container at all should naturally make them
fall through to where you want them.

--
 Laurent



Re: Understanding the syslogd-linux Service Script

2020-09-12 Thread Laurent Bercot




The s6-ipcserverd docs specifically state that it expects a bound
and listening SOCK_STREAM socket, and this led me to the `ipc_accept()`
call that the program makes. Out of curiosity, is there another
s6-family program that handles SOCK_DGRAM sockets? Otherwise, I'll
look in to socklog as Laurent suggested.


 There is not, because the UCSPI model - the one that s6-ipcserver
implements - is about handling data *streams*, and forking a child to
handle each separate client (as the venerable inetd does).

 There are workarounds and hacks to handle datagram sequences with a
similar architecture, but it's never perfect: clients have to connect
(which an additional restriction), you lose natural message boundaries,
and you don't have an explicit EOF so the child needs to die on a
timeout - which makes the thing awkward and actually *adds* complexity,
when the whole point of UCSPI is to make every component as simple as
possible.

 When you need to listen to a /dev/log datagram socket, socklog is
definitely the simplest, and correct, approach.

--
 Laurent



Re: s6-man-pages update

2020-09-09 Thread Laurent Bercot

Fair enough. :-) My thought is, to help distros with packaging, i can add tags 
to the s6-man-pages repo to match the corresponding s6 version on which they 
were based. So at the completion of this current process, i could add a 2.9.2.0 
tag (assuming that the current docs on skarnet.org are from the 2.9.2.0 
release). Does that sound reasonable?


 The docs on the website are updated at least at every release, but also
a bit more often: when I have fixes for the documentation, I add them to
the git repo and also to the website. So the website always has the most
up-to-date documentation (barring functionality that has not been
released yet).
 For instance, the current git head, as well as the website docs,
include the fixes to the typos you reported, whereas the v2.9.2.0 git
tag does not include them.

 So I don't know what policy would be best. It would probably be easier
for you to mimic the tags; I can commit to notifying you on git changes
impacting documentation, so you can have a "current" branch mimicking
the state-of-the-art. But it's really up to you, and all in all I think
tags are a good idea. :)

--
 Laurent



Re: s6-man-pages update

2020-09-09 Thread Laurent Bercot

Do you think it would make sense to include the man pages in tarball
releases of s6 software? It would help with packaging efforts across
distros, I think. Only downside would be that, since you dislike the mdoc
format, any changes to the man pages would have to be coordinated and
finished by others before you made a release, if the plan is to keep
everything consistent.


 That downside is simply too big, and also, I don't want to take
responsibility in the official tarballs for content that I don't audit
myself. I don't think it would be a good trade-off, even for users.
(And that does not mean I don't trust Alexis, or other people in the
community, to do good work! If I didn't, we wouldn't even be talking
about this.)

 What limits packaging efforts across distros is not the number of
packages or where and how to find them; if a distribution has a link
to get the man pages source, they will use it and will just be happy
that there are man pages. What limits packaging efforts, currently, is
only the willingness of distributions to understand and use s6.

--
 Laurent



Re: s6-man-pages update

2020-09-09 Thread Laurent Bercot

i've now completed the linting pass for the s6 man pages. The few remaining 
lint issues are either commented in the sources, or aren't actually an issue in 
this context.

As per Laurent's request, i've also added a Makefile to facilitate 
installation; details in the repo README:

https://github.com/flexibeast/s6-man-pages/


 Nice work! If you feel they're ready enough, I can add a link to them
in the s6 main page right away.

 Two comments:
 - The skarnet.org site is accessible in https, and is preferred.
Intra-site links have no default protocol, so they will link in http
if the client uses http, and in https if the client uses https; but
for absolute URLS, it would probably be best to write them as https.
 - You should list yourself in the AUTHORS section: I wrote the content,
but you wrote the man pages.



* links to other skarnet.org documentation which does not yet have  man pages: 
s6-networking (s6-tcpserver-access, s6-tcpserver,  s6-tcpserver4, 
s6-tcpserver6), execline (execline and execlineb,  maybe others as well?), and 
skalibs stuff. For an example of a  page referring to s6-networking software,  
cf. s6-connlimit.1.in. If that documentation was available as  man pages, these 
could just be made cross-references too.


 I don't want to be making suggestions on work I'm not going to do
myself and that I have no deep understanding of; but is there a way to
have an alternative in .Xr, as in "print as a cross-ref if the man
page exists, else print that text"? That would be ideal for placeholders
until the documentation for other packages is ported (which may very
well be "never").



* links to non-skarnet.org sites, such as djb's site. These  certainly seem 
amenable to use of the footnoting style you  described above.


 Yes, I don't think it's reasonable to expect the whole Web to be
converted to man pages. ;)

--
 Laurent



Re: Understanding the syslogd-linux Service Script

2020-09-09 Thread Laurent Bercot

One thing that I forgot to mention here is that depending on the age of
the libc this might not work properly as written. Newer libc's use
datagram connections for syslog and because of that you need to
hand-write the entire s6-ipcserver chain (since the s6-ipcserver
entrypoint doesn't currently support passing the SOCK_DGRAM option to
s6-ipcserver-socketbinder.


 The syslogd-linux example is a pretty old one, I should probably
rewrite it. It will work as is with glibc, because glibc's syslog()
has a SOCK_STREAM fallback, so the s6-ipcserver chain is fine.

 It will *not* work as is with musl, because musl's syslog() only
opens a SOCK_DGRAM socket. In that case, I recommend using socklog
instead: http://smarden.org/socklog/
and the run script would just be: "fdmove -c 2 1 fdclose 0 socklog"
(you lose readiness notification but that's not a big problem for
socklog, which listens to /dev/log very fast, and the worst case is
the first log lines are sent to /dev/console)

 Ultimately the syslogd service just needs to listen to the /dev/log
socket (as SOCK_DGRAM, as socklog does, or SOCK_STREAM, as s6-ipcserver
does) and translate the loglevel/priority in the log lines to human-
readable text and print them to stdout (as socklog or ucspilogd does).
The real work is done by the logger part, which is a s6-log process
that reads the translated log lines, matches them against some regexes
and logs them to various logdirs depending on what they match. The
s6-log script is the equivalent of syslogd.conf: that's where you
configure how to dispatch the syslog into different places.

--
 Laurent



Re: Update early logger logging path after remounting root as rw

2020-09-05 Thread Laurent Bercot




I'm configuring my linux to use s6-rc. It works fairly well so far. One
thing I want to improve though, is that the early logger logs to
/run/uncaught-logs. It's nice to have a log file during early boot and it
helped my debugging easier. But it would be cool to able to change the
location to a permanent location like /var/log after the root has been
remounted as read-write.


 Well the catch-all logger is supposed to be just that: a catch-all
logger. That means that ideally, every service should have its own
dedicated logger, typically writing to /var/log/something, and the
catch-all logger is only used for exceptional stuff such as error
messages from the supervision tree, so it doesn't matter that its logs
are stored under /run.



Is it possible to update the log path of early/catch-all s6-log process to
a new location, and perhaps copying the early logs there as well? Or if
not, is it possible to spawn a new s6-log process that acts as a catch-all
logger?


 It's difficult to do. It's possible in theory: you could have a oneshot
that modifies /run/service/s6-svscan-log/run, replacing the
"/run/uncaught-logs" string with the new location you want, then
copies /run/uncaught-logs into that new location and restarting the
s6-svscan-log service. But in practice you would add complexity to the
log infrastructure, and you need to pay close attention to all the
failure cases because you don't want to have a broken catch-all logger
for any reason at all. I don't think the benefits are worth the
additional effort; but feel free to disagree and write such a "log
mover" service.



BTW, is there a command to restart service managed by s6-rc? I've been
using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there
might be something simpler.


 For oneshots, no - but you rarely want to "restart" oneshots.
 For longruns, unless you want to also restart everything that depends
on the service, you can just bypass the s6-rc layer and directly tell s6
to restart your process: s6-svc -r /run/service/$sv

 Hope this helps,

--
 Laurent



Re: possible s6-rc redesign

2020-09-02 Thread Laurent Bercot

1. In order to provide the -sane- functionality of systemd, a system
needs to use linux-only interfaces. I am absolutely not saying that cgroups,
namespaces etc should be tightly integrated to a service manager/init system.
Most of the work can be done on dedicated programs but they are needed
if the goal is feature-parity...


 Yes, that is very true, and a concern I've increasingly had: OSes are
making *zero* coordination efforts for their interfaces, and software
is increasingly relying on OS-specific interfaces. So it's becoming more
and more difficult to write for Unix at large, and market fragmentation
is becoming more obvious by the day.

 For the various engines of an init system, though, I'm not too worried.
 - The early boot is inherently not portable (s6-*linux*-init is called
that way for a reason), but it's not a big or complex piece of code,
and it's easy to adapt it to another OS as soon as one has a working
knowledge of the details of init mechanisms on those OSes.
 - The supervision engine is portable, with a caveat.
 - The service manager is portable, even with dynamic event integration.
What is *not* portable is the various listeners that produce events,
but hooking a s6-rc-event call to them should not be difficult.

 The caveat for the supervision engine is, as you mentioned, cgroups
and namespace management, for the sole reason that the Linux way of
starting a process in a new namespace is contrary to the venerable Unix
pattern of changing your own process state then exec'ing: in some cases
you need involvement from the parent in changing the child's state.
There is a patch that has been submitted to this list that does exactly
that: tell s6-supervise to start the service in a new namespace. I have
not applied this patch because I didn't want to introduce system-
dependent features into s6, but with the way OSes are diverging, I might
soon have to bite the bullet and change my stance on this.

 But apart from that misdesign from the Linux kernel, most OS-specific
stuff can actually be done from inside services, in run scripts for
instance, so engines can be kept portable. If OS-specific tools are
needed to easily write service scripts, then those tools can be provided
as a separate, OS-specific package - like an extension.

 In other words: policy is definitely not portable and needs a lot of
extensions, but mechanism generally is, and I'm always starting with
mechanism - I'll address Linux policy later.



2. Distributions that use systemd are slowly depending more and more on
these tendrils (great analogy btw :P). I do not think they will make the effort
to get rid of them or provide alternate mechanisms to facilitate inclusion
of a different init system no matter how much better, cleaner and well-
designed it is. And s6-rc is all three of them IMHO. The only way to create
pressure is a successful set of distributions that use other solutions...


 Yes, but do not underestimate the general efforts that are being done
to have functionality outside of systemd. We're not the only bastion of
opposition here :) Alternate mechanisms do exist for most of what 
systemd

does; distributions mostly migrate to the systemd way of doing things
because it's more *convenient* to get the functionality from a single
source than it is to piece it together from various bits. systemd is
thriving on distribution laziness.
 But a distribution that is committed to not using systemd is (still)
very much able to offer a lot of functionality, it just requires
somewhat more effort.

 I don't expect to make it into Ubuntu or Red Hat right away. But if
a lot of other distros out there are running a credible alternative with
equivalent functionality, then at least users will have choice, and it
will be impossible for systemd advocates to pretend that their chosen
init system is inevitable. (insert Avengers: Endgame meme here)



I am not saying that policy should not be provided at all. I am saying that 
they are
 advantages if it is provided outside of the main project and downstreams 
tailor it
to their specs.


 Possibly; that is honestly a packaging problem. It's a balance to find
between flexibility and convenience. systemd and OpenRC have played the
convenience card to the max and it has worked well for them; s6 has
always played the flexibility card instead, and experts love that, but
most people don't, especially when *no* policy is provided. :)

 If a package provides a set of init scripts, distributions will tailor
it anyway. The RedHat /lib/systemd is not exactly the same as the Ubuntu
one. The Alpine OpenRC scripts are not exactly the same as the Gentoo
ones. It does not really matter where the policy is provided, but
distros have made it very clear to me that they prefer having a working
set to start with and to tailor to their needs, rather than having 
nothing

and needing to come up with everything themselves. If it's better for
s6-frontend to be pure mechanism with a s6-frontend-policy package 

Re: possible s6-rc redesign

2020-09-01 Thread Laurent Bercot

1. There is one **huge** disadvantage for the second option. It is not a
technical one, so it may carry little weight on your final decision.
There already different implementations of init systems based on s6/s6-rc,
some of them successfully used on distributions. They will almost certainly
have to be abandoned or rewritten to function with the redesigned suite.
The first option will allow these implementations to augment what they offer
with minimal or no disruption for their users and developers.


 You may have missed the part where I'm saying that the current version
of s6-rc will still be supported for a *long time*. I would also try,
obviously, to make the conversion to the new format as painless as
possible; ideally (but I don't know yet whether it's an attainable 
goal),

the source directories would have the exact same format as long as only
static services are defined - so tools using s6-rc-0.x would still work
with basically no changes.



2. Which distributions or groups of distributions will find the redesign 
appealing, so that they will adopt it ?
IMHO distributions that use systemd are not likely to change in the foreseeable 
future,


 And they definitely won't, if nothing challenges systemd. But
providing the same level of functionality as systemd (as far as the
init system is concerned, I'm not talking about all the tendrils it 
sends
into various parts of running a Linux machine) with a better design, 
that

is not completely alien to what those distros are used to, is the only
way to - someday - have a chance at challenging systemd.



 so that leaves the distributions that use something else. Some of them
already have s6-based solutions (official or not), so adopting the redesigned
will be akin to adding another init for them.


 No, absolutely not. The idea is to have something similar to s6-rc, but
with more functionality integrated in it. It would not be a different
init by any stretch of the imagination.



3. Yes, the compiled db of services or the many files in the servicedir can
discourage people to use s6-rc. But ANY new format will face similar
criticism and some of the problems can be alleviated with the proper
frontend. I use 66. Ι never touch servicedirs - it uses ini-based "frontend
service files"- and I have dabbled with the compiled once, indirectly.


 I agree with that - except that the compiled db is actually a very
small hurdle, that goes away once you explain that OpenRC and systemd
also use a compiled db except that it's hidden and everything is done
dynamically, and s6-rc only makes it cleaner and more upfront.

 Yes, the solution is a user-friendly interface, but that's not
relevant to what I'm talking about. What I'm saying is that 
distributions,

including ones that currently use s6/s6-rc under 66 or any other
interface, *will* need dynamic events integration to the service manager
at some point, and the question is, how to add the functionality to
s6-rc. I think Option 2 is *more* likely to get adopted by
distributions overall. Distros that have already drunk the s6-rc
kool-aid will be fine with either option 1 or option 2 (see above);
normie distros will very probably be repulsed by option 1.



4.  On the issue of mechanisms vs policy I fully agree that people want
a turnkey solution. However I do not think that is something that should
be done in a cetralised manner. There already at least 3 different working
sets of services created for s6-rc (artix, slew, 66) and someone can inspect,
use and modify them to fit their needs or policies.


 Yes, people can pick and choose what they want, and as it is now, 
things

are working for power users.
 That is still niche though, and in order to reach more people, a wide
array of distributions is the way to go.
 I'm very happy that distributions such as Obarun and Artix have done
the effort of actually porting services to a new format, be it 66
or raw s6-rc; but it is not something that can be expected from most
distributions, especially the big ones, which I *will* be aiming for at
some point. I don't like the idea of a software author providing policy,
any more than you do, but it is what worked for systemd and OpenRC, and
in case you haven't noticed, I'm trying to be very pragmatic here, and
to make the necessary sacrifices of principles in order to gain wider
adoption.

--
 Laurent



Re: [request for review] Port of s6 documentation to mdoc(7)

2020-09-01 Thread Laurent Bercot


* We negotiate a HTML schema your documentation can be written in, which
  is based on current documentation; existing HTML documentation will be
  converted to the schema, with minimised changes.  (The structuredness
  of HTML helps; now you also see why knowing some Lisp is beneficial :)


 I'm totally willing to use a HTML schema we can negotiate, to write
future documentation in. What I don't want to do is:
 - Touch existing documentation, unless I have to rewrite the content of
a page for some reason. Of course, if the conversion work is done by
somebody else, I have no objection to upstreaming the new documents.
 - Add heavy dependencies to the skarnet.org Makefiles.

 But honestly, if I'm going to change the way I write doc, I'd rather
write scdoc, which is simpler than plain HTML especially if we're
adding a lot of semantic tags to that HTML. To me, the best thing
would be if someone would add a HTML backend to scdoc. I may do it
myself if I need a break from interesting service management stuff and
experience an irresistible urge to work on HTML generation instead.
The odds are what they are.

--
 Laurent



possible s6-rc redesign (was: [request for review] Port of s6 documentation to mdoc(7))

2020-09-01 Thread Laurent Bercot

I have only seen one new feature committed to the Git repository so
far. Is it too early to ask what are you planning to change?


 The new feature is orthogonal - or, rather, it will be used if I end
up *not* redesigning s6-rc.

 The trend with distributions is to make service managers reactive to
external events: typically NetworkManager and systemd-networkd because
the network is the primary source of dynamic events, but even local
events such as the ones produced by a device manager, or basically
anything sent by the kernel on the netlink, are getting integrated
into that model.

 s6-rc is, by essence, static: the set of services is known in advance,
and there is no reacting to external events - there is only the admin
starting and stopping services. This has advantages - a compile-time
analysis is possible, with early cycle detection, etc.; but the model
doesn't integrate well with modern distro needs. So, I've been thinking
about ways to add dynamic event management to s6-rc; and I've found
two options.

 Option 1 is to add dynamic event management *on top of* s6-rc. That
is my natural instinct; that is what I've always done with software,
that's what keeps the various parts of my software as clean and simple
as possible. Here, it would mean:
 - having a classic s6-rc database for "static" services
 - having an additional "dynamic" database for services that can be
triggered by external events. (The database is static in essence, but
I call it "dynamic" because it would host the services that can be
started dynamically.)
 - having a s6-rc-eventd daemon listening to events and executing
s6-rc commands on the dynamic database depending on the events it
receives. Paired with a s6-rc-event program that sends events to
s6-rc-eventd, meant to be invoked in situations such as udevd/mdevd
rules, a netlink listener, etc.

 This model works in my head, the s6-rc-event[d] programs would be quite
simple to write, it would solve the problem in a modular way just like
the skarnet.org usual, so it seems like a no-brainer.
 Except for one thing: I don't think anybody would use this. Only me,
you, and the other 6 hardcore people in the world who actually like
this kind of design.

 If there's one thing that has been made painfully obvious to me these
past few years, it is that most people, and especially most *distro*
people - which are the ones I would like to reach -, perceive the s6
stack as very complex. They're intimidated by it; they find the
abundance of moving parts off-putting and difficult to get into.
With very few exceptions, the people who actually take the plunge and
make the time and energy investment necessary to understand the model,
what the various parts do and how they fit together, those people all
love it, and are very enthusiastic about it, and they're what keeps me
going. But the initial barrier of potential, the ultra-steep learning
curve, is indisputably the limiting factor in the spread of the s6
ecosystem.

 s6 as a supervision suite? okay, people will use it; but it's already
perceived as a bit complex, because there are a lot of binaries.
It's on the high end of the acceptable difficulty range.

 s6 as an init system? "what is this s6-linux-init thing? why do I need
this? runit is simpler, I'll stick to runit." Even though runit has
problems, has less functionality, and is barely maintained. There are,
for instance, several people in Void Linux who are interested in
switching to s6, but despite s6 being an almost drop-in replacement for
runit, the switch has not been made, because it requires learning s6 and
s6-linux-init, and most Void people do not feel the effort is worth it.

 s6-rc? "waah I don't like the source directory format, I want text
files, and why is it so different from 'service foo start'? And why
doesn't it come with integrated policy like OpenRC or systemd?" People
understand the benefit in separating mechanism from policy, in theory,
but in practice nobody wants to write policy. (Can't blame them: I find
it super boring, too.) Having the tool is not enough; it needs to be
gift-wrapped as well, it needs to be nice to use.

 If I add a s6-rc-event family of binaries to s6-rc, the fact that it
is yet another layer of functionality, that you now need *two*
databases, etc., will make a whole additional category of people just
give up. The outreach will be, mark my words, *zero*. If not negative.

 The fact is that a full-featured init system *is* a complex beast, and
the s6 stack does nothing more than is strictly needed, but it exposes
all the tools, all the entrails, all the workings of the system, and
that is a lot for non-specialists to handle. Integrated init systems,
such as systemd, are significantly *more* complex than the s6 stack, but
they do a better job of *hiding* the complexity, and presenting a
relatively simple interface. That is why, despite being technically
inferior (on numerous metrics: bugs, length of code paths, resource
consumption, actual 

  1   2   3   4   5   >