Re[2]: s6/s6-rc policy for Gentoo: config files for service scripts
The best would probably be for s6-rc-compile to allow for multiple definitions of a service, letting later definitions override earlier ones, e.g. s6-rc-compile ${OUTPUT_DB} ${SOURCE_1} ${SOURCE_2} where seatd-srv in ${SOURCE_2} overrides seatd-srv in ${SOURCE_1}. Would this be realizable Laurent? Overrides? No, this would make things confusing and error-prone. The behaviour of a set of source definition directories should not be influenced by whether or not you're compiling it with another set. Organizing services is something I don't have a lot of policy advice on. There's a balance to be found between ease of use and flexibility; a sound approach seems to be to have the service itself hardcoded in the system set of source definition directories (handled by the package manager) but keep the run script itself pretty configurable via environment definitions sourced from user-modifiable files in /etc. It's not the only possible approach, but it works for most services. -- Laurent
Re: notification-fd inadvertently closed by selfpipe_finish on FreeBSD?
Is this a bug, or am I holding it wrong? It is a bug indeed! Thanks for the report and the detailed analysis. I'll fix it asap. So maybe skalibs' cspawn... ...calls cspawn_fork() on FreeBSD if CSPAWN_FLAGS_SETSID is set. ...calls cspawn_pspawn() on macOS under the same conditions. There is no call to selfpipe_finish() in cspawn_pspawn. Yes. selfpipe_finish() must not be called in cspawn_pspawn() since only the parent process is running; the child process is only controlled by the posix_spawn attributes. With cspawn_fork(), selfpipe_finish() runs in the child. (I guess because of posix_spawnattr_setsigmask?) The signal mask must be restored before the exec in both cases. It's the sigmask attribute for posix_spawn(), and selfpipe_finish() for fork(). But all that is only where the problem manifests. The real issue happens much earlier: when notification-fd is read and accepted without vetting that it's not clobbering the selfpipe. This is what I'll fix. -- Laurent
Re: s6-socklog with -i ipport option
Yeah, sorry for not getting back to you earlier. This week has been... something. Problem reproduced and confirmed, and your fix is a good one, I applied something similar (fmt+m instead of &fmt[m], just personal preference). Thanks! The release is probably not coming soon because I'm working on something and would like to keep things in sync, but if there are enough bugs that a bugfix release is warranted I can cut one this month. -- Laurent
Re: [PATCH] Resolve PROTOREMOTEHOST or PROTOLOCALHOST when only one is specified
Previously, specification of PROTOLOCALHOST on the command line with the -l option would lead to PROTOREMOTEHOST not being resolved. This did not seem to be the intended behaviour, especially as the call to s6dns_resolven_parse_g works with any combination of the state of previous hostname definitions. This change alters the conditional to conduct the necessary resolutions in all cases. I apologize if I misinterpreted or was mistaken about anything. Your diagnosis is correct! Indeed, the call to s6dns_resolven_parse_g must happen when at least one of (localname, remotelen) is 0, which means they haven't yet been resolved another way. +if (!localname || !remotelen && !s6dns_resolven_parse_g(blob + !!localname, !localname + !remotelen, &infinite)) This, however, is incorrect: && has precedence over ||, so the call would not happen when localname is 0. You want parentheses around that ||. :) I've pushed a fix. Thanks for the report! s6-tcpserver-access.c is more spaghetti and difficult to read than normally would be warranted for a program that should be simple on paper, and I'm not surprised it's not exempt of bugs. Unfortunately, it's in the hot path of new TCP connections from clients, and it performs network resolutions such as DNS, so it needs to be optimized for low latency prior to any other consideration. Thus, it's a maze of tests and DNS queries are only performed when there's no other option, and always in parallel when there are several of them. That's why it looks like a mess with variables and conditions everywhere which is a good breeding spot for the kind of bug you found. -- Laurent
Re: Possible to shut down an s6 service via command rather than signal?
From a similar Github issue [1], the suggested way of implementing "stop commands" on top of s6 is running these two commands: ``` s6-svc -O /path/to/service stop-command ``` I'm not aware of any s6-rc sugar around it. Indeed. Every long-lived program should have a graceful stop signal, instead of relying on an external command. That's how Unix has always worked, and it's a misdesign of qemu not to support that. Sorry that the s6 way of handling shutdown commands is kludgy, but s6 really wasn't designed to accommodate them. :/ -- Laurent
Re: s6/s6-rc policy for Gentoo: XDG Base Directory Specification
Is there an equivalent chainloader ("envfile"-style) that can be plugged into the instanced user service directories' command line? Having the same configuration file be read by tools with different behaviors (for instance, it appears the pam_env's quoting support is far more limited than envfile's) may not lead to the same results. I wanted envfile to be compatible with intuitive and expected behaviour. If you write your files conservatively wrt what other tools accept, you should be able to read them with envfile. -- Laurent
Re: s6/s6-rc policy for Gentoo: user session tracking
That said, it might be of interest to you, that s6-setlock does not appear here: https://skarnet.org/software/s6/index.html. Whoops. And nobody noticed that in 13 years? Shame on all of you. :P Fixed, thanks! -- Laurent
Re: s6/s6-rc policy for Gentoo: user session tracking
Good point. If I understood everything correctly, this can be easily solved by the "flock" command, right? Yes - but since you're using s6 anyway, you also have s6-setlock, which features more options than flock. ;) -- Laurent
Re: s6/s6-rc policy for Gentoo: logging settings
Should every s6-log daemon have it's own config file or shall they all share one? What kind of stuff are you putting in the config file? Is it just a template for a logging script? Is there any general issue with the global env's from s6-linux-init's env directory, which makes sourcing the same config file in every s6-log script a better choice for a global config? The global environment defined in s6-linux-init applies to *all* processes in the system. It's not nice to pollute it with stuff that would only apply to a pretty restricted subset of all the processes on the machine. Source your config file in every logger run script you have, and leave the global environment alone. There will be no performance issue. For more configurability, you could e.g. have a default logging template hardcoded in your run scripts, that you could override via a service- specific configuration file. -- Laurent
Re: s6/s6-rc policy for Gentoo
Wouldn't anything this does be solved by just dropping privileges of the corresponding daemons in their startup scripts? No, because the point is to allow users to have their own services, and you definitely don't want them to have access to the root supervision tree. So you basically have to have a separate subtree for each user (a privilege-dropped s6-svscan), and there are a few more policy questions to answer, in particular how to make sure the users cannot flood the root logs by misconfiguring their own logging system. It's doable, but not quite obvious, and it's also not quite obvious how to set up the system so that you don't spawn a supervision tree for inactive users or users who don't use the feature. So there's a little more work involved. This seems like a worse and less flexible version of B to me, I guess the only reason it is even considered is because the seat daemon does all the tracking work for you. Option B does all this does and more if I understand you correctly, thus I think B is straight up the better way. It really depends on your objective. B makes a lot of sense if you want to treat remote logins the same way as seated logins, but there is also an argument to have remote logins be as simple as possible while not minding deploying the red carpet and a whole lot of extra services for seated logins, where the overhead of setting everything up will be negligible compared to the desktop stuff anyway. Does anyone see any other problem arising from B? B is entirely legit if you make it clear that's what users get. Just to make sure users don't run xeyes when logging in remotely. And you probably want an extra supervision tree just for seated stuff anyway, see below. A) The format, do you suggest a FILENAME=content which seems easier to handle with execline and other software's(e.g. dbus) ability to write specific info to an fd which can be redirected to a file? Or do you think a file containing KEY=value, probably playing more nicely with bash and other interactive shells? The format doesn't matter if you control the readers. s6-envdir reads envdirs, envfile reads key=value shell-like files. Choose whatever format works best for you; if you need the variables to be read from a shell as well, the envfile format will work best. It's a question of convenience and preference, really. B) I think I need to make an example to explain this issue properly: I use Hyprland(https://hyprland.org/), which I start from an agetty instance. Hyprland, as any other wayland compositor/DE, sets env's. Most importantly WAYLAND_DISPLAY. It does not write those to a file. Thus I would require the user to add a script launch command to Hyprland's config which launches a script exporting the required(or all?) env's to a dir. This is unacceptable. That means WAYLAND_DISPLAY, by design, is only be accessible to processes that descend from Hyprland. Why do you need to subvert it so unrelated processes can access it? When Hyprland isn't running, there shouldn't be any living processes relying on WAYLAND_DISPLAY. That suggests that you want a separate supervision tree, launched under Hyprland, to host processes that only make sense for a seated login - just like the C case above. I don't think that any hack to get the value of WAYLAND_DISPLAY and store it for access by unrelated processes makes sense. Supervision trees are cool, but they shouldn't be used to bypass normal process hierarchies. Write down exactly what your services' lifetime should be and design accordingly. Thank you for the info, I have the greatest respect for you writing only UNIX compatible software(if that's how to correctly describe it?). I am looking forward to your alternative to PAM. Don't hold your breath - this is a longterm plan, and a considerable amount of work. -- Laurent
Re: s6/s6-rc policy for Gentoo: The logging directory
Concerning the system tree loggers, my first idea is to set an environment variable somewhere in the scripts s6-linux-init provides. Is this even possible? What would be the best place to do this? Any environment variable you want to make available for a whole supervision tree should be added via a "-e var=value" option to your s6-linux-init-maker invocation. For user supervision trees, I suppose you're not using s6-l-i-m, so the best place would be to put the variables in an envdir and use "-E $envdir" (and possibly "-e $var" if you need to substitute $USER or something) in your s6-usertree-maker invocation. If you're not using these tools, the best way is to do the same thing they do: put the variables in an envdir somewhere, and source the envdir both before running s6-svscan and before running your s6-rc-init and s6-rc script. -- Laurent
Re: s6/s6-rc policy for Gentoo
1) A service monitoring logged in users, starting the user supervision tree on the first login of a user and stopping it once the user logs out from the last shell/display manager Ah, user services. They're a complicated matter, and their exact definition varies depending on the distro. They're very much tied to distro policies, which is why I haven't written support for them in s6, apart from s6-usertree-maker that may or may not be useful to you. When you say "user services", it generally means one of these things: A. a permanent supervision tree, belonging to a normal user, that runs services as the user, independent from whether the user is logged in. B. A supervision tree that starts with the first login and ends with the last logout. This appears to be what you're working on. C. A supervision tree that is tied to a graphical session on one seat. That would exclude remote logins via ssh. This seems to be what the Freedesktop people have in mind when they talk about XDG specifications. You're not the only one working on s6-rc user services for Gentoo: see https://skarnet.org/lists/supervision/3074.html :) You should definitely try and coordinate with Alexis! And also with other people who hang on the #s6 channel on oftc.net and may have more ideas to share. 2) An elegant way of synchronizing environment variables between the users supervision tree and its services and the users login shell/compositor/... Generally speaking, when I have a set of environment variables that need to be shared between unrelated sets of processes, I put them in an envdir, and source the envdir whenever the variables are needed. One way or the other, the only reliable data sharing tool you have across process trees is the filesystem. Regarding 1) the best idea I currently is to have a daemon monitoring /var/run/utmp, starting an sub-process/s6-rc service for each user appearing there and stopping this sub-process/s6-rc service once the last login instance of the user vanished from that file. Since I have very little programming experience, I wanted to as you, Bercot, whether you consider this a good idea and if you do, whether this would be worth a s6 program. Can this be achieved in a better way? Since you're going with option B, indeed you'll need something to detect logins, but relying on utmp is a risky business, and you don't want to interface with PAM yourself. Having a third-party program dedicated to that is probably the right approach; Mobin mentioned turnstile, this looks like a promising option. I'm not going to write such a program in the s6 suite, because all the stuff around PAM and utmp is 1. very Linux-specific and 2. extremely brittle and badly/underspecified. At some point, however, I intend to take a good look at PAM and brainstorm about a replacement, because the current situation is just holding with massive amounts of duct tape, good will and headaches, and this makes me fearful for the future. In any case, be wary of the trap of using XDG_RUNTIME_DIR to host your user supervision tree. It *looks* like the ideal place to do so, but in case of a desktop session, it's deleted too early and you don't have time to properly terminate your services. Other people who have experience with running user services on a desktop will talk about this better than me. -- Laurent
Re: [PATCH·s6-linux-init] Remove error message when rc.shutdown.final script is not found
- EXECLINE_EXTBINPREFIX "foreground { ") < 0 + EXECLINE_EXTBINPREFIX "foreground { tryexec {") < 0 Good idea. Applied, thanks! -- Laurent
[announce] skarnet.org June 2024 update
Hello, New versions of some skarnet.org packages are available. This is a light update, focused on quality of life, improved support for old platforms, and in preparation for larger updates later. The exception is a breaking change to s6, which adds support for addressing a service's process group - useful e.g. if you want to kill straggler processes when the service dies and you're not using cgroups. skalibs-2.14.2.0 (minor) execline-2.9.6.0 (minor) s6-2.13.0.0 (major) s6-rc-0.5.4.3 (release) s6-dns-2.3.7.2(release) s6-networking-2.7.0.3 (release) mdevd-0.1.6.4 (release) smtpd-starttls-proxy-0.0.1.4 (release) tipidee-0.0.5.0 (minor) shibari-0.0.1.1 (release) Details below. If there are no details for a package, the release only contains bugfixes and/or code cleanups. * skalibs-2.14.2.0 This version underlies all the changes in the other packages. It brings support for the midipix environment and old MacOS versions; it adds more cspawn functions for improved posix_spawn() use. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * execline-2.9.6.0 This version adds "elglob -d" for encoding the result of a glob into a single word. It also adds "importas -S" to import a variable with the same name as the substitution value, just like the old "import" command. https://skarnet.org/software/execline/ git://git.skarnet.org/execline * s6-2.13.0.0 --- s6-supervise now tracks the service's process group! It is passed to the finish script as the 4th argument, so finish scripts can kill the whole process group if the service leaves unwanted children behind. Note that this isn't completely foolproof: a very badly behaved service can create children in different process groups. But daemons don't normally mess with process groups; and any alternative is non-portable. While the service is running, s6-svstat -o pgid prints the process group id, and s6-svc -K can send a SIGKILL to the process group (as well as -P for SIGSTOP and -C for SIGCONT). Additionally, a small bug has been fixed and an arbitrary limit has been lifted in s6-ftrigrd: a service can now wait on as many fifodirs as it wants. Your supervision trees need to be restarted after you upgrade to the new version. https://skarnet.org/software/s6/ git://git.skarnet.org/s6 * s6-rc-0.5.4.3 - An arbitrary limit has been lifted: the internal fd-holder can now be autorefilled with as many pipes as it can hold. https://skarnet.org/software/s6-rc/ git://git.skarnet.org/s6-rc * s6-dns-2.3.7.2 -- Refactor of some APIs to allow for dns-0x20 implementation, which is useful for interoperating with recent versions of Unbound and similarly paranoid software. https://skarnet.org/software/s6-dns/ git://git.skarnet.org/s6-dns * tipidee-0.0.5.0 --- tipidee.conf now accepts "" as an extension indicator for the content-type directive, which allows the user to specify a default Content-Type for files without an extension (previously, the application/octet-stream default was not overridable). https://skarnet.org/software/tipidee/ git://git.skarnet.org/tipidee * shibari-0.0.1.1 --- shibari now implements dns-0x20, which means the latest Unbound resolvers won't fail when querying it. https://skarnet.org/software/shibari/ git://git.skarnet.org/shibari Enjoy, Bug-reports welcome. -- Laurent
Re: s6 daemon restart feature enhancement suggestion
Let me say that a $daemon i.e. wpa_supplicant or iwd providing $service=WiFi{wpa} has been pulled into the s6-rc compiled db and started in the supervision tree. But the system doesn't have the hardware to support that, or some important resource is unavailable. So, here's my question: if the system doesn't have the hardware to support that, why is the daemon in the database in the first place? s6-rc, in its current incarnation, is very static when it comes to its service database; this is by design. The point is that when you have a compiled service database, you know what's in there, you know what it does, and you know what services will be running when you boot your system. Adding dynamism goes against that design. I understand the value of flexibility (this is why most distributions won't use s6-rc as is: they need more flexibility in their service manager) but there's a trade-off with reliability, and s6-rc weighs heavily on the reliability side. If you are building a distribution aimed at supporting several kinds of hardware, I suggest adding flexibility at the *source database* level, and building the compiled database at system configuration time (or, in extreme cases, at boot time, though I do not recommend that if you can avoid it, since you lose the static bootability guarantee). If your machine can't run wpa_supplicant, then the service manager should not attempt to run wpa_supplicant in the first place, so the wpa_supplicant service should not appear in the top bundle. Lacking resources is a different issue: it's a temporary error, and it makes sense for the service to fail (and be restarted) if it cannot reserve the resources it needs. If you want to report permanent failure, and stop trying to bring the service up, after a certain amount of time, you can write a timeout-up file, or have a finish script exit 125, see below. A mechanism should be prepared, to let $daemon inform it's instance of s6-supervise that it can't run, or can't provide $service / it's services. If you have the information before the machine boots, you should use the information to prune your service database, and compile a database that you know will work with your system. If you don't have the information before the machine boots, then a service failing to start is a normal temporary failure, and s6 will attempt to restart the service until it reports permanent failure. You have several ways of marking a service as permanently failed: - (only with s6-rc) you can have a timeout-up file, see https://skarnet.org/software/s6-rc/s6-rc-compile.html and look for "timeout-up" - (generic s6) you can have a finish script that uses data that has been collected by s6-supervise to determine whether a permanent failure should be reported or not. A finish script can report permanent failure by exiting 125. For instance, using s6-permafailon, see https://skarnet.org/software/s6/s6-permafailon.html , allows you to tell s6 that if the service exits nonzero too many times in a given number of seconds, then it's hopeless. Does this help? -- Laurent
Re: Building lh-bootstrap-master
Should the lh-bootstrap README be changed so that: Yes, I updated the README.md, good point.
Re: Building lh-bootstrap-master
Indeed, my version of make doesn't have the --jobserver-style option. If I remove the reference to --jobserver-style option from ./make the package builds without any obvious errors but doesn't appear to be a complete build. Yeah, for some reason parallel builds weren't fully working before this option (I knew more when I debugged this but I've since forgotten). I don't think I realized the option was so recent. As Alexis says, your best choice is to use GNU Make 4.4, but if you cannot, the workaround is not to use any -j option to make, which should eventually, very slowly, get you a complete build. -- Laurent
Re: Sample Configuration
It is a very elegant implementation of s6 Vocabulary nitpick: no, 66 is not an "implementation of s6". s6 is not a specification. It is a software package. So, there is no other "implementation of s6" than the s6 software package itself. 66 is a service manager than runs on top of the s6 process supervision suite. It is an implementation of the "service manager" concept. It uses the s6 interface. Just making sure we're using the correct terms. It helps prevent misunderstandings. :) -- Laurent
Re: Sample Configuration
was probably going to use a simple runit configuration because the s6 universe seemed to complex for me to figure out in a reasonable amount of time. I haven't A basic s6 system is barely more complex than a runit system. This page should help you start: https://skarnet.org/software/s6-linux-init/quickstart.html HTH, -- Laurent
Re: Sample Configuration
I'm building a Linux system based on musl and Busybox. I'm considering using s6/s6-rc as an init/supervision system. I see there are good docs on all the skanet.org programs, but I can't find an example of a working configuration that puts them together. Is there an example available somewhere? In addition to everything that has been said, you could check out lh-bootstrap. https://github.com/skarnet/lh-bootstrap It's a tool I use to build a complete VM using musl, busybox and skarnet.org utilities from scratch, so I can test it under qemu. The layout of the filesystem there should give you an idea of how I intend the whole thing to be used. ;) -- Laurent
Re: s6-rc user services on Gentoo
I have been using `--print-pid=3` as readiness notification for dbus-daemon for quite a while now on my user services and I haven't had any problems with it so far. IIRC, looking at dbus-daemon code, it actually prints the socket address first then its pid. So, I use the `--print-address=` option to save the socket address to a file for consumption by s6-envdir. Interesting. But it's also an implementation detail, because dbus-daemon could print its pid at any time, no matter its readiness state. That's why I chose to go with --print-address - but of course reading the code instead of speculating is the right thing to do :) -- Laurent
Re: s6-rc user services on Gentoo
But then, there is a problem if one actually wants the server address information that --print-address provides. Alexis' 'run' script for example wants to save that to a file (apparently in a directory suitable for s6-envdir). If the output is sent to the notification pipe instead, s6-supervise will 'eat' it while waiting for the final newline character, and then the information is lost. That is true. The option is only usable as readiness notification if you don't otherwise need the information. And more generally, there's also the question about how 'ready' dbus-daemon actually is when the point in its code that prints the server address is reached. I can't really say without looking at the code; dbus-daemon has many duties once its UNIX domain socket is bound. That doesn't matter. The important thing with readiness is that clients need to be able to send messages to the daemon. As soon as the socket is bound, it's okay; if the daemon hasn't finished initializing yet, then that's what kernel socket buffers are for. Of course, if the daemon still needs to allocate resources and can *fail* after binding the socket and printing the line, that's not good, but well, it's a balance between reliability and simplicity. -- Laurent
Re: s6-rc user services on Gentoo
2) The presence of a notification-fd file tells s6 that dbus-daemon can be somehow coerced into producing an s6-style readiness notification using file descriptor 3 without changing its code, are you sure that's the case with this script? My service definition for the system-wide message bus polls for readiness using s6-notifyoncheck and a dbus-send command... "dbus-daemon --print-address=3" should produce a suitable notification. The address of the bus can only be printed once the bus exists. :) -- Laurent
Re: s6-rc: dependencies across scandirs
Hi, I would like to have some kind of dependency handling across multiple scandirs. For example an instanced service that starts and waits for a dependency before the first instace has been started. It's generally a bad idea to add, in a service's run script itself, conditions for your service to work, or to restart. It tends to make your service *less* reliable, instead of *more*. Supervision is about keeping the services up, not monitoring when they should or should not run. The latter is for service management. Instead, design your services so they exit if they have unmet hard dependencies. The supervisor will try to bring them up again and again, and it will eventually work, when the dependencies are resolved. The job of avoiding too much looping, again, belongs to a service manager. A more complicated variation I would like to have is a user service depending on a system service (and starting it!). For example a user service for sway depends on the system service for seatd. Start your sway. If seatd isn't up, sway will exit, and the supervisor will restart it. This works as intended. Now, start propagation is a different question, and a more difficult one, because it crosses privilege boundaries. Do you want users to be able to start system services? To stop them? In the general case, no, you certainly don't. In this particular case, you probably want seatd to be controllable by the user at the console. That's what s6-svperms is for: https://skarnet.org/software/s6/s6-svperms.html When relevant (at user login time? earlier?) make seatd controllable by a group only the console user is in. Then you can simply add a s6-svc -u for seatd in your sway run script, if that's what you need. -- Laurent
Re: OpenVPN runit service flooding tty
Hi, I am using the Artix Linux runit service for OpenVPN, available here: https://gitea.artixlinux.org/packages/openvpn-runit Sounds like two compounding problems: 1. the default logging place for the runit supervision tree is the same terminal you're using to log in 2. there is no dedicated logger for the openvpn service. IIRC, 1. is intrinsic to runit, there is no way to redirect the default logging, and that means your console is at the mercy of verbose services - so every service needs a dedicated logger. You can use --syslog as a workaround indeed, but 2. should be fixed: a runit service should use the runit logging facility. That is something you should report and suggest to the maintainer of the openvpn service in Artix. -- Laurent
[announce] s6-2.12.0.2
Hello, I don't normally spam all of you for bugfix releases, but this one is important. You definitely want to grab the 2.12.0.2 version of s6, not the 2.12.0.1 one. The bug could prevent a shutdown from completing. https://skarnet.org/software/s6/ git://git.skarnet.org/s6 Sorry about that, -- Laurent
[announce] but what about *second* skarnet.org November 2023 release?
Hello, New versions of some skarnet.org packages are available. This is mostly a bugfix release, addressing the problems that were reported since the big release two weeks ago. Despite that, s6-dns got a minor version bump because the fixes needed an additional interface; and s6-networking got a major bump, because it needed an interface change. Nothing that *should* impact you, the changes are pretty innocuous; but see below. skalibs-2.14.0.1(release) s6-2.12.0.1(release) s6-dns-2.3.7.0(minor) s6-networking-2.7.0.0(major) tipidee-0.0.2.0(minor) * skalibs-2.14.0.1 This release is important if you want the fixes in s6-dns: the ipv6 parsing code has been revamped. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * s6-2.12.0.1 --- It's only a bugfix, but you want to grab this version, because the bug was impactful (s6-svscanctl -an not working as intended). https://skarnet.org/software/s6/ git://git.skarnet.org/s6 * s6-dns-2.3.7.0 -- - The parsing of /etc/hosts now ignores link-local addresses instead of refusing to process the whole file. - New interface to only process /etc/hosts if a client requires it. https://skarnet.org/software/s6-dns/ git://git.skarnet.org/s6-dns * s6-networking-2.7.0.0 - - s6-tlsc-io has changed interfaces; now it's directly usable from a terminal. This change should be invisible unless you were using s6-tlsc-io without going through s6-tlsc (which, until now, there was no reason to do). - s6-tcpserverd now logs "accept" and "reject" instead of "allow" and "deny", this terminology now being reserved to s6-tcpserver-access. - The -h option to s6-tcpclient and s6-tcpserver-access has changed semantics. Previously it was used to require a DNS lookup, and was hardly ever specified since it was the default (with -H disabling DNS lookups). Now it means that DNS lookups must be preceded by a lookup in the hosts database. - A new pair of options, -J|-j, are accepted by s6-tlsc-io and s6-tlsd-io, and by extension the whole TLS chain of tools. -J means that s6-tls[cd]-io should exit nonzero with an error message if the peer fails to send a close_notify before closing the connection; -j, which is the default, means ignore it and exit normally. - The TLS tunnels work as intended in more corner cases and pathological situations. https://skarnet.org/software/s6-networking/ git://git.skarnet.org/s6-networking * tipidee-0.0.2.0 --- - Bugfixes. - New configuration options: "log x-forwarded-for", to log the contents of the X-Forwarded-For header, if any, along with the request; and "global executable_means_cgi", to treat any executable file as a CGI script (which is useful when you control the document hierarchy, but dangerous when it's left to third-party content manager programs). https://skarnet.org/software/tipidee/ git://git.skarnet.org/tipidee Enjoy, As always, bug-reports welcome. -- Laurent
Re: restarting s6-svscan (as pid 1)
> I believe (have not yet tested) that I can relatively simply create the maintenance system on the fly by copying a subset of the root fs into a ramdisk, so it doesn't take any space until it's needed. The problem with that approach is that your maintenance system now depends on your production system: after a rootfs change, you don't have the guarantee that your maintenance system will be identical to the previous one. Granted, the subset you want in your maintenance fs is likely to be reasonably stable, but you never know; imagine your system is linked against glibc, you copy libc.so.6, but one day one of the coreutils grows a dependency to libpthread.so, and voilà, your maintenance fs doesn't work anymore. You probably think the risk is small, and you're probably correct. I just have a preference for simple solutions, and I believe that a small, stable, statically-linked maintenance squashfs would be worth the space it takes on your drive. :) -- Laurent
Re: restarting s6-svscan (as pid 1)
This may be a weird question, maybe: is there any way to persuade s6-svscan (as pid 1) to restart _without_ doing a full hardware reboot? The use case I have in mind is: starting from a regular running system, I want to create a small "recovery" system in a ramdisk and switch to it with pivot_root so that the real root filesystem can be unmounted and manipulated. (This is instead of "just" doing a full reboot into an initramfs: the device has limited storage space and keeping a couple of MB around permanently just for "maintenance mode" doesn't seem like a great use of it) As Adam said, this is exactly what .s6-svscan/finish is for. If your setup somehow requires s6-svscan to exec into something else before you shut the machine down, .s6-svscan/finish is the hook you have to make it work. Don't let the naming stop you. The way to get strong supervision with BSD init is to list your services in /etc/ttys. You can't get any cheesier than that. That said, I'm not sure your goal is as valuable as you think it is. If you have a running system, by definition, it's running. It has booted, and you have access to its rootfs and all the tools on it; there is nothing to gain by doing something fragile such as exec'ing into another pid 1 and pivot_rooting. Unless I've missed something, the amount of space you'll need for your maintenance system will be the exact same whether you switch to it from your production system or from cold booting. -- Laurent
Re: s6-svunlink hangs with s6-2.12.0.0
calling s6-svunlink for a s6-svlink'ed service hangs with s6-2.12.0.0 and succeeds with previous s6-2.11.3.2 It's very probably caused by a known issue with s6-svscan. Could you please try with the latest s6 git and tell me whether it's fixed there? -- Laurent
[announce] skarnet.org November 2023 release
Hello, New versions of all the skarnet.org packages are available. This is a big one, fixing a lot of small bugs, optimizing a lot behind the scenes, adding some functionality. Some major version bumps were necessary, which means compatibility with previous versions is not guaranteed; updating the whole stack is strongly recommended. Also, tipidee is out! If you've been looking for a small inetd-like Web server that is still standards-compliant and fast, you should definitely check it out. skalibs-2.14.0.0 (major) nsss-0.2.0.4 (release) utmps-0.1.2.2(release) execline-2.9.4.0 (minor) s6-2.12.0.0 (major) s6-rc-0.5.4.2(release) s6-linux-init-1.1.2.0(minor) s6-portable-utils-2.3.0.3(release) s6-linux-utils-2.6.2.0 (minor) s6-dns-2.3.6.0 (minor) s6-networking-2.6.0.0(major) mdevd-0.1.6.3(release) smtpd-starttls-proxy-0.0.1.3 (release) bcnm-0.0.1.7 (release) dnsfunnel-0.0.1.6(release) tipidee-0.1.0.0 (new!) * skalibs-2.14.0.0 This version of skalibs adds a lot of new sysdeps, a lot of new functions, and changes to existing functions, in order to support the new features in other packages. The most important change is the new cspawn() function, providing an interface to posix_spawn() with support for most of its options with a fork() fallback for systems that do not have it. What this means is that on systems supporting posix_spawn(), the number of calls to fork() in the whole skarnet.org stack has been significantly reduced. This is important for programs where spawning a new process is in a hot path - typically s6-tcpserver. Updating skalibs is a prerequisite for updating any other part of the skarnet.org stack. Once you've updated skalibs, you probably don't *have to* update the rest; old versions of packages should generally build with the new skalibs as is, and if indeed they do, nothing should break. But it is a major update, so there are no guarantees; please update to the latest versions at your convenience. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * execline-2.9.4.0 - execlineb now has a dummy -e option (it does nothing). This is so it can be used as a replacement for a shell in more environments. Also, execline programs use fork() a lot less, so overall execline script performance is better. - The multicall setup did not properly install symbolic links for execline programs; this is fixed, and is fixed as well as in other packages supporting a multicall setup (s6-portable-utils and s6-linux-utils). https://skarnet.org/software/execline/ git://git.skarnet.org/execline * s6-2.12.0.0 --- - s6 programs use fork() less. - New -s option to s6-svc, to send a signal by name or number. - s6-svscan has been entirely rewritten, in order to handle logged services in a more logical, less ad-hoc way. It should also be more performant when running as init for a system with lots of s6-supervise processes (improved reaping routine). - The obsolete (and clunky) s6lockd subsystem has been deleted. s6-setlock now implements timed locking in a much simpler way. https://skarnet.org/software/s6/ git://git.skarnet.org/s6 * s6-linux-init-1.1.2.0 - - New -v option to s6-linux-init-maker, setting the boot verbosity. - Several small bugfixes, one of them being crucial: now your systems shut down one second faster! https://skarnet.org/software/s6-linux-init/ git://git.skarnet.org/s6-linux-init * s6-linux-utils-2.6.2.0 -- - Support for the minflt and majflt fields in s6-ps. https://skarnet.org/software/s6-linux-utils/ git://git.skarnet.org/s6-linux-utils * s6-dns-2.3.6.0 -- - Support for on-demand /etc/hosts data in s6-dnsip and s6-dnsname. It is achieved by first processing /etc/hosts into a cdb, then looking up data in the cdb. You can, if you so choose, perform this processing in advance via a new binary: s6-dns-hosts-compile. https://skarnet.org/software/s6-dns/ git://git.skarnet.org/s6-dns * s6-networking-2.6.0.0 - This is the package that has undergone the biggest changes. - No more s6-tcpserver{4,6}[d]. IPv4 and IPv6 are now handled by the same program, s6-tcpserver, which chainloads into a unique long-lived one, s6-tcpserverd. - s6-tcpserver now exports TCPLOCALIP and TCPLOCALPORT without the need to invoke s6-tcpserver-access. - s6-tcpserver-access does not hardcode a warning when it is invoked without a ruleset. It can now just be used for additional data gathering (such as TCPREMOTEHOST) without jumping through hoops. - s6-tcpserverd has been thoroughly optimized for performance. It will handle as heavy a load as the underlying system will allow. - Yes, this means you can now use s6-tcpserver to serve
Re: s6-rc-update does not create pipes when already-running service becomes a consumer
I agree with all you're saying here... ... except that this case makes no sense in the first place. A consumer and a non-consumer are fundamentally different things. A running service that is not a consumer does not read anything on its stdin. It runs autonomously. A consumer, by essence, is waiting on stdin input, in order to process it and do something with it. If a non-consumer were to become a consumer, it would mean its very nature would be transformed. It would not behave in the same way at all. And so, it makes no sense for it to be the "same" service. It should not be treated the same way; it should not keep the same name. So, if anything, the change I would make is, when a non-consumer becomes a consumer, s6-rc-update just throws an error and exits. The situation is different with a non-producer becoming a producer, because a non-producer can already naturally write to its stdout, and its output simply falls through to the catch-all logger. When becoming a producer, it just needs to write into a pipe instead, and the pipe already exists since it's created on the consumer side, which already exists, so indeed it's all a matter of restarting the service. -- Laurent
Re: timestamp or timing info for s6-rc services?
Hi all, I am running s6 on buildroot with a bunch of custom services (and bundles) with dependencies. Was there a way to turn on timestamps for s6-rc logging? I am currently logging with -v 2 option set. Or is there any timing information saved between a service starting and successfully started (or stopped) that I get at that is already saved somewhere? I don't know exactly what customization options Buildroot gives you, but since the output from oneshots and from longruns without a dedicated logger goes to the catch-all logger, you would achieve what you want by making sure your catch-all logger has timestamps activated. If the setup is using s6-linux-init, the -t[1|2|3] option to s6-linux-init-maker will do that; it should be an option in the Buildroot configuration. -- Laurent
Re: Systemd unit parser and s6 generator
what I can do, if it is of interest to you, is list all the directives in a service file and rate their conversion difficulty, so you can then evaluate your own service files and assess the feasability of an automated conversion tool. It took way longer than expected, and was a harrowing task, but it's finally done - and I'm glad I did it because it was a fascinating dive into the systemd worldview, and I'm all the more knowledgeable for it. (And a little dumber, too.) The results can be found here: https://skarnet.org/software/s6/unit-conversion.html I could probably have written a comment - and, sometimes, even an informative and non-snarky one! - for every single directive I listed, but I had to restrain myself in order to finish the document before the end of the year. Enjoy. -- Laurent
Re: [svlogd] / -ttt / why UTC?
I don't know exactly what u mean with "TAI-10". I guess u are refering to those 10seconds Yes. You cannot set your system clock to TAI, unless you want wildly incorrect results from time() and similar system calls. Setting it 10 seconds earlier than TAI is the best you can do; and that's what the right/ timezones expect. here i disagree a little bit: As long as the software uses glibc's time functions to break down "seconds since the start of 1970* to year, month, mday, hours, minutes, seconds, the software does not need any patches... Ur link is a little bit misleading or outdated in that point... Have you tried it? localtime() will work if your timezone is correctly set, yes. gmtime() *will not work*, because it assumes a UTC system clock. Programs making their own assumptions, and there are a lot of these, will not work. Don't think everyone uses the libc; new compiled language ecosystems such as for instance Go or Rust have their own time libraries, and can you be certain they don't assume UTC? "As long as the software pays strict attention to the standards and follows them to a T even for non-100%-cookie-cutter cases" is a very strong hypothesis to make, even when you perform minor changes. Changing the way the system clock counts time is NOT a minor change. What would be wrong about the "-" (localtime time stamp) option? Then I would not have to build/write my own *log daemon... And svlogd just needs to use localtime_r(3)... In theory, nothing, except that it's a bad idea to timestamp logs with local time (hello, we're in the 21st century, we manage computers worldwide, we want logs to be shareable and mergeable across timezones). In practice, you're asking a maintainer to perform work that *you* need, and that very few other people need. That's generally considered bad form. The way to go about it would be to implement the functionality yourself, and submit a patch; that said, since the last time we saw svlogd's maintainer was four years ago in a flash, I probably wouldn't bother if I were you. By recommending s6-log, I gave you the solution that requires from you the least amount of work and the least amount of waiting. -- Laurent
Re: [svlogd] / -ttt / why UTC?
My boxes use TAI (international atomic time) in order to have SI-seconds and 60sec minutes and 24hrs days... If your system clock is set to TAI-10, then *all* the time-handling software on your machine must be aware of it, in order to perform time computations accurately. It is not sufficient to use the right/ time zones: a TAI-10 setup changes the interpretation of what system calls such as time() and clock_gettime() return. It is only possible to do this and have a consistent view of time if you control any and every piece of software on your machine that needs to read and display time. See https://skarnet.org/software/skalibs/flags.html#clockistai for a description of the difference between "system clock set to TAI-10" and "system clock set to UTC". If you're using a distribution - in your case, Void Linux - then TAI-10 is not an option for you, unless the distribution explicitly supports it and has built all its binaries to support it. Even if you could make svlogd work with it, you would have trouble with other software - this may be benign, like "date", or less so, like software that checks the validity of X.509 certificates against certain things like expiration dates. Believe me, I understand your plight, and I wish TAI-10 were the de facto standard instead of UTC. I wish it were at least more widely known and supported. I use it on my servers - but it comes at a heavy price: not letting any third-party software, 99.9% of which is hardcoded for UTC, interact with time, and only using my own tools, which support the setup, to do that. I don't think you'll be able to keep running Void with a full TAI-10 setup and have everything work perfectly, at some point you'll have to make a real choice. But right now, to superficially solve your current issue, you could try running s6-log (from s6) in place of svlogd. It does the same thing, but it supports TAI-10, if you build skalibs with the --enable-tai-clock option, as described by the link above. You probably have to build skalibs and s6 yourself: I don't think the Void package for skalibs uses that option. Good luck, -- Laurent
Re: *-man-pages: s6-rc port, Makefile fix
An mdoc(7) port of the documentation for s6-rc is now available: That's awesome, thanks a lot Alexis! 🥳 -- Laurent
[announce] April 2023 bugfix release
Hello, New versions of some skarnet.org packages are available. They fix a few visible bugs, so users are encouraged to upgrade. I usually do not announce bugfix releases. This e-mail is sent because two new functionalities were also in git when the bugfixes needed to be made, so they're now available: - A new -D option to execline's elgetopt, allowing customization of the value for the ELGETOPT_n variable when the -n option is given and does not expect an argument. - A new -R option to s6-linux-init-maker, allowing you to set hard resource limits for the system at boot time. skalibs-2.13.1.1 (release) execline-2.9.3.0 (minor) s6-2.11.3.2(release) s6-linux-init-1.1.1.0 (minor) s6-portable-utils-2.3.0.2 (release) Enjoy, Bug-reports always welcome. -- Laurent
Re: s6-rc how to log oneshot services
Artix Linux achieves this with pipeline -dw { s6-log ... } actual-oneshot Not ideal due to the lack of supervision for s6-log, but it works. Yes, it works. The lack of supervision doesn't matter here because the s6-log is short-lived: it will die when actual-oneshot exits. The main drawback here is that it's overengineered: chances are that actual-oneshot is not going to write enough data to trigger a rotation, so directly appending to a file would be simpler and faster. Additionally, the -d option to pipeline isn't necessary: unless actual-oneshot does very suspicious things, it shouldn't notice it has a child (which will outlive it by a few microseconds). -- Laurent
Re: s6-rc how to log oneshot services
The default stderr for oneshot services is the default stderr of the supervision tree you're running s6-rc with. So, if you're using s6-linux-init, stderr for oneshot services goes directly to the catch-all logger and you don't need the redirection. If you want your oneshots to log to a different place, you need to redirect the oneshot's stderr to where you want. Since oneshots are, by definition, only run once, it's generally not necessary to have a logger process for them; you can redirect to a file (but be careful of concurrently running oneshots if you have several oneshots logging to the same file!) You can send these logs to a logger process as well, but you cannot use s6-rc's pipeline feature for that; you'll need to declare a fifo manually, have a longrun service reading from that fifo, and have your oneshot depend on that longrun and redirect its stderr to the fifo. It's probably more trouble than it's worth. -- Laurent
Re: Systemd unit parser and s6 generator
We've discussed internally if we change that process and try to write a systemd unit parser, because all units are there in Ubuntu. If we could catch 90% of all cases, we need, we would be happy. If it would take 2 weeks of work, that would be fine. Did somebody of you try to implement something? What are your thoughts? Hi Oli, This is a subject that comes up regularly in the #s6 IRC channel. And I always answer something like this: The difficulty of converting systemd services to s6 services is not a syntax problem. The unit file syntax is mostly fine. The difficulty is the difference of world modelization between systemd and s6. systemd's view is very holistic, every component can depend on another one or rely on a systemd-only feature, and the unit file syntax reflects that; while s6's view is more local, every service is its own independent thing. But more importantly, the way systemd maps the system into concepts is pretty different from the way s6 maps the system into concepts, and these two views are not 1-to-1 compatible. Translating a setup between systemd and s6 requires intelligence; it is not possible to write an automated tool that does it accurately and idiomatically without going *deep* into things. In practice, yes, it is possible to write a converter that will take systemd.service files and turn them into s6 service directories, and that will work "most" of the time, depending on how your service files look. Most directives in systemd.service are directly translatable to s6 features. However, as soon as you start using esoteric systemd directives, the complexity explodes, and it's all a balancing act on what you want to support vs. how difficult it is. And only systemd.service files are automatically convertible; other unit files such as systemd.socket, systemd.path and systemd.target are all dependent on the systemd-centric system view and it is impossible to treat them without analyzing the entire system. I still owe you a couple hours of work, so what I can do, if it is of interest to you, is list all the directives in a service file and rate their conversion difficulty, so you can then evaluate your own service files and assess the feasability of an automated conversion tool. 90% coverage is doable if your services are super simple and don't rely on systemd-specific features, but it's very easy to get lost in the weeds. -- Laurent
[announce] skarnet.org February 2023 release
Hello, New versions of some skarnet.org packages are available. It hasn't been long since the last release, but lots of small things have happened and it doesn't make much sense to let them rot in git. The main addition is a new multicall configuration for the execline, s6-portable-utils and s6-linux utils packages. When you give the --enable-multicall option to configure, a single binary is compiled, and 'make install' installs this binary and creates symlinks to it. This is useful to setups that focus on saving disk space. Credit for this addition goes to Dominique Martinet, who nerd-sniped me into actually testing such a configuration; and it turned out the disk space gains were very impressive for execline (up to 87%!) I applied the idea to the s6-portable-utils and s6-linux-utils packages, which are also made of small, very simple, independent programs, to see whether it was viable in the general case; but as I suspected, the gains were not as impressive, and making it work required a significant refactoring effort. Since other skarnet.org packages would have an even worse gains/effort ratio, the experiment is stopping there. execline is an outlier, with a 177 kB amd64 static binary being able to replace a 1.3 MB set of binaries; that's much better than I thought it would be, so it's worth supporting. Enjoy. Other changes include mostly bugfixes and quality-of-life improvements. The new versions are the following: skalibs-2.13.1.0 (minor) nsss-0.2.0.3 (release) execline-2.9.2.0 (minor) s6-2.11.3.0 (minor) s6-rc-0.5.4.0 (minor) s6-linux-init-1.1.0.0 (major) s6-portable-utils-2.3.0.0 (major) s6-linux-utils-2.6.1.0(minor) s6-networking-2.5.1.3 (release) mdevd-0.1.6.2 (release) Details of some of these package changes follow. * skalibs-2.13.1.0 - Bugfixes. - New function: sals, listing the contents of a directory in a stralloc. Straightforward, but a large-ish piece of code that was used in multiple places and needed to be factored. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * execline-2.9.1.0 - New --enable-multicall configure option. This is the big one for some distributions, that don't want to spend 1 MB of disk space on execline binaries. (They already know my position on that.) https://skarnet.org/software/execline/ git://git.skarnet.org/execline * s6-2.11.3.0 --- - Bugfixes. - Instance-related internal changes. Instanced service directories need to be recreated with the new version of s6-instance-maker. - New s6-svc -Q command, instructing s6-supervise not to restart the service when it dies (like -O) and to additionally create a ./down file in the service directory. - s6-ioconnect will now always shutdown() socket endpoints at EOF time; the -0, -1, -6 and -7 options are still supported, but deprecated. https://skarnet.org/software/s6/ git://git.skarnet.org/s6 * s6-rc-0.5.4.0 - - Bugfixes. In particular, s6-rc-update now conserves the existing instances in an instanced service, whether the service is currently active or not. In case of a live update, the current instances keep running, but will restart with the new template next time they die (which can be forced by a s6-instance-control -r invocation). - New s6-rc subcommands: start and stop, equivalent to "-u change" and "-d change" respectively. https://skarnet.org/software/s6-rc/ git://git.skarnet.org/s6-rc * s6-linux-init-1.1.0.0 - - s6-linux-init-maker: -U option removed. No early utmpd script is created. The reason for this change is that distros using utmps need stage 2 utmp services anyway (because the wtmp database needs to be persistent so wtmpd and btmpd can only be started after a log filesystem has been mounted), so utmp is unusable before that point no matter what. Distros should have an utmpd service started at the same time as wtmpd and btmpd; so utmp management goes entirely out of scope for s6-linux-init. https://skarnet.org/software/s6-linux-init/ git://git.skarnet.org/s6-linux-init * s6-portable-utils-2.3.0.0 - - s6-test removed, hence the major update. - New --enable-multicall configure option. https://skarnet.org/software/s6-portable-utils/ git://git.skarnet.org/s6-portable-utils * s6-linux-utils-2.6.1.0 -- - s6-mount option support updated. - New --enable-multicall configure option. https://skarnet.org/software/s6-linux-utils/ git://git.skarnet.org/s6-linux-utils Enjoy, Bug-reports welcome. -- Laurent
[announce] skarnet.org January 2023 release
Hello, New versions of the skarnet.org packages are available. This release is overdue, sorry for the delay - but finally, happy new year everyone! skalibs' strerr_* functions and macros, meant to provide shortcuts for error message composition and output, have been rewritten; they're no longer split between strerr.h and strerr2.h, but are all gathered in strerr.h - the skalibs/strerr2.h headers is now deprecated. This is released as a major version upgrade to skalibs because some hardly ever used strerr macros have been outright removed; and the deprecation of strerr2.h also counts as an API change. However, unless you were using the deleted strerr macros (highly unlikely, as there was no reason to, which is why they're being deleted in the first place), your software should still build as is with the new skalibs, maybe with warnings. The rest of the skarnet.org software stack has undergone at least a release bump, in order to build with the new skalibs with no warnings. Most packages also include several bugfixes, so upgrading the whole stack is recommended. The new version of s6 includes a feature that has often been asked for: an implementation of dynamically instanced services. Six new commands allow you to create and manage dynamic instances of a given service directory template, parameterized by an argument you give to the run script. It also comes with a few quality-of-life changes, such as s6-log line prefixing, as well as a good number of minor bugfixes. The "s6-test" program, formerly in s6-portable-utils, has migrated to the execline package, where it is named "eltest". It still exists in s6-portable-utils, but is deprecated and will be removed in a future release. The new versions are the following: skalibs-2.13.0.0 (major) nsss-0.2.0.2 (release) utmps-0.1.2.1(release) execline-2.9.1.0 (minor) s6-2.11.2.0 (minor) s6-rc-0.5.3.3(release) s6-linux-init-1.0.8.1(release) s6-portable-utils-2.2.5.1(release) s6-linux-utils-2.6.0.1 (release) s6-dns-2.3.5.5 (release) s6-networking-2.5.1.2(release) mdevd-0.1.6.1(release) smtpd-starttls-proxy-0.0.1.2 (release) bcnm-0.0.1.6 (release) dnsfunnel-0.0.1.5(release) Details of some of these package changes follow. * skalibs-2.13.0.0 - Bugfixes. - New functions: buffer_timed_put, buffer_timed_puts, for synchronous writes to a file descriptor with a time limit. - strerr2.h deprecated. strerr.h entirely revamped. Every existing strerr interface is now a variable argument macro around the new strerr_warnv, strerr_warnvsys, strerr_diev and strerr_dievsys functions, which just prints arrays of strings to stderr. This reduces the amount of adhocness in the strerr code considerably, allows calls without an upper bound on the number of strings, and should save some bytes in resulting binaries. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * execline-2.9.1.0 - Bugfixes. - New program: eltest. This is the program formely available in s6-portable-utils as "s6-test", that has changed packages and be renamed. It's a quasi-implementation of the POSIX "test" utility, that was too useful in execline scripts to be off in a separate package. (Quasi because the exact spec is bad.) It understands -v, for testing the existence of a variable, and =~, for regular expression matching. https://skarnet.org/software/execline/ git://git.skarnet.org/execline * s6-2.11.2.0 --- - Bugfixes. - The one-second service restart delay can now only be skipped when the service is ready. This prevents CPU hogging when a heavy service takes a long time to start and fails before reaching readiness. - The name of the service is now passed as the first argument to ./run and as the third argument to ./finish. - s6-log now understands a new directive: p. "pfoobar:" means that the current log line will be prepended with the "foobar: " prefix. This allows service differentiation in downstream log processing, which was an often requested feature. - New commands available: s6-instance-maker, s6-instance-create, s6-instance-delete, s6-instance-control, s6-instance-status, s6-instance-list. They allow you to manage supervised sets of services created from the same templated service directory with only a parameter (the name of the instance) changing. https://skarnet.org/software/s6/ git://git.skarnet.org/s6 * s6-portable-utils-2.2.5.1 - - s6-test is now deprecated, replaced with the eltest program in the execline package. https://skarnet.org/software/s6-portable-utils/ git://git.skarnet.org/s6-portable-utils Enjoy, Bug-reports welcome as always. -- Laurent
Re: s6 vs shell wrappers for client connections
if s6-svstat myserver; then client_binary else send_email_to_admin faux_client_binary fi Please don't do this. - As Steve would be able to tell you if he had read the documentation page for s6-svstat (https://skarnet.org/software/s6/s6-svstat.html), "s6-svstat myserver"'s exit code does not mirror whether myserver is running or not. If myserver is a live service directory, s6-svstat will print a line to stdout summarizing the status of the service, then exit 0. Zero, even if the service is down, so client_binary will fail. s6-svstat will only exit nonzero if an error occurs or if no supervisor is running on myserver, which is not what this script aims to test. - It is good to notify the admin when something on the system is not working correctly. Doing it automatically from a script, however, is not a good idea. Spamming the admin's mailbox via automated messages is an great way to get yourself ignored at best (which defeats the purpose of reporting bugs) or banned at worst. - If you're using s6, as I said earlier, you're supposed to be able to assume that myserver is running, and you don't have to write such wrappers, even ones that would work. -- Laurent
Re: s6 vs shell wrappers for client connections
You have a program that can be started normally or as a service that accepts connections through a socket. For client connections, an additional binary is supplied. The simplest way to make sure that the program launches regardless of whether there's a server running or not is a wrapper script that executes the right binary based on socket's availability. In a vacuum, for a generic client, yes. On a machine you control, there is a simpler way: by policy, you should *know* if a server is available or not. s6 won't help you much with making a client that works in uncertain situations. What it will help you with, however, is reduce the unknowns on a machine you use it with. In this case, it can ensure that you *know*, by policy, that a given server is running, so you can assume its presence - and if the assumption is wrong, it's a bug, and the machine's admin should fix it. Example of this is the 'foot' terminal emulator - it has 'foot [--server]' and footclient binaries. How, if at all, could s6 help remove this executable ambiguity, the need for checking and wrapping? (...) To continue with the example: I set up 'foot' as described above. The result is that s6-svscan/supervise starts the 'foot' server, places an 'S' socket in the service directory and sits there. I can connect to the server by pointing to socket in service directory $ footclient -s "$s6-foot-servdir/S" This however, still requires me to check for the socket and if-else the binary and options each time I want to start the program, doesn't it? Does s6 offer a remedy? If you're running a machine with an s6-supervised foot service, then the point of it is that the foot server is always running. The supervision tree will start it at boot time, and maintain it through the machine's lifetime: so, the footclient invocation is always the correct way to run a client. You don't have to plan for the case where there's no server: if there's no server, it's a bug, and outside the responsibility of the client to plan for. If you don't want to have a server permanently listening to a socket, then don't add a supervised service; but in this case, you will have to deal with the unknown state of your machine, and script your client so it spawns a server as needed. Needless to say, I think that's a technically inferior solution. ;) (We had a good IRC discussion not long ago about the merits and drawbacks of on-demand server spawning. My point of view is that on-demand isn't nearly as useful as people pretend it is, because you provision a server for max charge anyway, and idle services do not consume resources - so on-demand spawning does not increase the *power* of your setup, it only increases its *flexibility*, which is very overrated on most modern setups which are, for better or worse, built for a fixed set of tasks.) Hope this helps, -- Laurent
Re: [PATCH] Fix documentation typos
-file named S6_FDHOLDER_STORE_REGEX is found is the env/ +file named S6_FDHOLDER_STORE_REGEX is found in the env/ Applied, thanks. -service readiness, you should give this option along with up: the service is ready iff +service readiness, you should give this option along with up: the service is ready if Not a typo. "iff" means "if and only if", as any mathematician will tell you. :) -- Laurent
Re: ca catchall logger prefix log lines with service names?
I am trying to figure out if I can set up svscan catchall logger in such a way that it prepends a service name to every log line, so that it can be clear where the log came from. I am trying to avoid s6-rc setup where I need to explicitly create a matching logger service. You are saying "I want to use s6 without adopting s6 idioms because I'm used to the way other supervisors do things, even when they're giant hacks." :) It's much easier to have one dedicated set of logs per service, and merge them later if that's what you want, than to have one big blob of logs and separate them later for analysis. In other words: "cat | sort" is cheaper than a series of "grep". The catch-all logger is exactly what it says: a catch-all. It only gets the logs that are not diverted; it's not supposed to get *everything* - although it can, if that's what you want, but then as you noticed you get everything as is: output from the services go directly to the catch-all logger. There is no intermediary process that can prepend your log lines with information, because the services have their output directly plugged into the pipe to the catch-all logger. The only way to modify the log lines is to insert an additional process between your service and the catch-all logger that performs the modification. And the simplest way to achieve that is to have a foo/log service (or a foo-log consumer, if you're using s6-rc) for every foo producer, that processes its stdin and writes the modified lines to stdout, something like s6-format-filter "foo: %s" But that's still defining a consumer for your service. There is, unfortunately, no way to do that without a consumer for every service. And if you're going to add consumers anyway, you may as well write dedicated loggers directly instead of line processors that send everything back to the catch-all logger. Note that it's not s6 being difficult; it's s6 being out of the way by default, by not touching your logs. If another supervisor allows you to modify log lines by default, it means it's inserting itself between the service and its logger, which is terribly inefficient. But creates a log file where all logs from all services are mixed up and are impossible to distinguish. There is already a strange looking thing prepended to every line (@40006356da4b2cb3ba0a - a timestamp?) Yes, it's a TAI64N timestamp. Process the file through s6-tai64nlocal to get human-readable timestamps. One of the advantages of TAI64N stamps is that they're alphabetically sorted, so you can merge several log files into one via "cat log1 log2 log3... | sort". -- Laurent
Re: Disconnect between s6-svscan and s6-rc
The user of s6-rc gets no error message, and waits forever. The error message is captured by s6-svscan (or a corresponding logger for that service) and is either saved into a log file, or printed to a tty on which svscan is running. The user is almost never on the same tty with svscan. The user never gets an error message from the service, unless they explicitly know where to look. There's no "tty on which svscan is running". s6-svscan should not run in a terminal. You *can* do it, but it's fragile: it means your whole supervision tree is vulnerable to a ^C in a terminal. The output of s6-svscan, and the supervision tree by default, should go to a catch-all logger and potentially to the system console. It would be nice if a user can get a temporary view into the service's stdout for the short duration of s6-rc execution? Such that when s6-rc never exits waiting we can see what the error message that service prints? For oneshots, it is the case: the output of oneshots will go to the output of s6-rc. Unfortunately, for longruns, that's pretty difficult. One of the points of supervised longruns is that they're always started with the exact same environment, and that includes their descriptors. That means they're outputting to the same place whether or not s6-rc is running. In order to temporarily display error messages from longruns, s6-rc would have to find out where the catch-all logger is, and display it to the user while it's running. That's impossible while s6-rc is independent from s6 policy. Unified policy for better interaction between tools in the toolset is one of the goals for s6-rc v1, which is still a long ways away. For now, my advice would be to always use a timeout when running s6-rc (you will have an error message if it times out), and/or to have the supervision tree's output on the system console, and have the reflex to check the console when you feel that s6-rc is taking more time than it should. Sorry I don't have any better immediate solution. -- Laurent
Re: s6-rc as user service manager
Thanks Peter, this was actually helpful and enchanced my mental model. I think I get get away for now with a user's tree rooted in the system tree. My graphics environment (sway) can start necessary services when it is started. Yeah, it's a recurring discussion on the IRC channels, and my answer is always that "user services" as systemd does them aren't a well- defined concept - "logging in" doesn't always have a clear meaning: do sshd logins count? Or only seat sessions? What happens when the same user logs in via 2 different seats and 1 sshd, then logs out of one seat? then of the second seat? Should the services be stopped, and if so, when? systemd has to make choices and makes things work, more or less, in the common case of one user at one seat - but that's a very unsatisfying answer from a developer's point of view. s6 users are also more likely to log in remotely more often than systemd users, so maybe systemd's choices aren't the best ones for s6. "User services" are a can of worms, and since I'm always very reluctant to enforce policy on users or make choices that will not work in all cases, it's not one that I'm willing to open beyond what's provided by s6-usertree-maker. I'm happy that you can work with a permanent user tree - that is a well-defined concept that can be implemented and wholeheartedly supported with s6. By testing I meant checking if the directory has an active process watching it. I believe there is a function in skalibs fd_lock [1] that svscan uses to check if another svscan runs there. I think it is just a matter of exposing that function as standalone executable. There are no executables to test whether s6-svscan or s6-rc are running on a given directory, because these are not dynamic properties. By policy, decided by you or your distro, you should *know*, at all times, whether a given directory is a scandir with an s6-svscan running on it - or whether a given directory is a livedir with s6-rc running on it. If you think a given directory should have an s6-svscan running on it, then you're right; ensure that s6-svscan is started at boot time, and write your scripts assuming that it's there. If something fails because it's not there, that's a bug or a system problem, and needs to be fixed, not accommodated by your scripts. -- Laurent
Re: s6-rc user experience question
Perhaps a higher-level orchestration tool(s) is/are needed, that will accomplish most typical workflows like: (...) These are all valid points, and things that ultimately s6-frontend, the future UI over s6/s6-linux-init/s6-rc, aims to solve. "Higher-level interface" is the (now) #1 feature request for the s6 family of tools, so be assured I've heard it. :) No promises on the date of delivery, but it's definitely on the radar. -- Laurent
Re: s6-rc user experience question
Perhaps I can offer a few suggestions how to improve usability: - combine compile + svscan on empty dir + init steps into one, like `s6-rc init source_dir` and it does those steps under the hood. No, because these are operations that are ideally done at different times. - Compilation happens "offline", before you (re)boot the machine. - s6-svscan is run independently from s6-rc, typically very early during the boot sequence. (Ideally it's process 1.) In any case the preexistence of a supervision tree is assumed by s6-rc, so spawning one as part of an s6-rc command isn't good workflow. - Initialization happens when the service manager is ready to run, which means when the machine is online and already has an s6-svscan running. Combining these operations is only useful if you're doing things in a very convoluted, suboptimal way. - maybe instead of creating file based database take on sqlite as a dependency and keep compiled + live state there? sqlite is ubiquitous and very light weight. It can save the trouble of inventing our own compiled database formats and folder strucutres The format of the compiled database is entirely orthogonal to the usability. Adding a dependency to an external package would do exactly nothing to improve the UX here; it would just make the code bigger, slower, more complex, harder to maintain and less secure. And that's without mentioning that a relational database is not at all the right structure for the kind of data that s6-rc deals with. Not a great trade-off, if you ask me. However, there's a change I can make that would immediately improve s6-rc's usability: since it's largely a perception issue, changing the vocabulary from "s6-rc-compile" to "s6-rc-snapshot" and "compiled database" to "opaque snapshot of the set of services" would be a good idea. It would spook fewer people, and send fewer people into red herring trains of thoughts. :) What are your plans / thoughts on s6-rc redesign? It's ongoing. It's difficult, and I always find a myriad of things to do to distract me from working on it, so it's going (very) slowly. But it's always lurking in the back of my mind, and there isn't a day that I don't wonder what I have gotten myself into. The initial call for funds is still readable here: https://skarnet.com/projects/service-manager.html -- Laurent
Re: s6-rc user experience question
1. First we create 'scandir1', and put services there. Each service is a svcdir. We put dependencies file and type file in each svcdir. (We do not run svcscn on it, because it doesn't really manage dependencies) That's a bad reason. :) The real reason why you don't run s6-svscan on a source s6-rc directory is because it's not a suitable scan directory. It doesn't contain service directories: it contains s6-rc definition directories, which may be for longruns (and in this case *look like* service directories but they're not, for instance the "run" file doesn't have to be executable), but also for oneshots or bundles, and s6-svscan would not know how to deal with those. And they have dependency information, which s6-svscan cannot interpret, etc. Even if their formats are similar, a source directory and a scan directory are not the same and aren't used for the same purposes. 2. We run s6-rc-compile pointing at scandir and get 'compiled' dir as output. 3. We run svcscan process on an emtpy dir - 'scandir2' 4. We run s6-rc-init , feeding 'compiled' and 'scandir2' dirs we get 'live' dir. At this point things seem to be working and I can use s6-rc to bring up and down services with dependencies. That is indeed the correct way to proceed. But this gets very confusing and does not look like a good user experience: I agree it's confusing, and one of the reasons (but not the main one) why s6-rc needs a redesign: you currently get access to all the internal workings of the service manager, even when they're not relevant to you, which is not ideal. So same information is duplicated 3 times and symlinked 3 times. Is this the intended flow? Or have I messed something up really badly? It's the intended flow, and the information duplication is necessary. - The source directories are where you, the user, edit the information. It's your working copy of it. You should be able to change it as you wish; the modifications you make should not impact the system. - The compiled directory is where the information is stored when s6-rc-compile validates your source directories. It's an automation- approved snapshot of the state of your source directories at the time you called s6-rc-compile. It is immutable: neither you, nor the system can modify it - and you can store it on a read-only filesystem. The point is that once you have it, and it works once, it will *always* work. - The live directory is the working copy of the *system*. It manages what is going on *right now* on your machine. Some of it is symbolic links to a compiled directory, when the system only needs to read information; but some of it is state that can be changed by the system and that needs to be a copy. That includes longrun service directories, that maintain running state, which is why these need to be copied from the compiled directory. Having this duplication is the only way of ensuring that your modifications do not change the snapshots you take when compiling and do not impact the current machine state, and also that the operation of your current services does not impact the set of services you're booting on (which could lead to failed boots). Other service managers do not make the distinction between the user working set, the system working set, and immutable snapshots; that is a big issue I have with them, because it's difficult to make them safe. Hope this helps, -- Laurent
Re: "Back off" setting for crashing services with s6+openrc?
To me this seems like a relevant improvement that would catch a problematic edge case issue. I pushed such a change to the s6 git. A new numbered release should be cut soon-ish. -- Laurent
Re: ftrig pipe naming convention
(I was on vacation, sorry for the delayed answer.) Could you please elaborate on the possible race condition? This is simply for curiosity and educational purposes. It feels like a lot of thought was put into s6 codebase, and a lot of ideas are not immediatedly obvious for people not intimately familiar with OS interface. When you want to listen to notifications, one of the first questions that come to mind is: what happens if a notification comes before I start listening to it? How do I make sure I don't miss anything? That's the primary race condition for notifications, and the reason why a simple tool such as s6-ftrig-wait can only be used in the simplest of settings: when you run s6-ftrig-wait, what you get is notifications from the moment you run it, and you don't know what happened before. The answer is synchronization between the listener and the sender. In order to make sure the listener misses nothing, *first* the listener starts listening, *then* the sender is run and can notify the listener. That's how s6-ftrig-listen1 works: the rest of its command line is spawned *after* it has made sure there's a fifo listening in the fifodir, and that command line is supposed to be something that tells the sender that it's okay to start sending notifications now. ftrigr_subscribe() is the primitive that makes a program listen to notifications in a fifodir, and returns control to the client. It is important because it is asynchronous: notifications will be read and processed as soon as ftrigr_subscribe() returns, and the client can do whatever it needs to do, such as read a state or prime the notification sender, and then handle the notifications in its own time by calling ftrigr_update(). The fact that you can do something between subscribing and handling the notifications is fundamental, and what makes the model strictly more powerful than "cat < $fifo". Internally, it's the s6-ftrigrd helper program, spawned by the ftrigr_startf() primitive, that performs the system calls in the fifodirs the client is subscribed to, filters notifications according to regexps, and sends the relevant notifications to the client; it is necessary to have an external program doing that, in order to save a lot of menial work from the client and avoid forcing it into a given asynchronous programmation model. s6-ftrigrd hides all the low-level details from the client and allows the ftrigr library to remain usable in a variety of programming models. ftrigr_subscribe() simply tells s6-ftrigrd to open a new fifo in a fifodir, and waits for an answer code. If it gets a successful answer from s6-ftrigrd, it means the fifo is open and from now on every notification will be correctly received and processed. The client can then proceed to the operation that can cause notifications to be sent. s6-ftrig-listen1 runs ftrigr_subscribe(), *then* spawns the rest of its command line - that's how race conditions are avoided. In the case of supervision, this is used to track the state of a service. When a command such as s6-svwait wants to wait until a service is in a given state, *first* it runs ftrigr_subscribe(), *then* it looks at the current registered service state (in the supervise/status file), and then it computes the new service state according to the data it receives from s6-ftrigrd. There is no race window during which s6-svwait would have read the status file but not be reading notifications yet, which would risk missing state changes. That is the main race condition that the ftrigr library solves. Now, additionally to that, there is another, less serious race condition that is more directly related to what you were asking about, with directly creating fifos in fifodirs. The "send a notification" primitive is ftrigw_notify() (with its close family members for various details). It will open all fifos in a fifodir that match the hardcoded name pattern in succession, and write a byte to them. Normally, this write succeeds: there's a s6-ftrigrd reader behind each one of these fifos - and anything else means there's a problem. Most likely, it's a benign problem, such as a stale fifo: s6-ftrigrd was killed before it had the chance to clean up, so there's a useless, unused fifo lying around. ftrigw_notify() will then report, via its return code, that there was a problem; and just in case it has the rights to do so (which is most of the time), it tries and unlink the stale fifo, which cleans things up for the next notification, and makes a manual invocation of s6-cleanfifodir unnecessary. Simple and efficient. The Unix mkfifo() (or mknod()) system call just creates the fifo in the filesystem. It does not open it. In order to open a fifo to read on, you need to mkfifo() then open(). See where I'm going here? If you "mkfifo $fifo" then "cat < $fifo", and a notification just happens to arrive in between the two, ftrigw_notify() will see a fifo that has been created but is without a reader, and assume it's stale. Be
Re: "Back off" setting for crashing services with s6+openrc?
I feel like this whole thread comes from mismatched expectations of how s6 should behave. s6 always waits for one second before two successive starts of a service. This ensures it never hogs the CPU by spamming a crashing service. (With an asterisk, see below.) It does not wait for one second if the service has been started for more than one second. The idea is, if the service has been running for a while, and dies, you want it up _immediately_, and it's okay because it was running for a while, and either was somewhat idle (in which case you're not hurting for resources) or was hot (in which case you don't want a 1-second delay). The point of s6 is to maximize service uptime; this is why it does not have a configurable backoff mechanism. When you want a service up, you want it *up*, and that's the #1 priority. A service that keeps crashing is an abnormal condition and is supposed to be handled by the admin quickly enough - ideally, before getting to production. If the CPU is still being hogged by s6 despite the 1-second delay and it's not that the service is running hot, then it means it's crashing while still initializing, and the initialization takes more than one second while using a lot of resources. In other words, you got yourself a pretty heavy service that is crashing while starting. That should definitely be caught before it goes in production. But if it's not possible, the least ugly workaround is indeed to sleep in the finish script, and increasing timeout-finish if needed. (The "run_service || sleep 10" approach leaves a shell between s6-supervise and run_service, so it's not good.) ./finish is generally supposed to be very short-lived, because the "finishing" state is generally confusing to an observer, but in this case it does not matter: it's an abnormal situation anyway. There is, however, one improvement I think I can safely make. Currently, the 1-second delay is computed from when the service *starts*: if it has been running for more than one second, and crashes, it restarts immediately, even if it has only been busy initializing, which causes the resource hog OP is experiencing. I could change it to being computed from when the service is *ready*: if the service dies before being ready, s6-supervise *always* waits for 1 second before restarting. The delay is only skipped if the service has been *ready* for 1 second or more, which means it really serving and either idle (i.e. chill resource-wise) or hot (i.e. you don't want to delay it). Does that sound like a valid improvement to you folks? Note that I don't think making the restart delay configurable is a good trade-off. It adds complexity, size and failure cases to the s6-supervise code, it adds another file to a service directory for users to remember, it adds another avenue for configuration mistakes causing downtime, all that to save resources for a pathological case. The difference between 0 second and 1 second of free CPU is significant; longer delays have diminishing returns. -- Laurent
Re: ftrig pipe naming convention
I wonder what is the reason behind the naming convention? What is the downside of simply writing to any present fifo file ? It could work like you're suggesting. But : - checking the type of a file is an additional fstat() system call - there may be reasons in the future to store other files in the fifodir that do not receive the event - it is nice to detect stale fifos, if any, and delete them as soon as you can (#L39), and you don't want to delete unrelated files - but most importantly: creating a fifo in a fifodir that allows you to receive events without a race condition, which is the whole point of the ftrig library, is slightly more complex to do safely than just "mkfifo event/foobar", and I don't want people to think that this is the API. No, the API is ftrigr_subscribe(), and everything under it is implementation details. Restricting the naming is a way of ensuring (as much as possible) that the fifos were indeed created by the appropriate programs. Don't create fifos willy-nilly in a fifodir, and since you found the naming convention, don't use it to work around the check to create your fifos outside of ftrigr_subscribe(). If you do, it will work, until the time when it doesn't, and it will be a complete PITA to debug. -- Laurent
Re: Trouble starting services on boot with s6-rc
I am a bit ashamed to admit I cannot find the logs. From reading https://wiki.gentoo.org/wiki/S6_and_s6-rc-based_init_system#logger I thought maybe I should be looking for file /run/uncaught-logs but could not find any such file in my docker instance(I understand, docker is not Gentoo). By default, s6-overlay containers don't have a catch-all logger, so all the logs fall through to the container's stdout/stderr. You can have an in-container catch-all logger that logs to /run/uncaught-logs if you define S6_LOGGING to 1 or 2. While the docs did speak a lot to the directory structure used by s6, I still am finding it quite hard to figure out what the default directories are for some things. (e.g. I was clear on where my uncompiled s6-rc service directories should go but they seemed to "magically" get complied on boot and show up in a scan_dir) That's a thing with s6, and even s6-rc: it does not define policies, but only mechanism. In other words: it lets you place things wherever you want, there's no default. Of course, higher-level software that uses the s6 bricks needs to define policies in order to get things done; that's why s6-overlay is a set of scripts putting directories in certain places and calling s6 binaries with certain arguments and options. s6-overlay uses s6-rc-compile under the hood, so it compiles your database for you at boot time, to keep things as simple as possible (even though it's not the optimal way of doing it). One additional item. As seems like not a great idea to smash the timeout for all services. Is there any way to adjust it on a per service basis? If not consider me a +1 to kindly add it to a wishlist somewhere. You can define timeout-up and timeout-down files in your s6-rc source definition directories, they do just that. :) n.b. You should put up a BTC lightning "tip bucket" somewhere :) Thank you. Please see this Twitter thread: https://twitter.com/laurentbercot/status/1209247040657674240 -- Laurent
Re: Trouble starting services on boot with s6-rc
Hoping this is the right place to ask for some help as I am very new to s6 and not well versed on any init system. s6-overlay questions are normally asked in the "Issues" section of the s6-overlay GitHub repository, but since yours are really s6-rc questions, it's fine :) (even though the answers are related to s6-overlay!) I can see that my services are registered with s6-rc [Aug 19 18:43:04 root@s6 ~]# s6-rc-db list all | grep -e longrun_test -e sleeponstart sleeponstart longrun_test That proves your services are *in the database*, not that they're going to get started. To make sure your services are started on boot, you should also check that they're declared in the "user" bundle (that's the bundle where users declare what services are part of the final machine state, by s6-overlay policy). $ s6-rc-db contents user | grep -e longrun_test -e sleeponstart But since they *are* started when you boot your container, it means they're indeed declared there, so that's not the issue. The real issue is here: root 34 0.0 0.019252 pts/0S+ 18:53 0:00 /package/admin/s6-2.11.1.0/command/s6-sudoc -e -t 3 -T 4999 -- up 3 The "-T 4999" part means there's a 5 second timeout on running your sleeponstart service. And since this service is waiting for 30 seconds before succeeding, and 30 > 5... it means that it times out after 5 seconds and gets killed, so the transition fails, and longrun_test (which depends on it) is never brought up. s6-overlay-specific section: - You should see all that happening in your container's logs - s6-rc prints informative messages when something fails. Unless you have changed the S6_VERBOSITY value from the default 2 to something lower. - The 5 seconds timeout likely comes from the fact that you have not modified the S6_CMD_WAIT_FOR_SERVICES_MAXTIME value, which is 5000 by default. As is written in both the README[1] and the migration guide[2] :P If your real service needs it, you can disable the timeout by adding to your Dockerfile: ENV S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 or you can adjust it to a bigger value than 5000. Good luck, -- Laurent [1]: https://github.com/just-containers/s6-overlay/blob/master/README.md [2]: https://github.com/just-containers/s6-overlay/blob/master/MOVING-TO-V3.md
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
That would just move 3 components to another level but they are still needed: scanning existing service directories, diffing between desired and current state and applying - so creating or removing directories. So, diffing between desired and current state, and applying the modifications are components of a *service manager*, not a supervision suite, and it is important to maintain the distinction in order to avoid scope creep in s6. Even when a service is *not* instanced, these components are somewhat needed; it's just not noticed because their implementation over a single supervised service is trivial. But it is important to remember that the job of a supervision suite is to maintain the service in its current state (up or down), *not* to manage the wanted state or apply it. (Of course, it does provide tools to perform state transitions for longruns, but it comes with no policy on when to call these tools.) The components you want definitely have their place in s6-rc; but in the meantime, they can also be scripted on top of regular s6 if you have a good modelization for implementing instances, which I will add in the near future. I see there a problem with multiple dynamic services. I'm not sure about concurrency behaviour of updating processes in the service directory. Maybe Laurent can explain problems in that area, if they exist. s6 manages processes and every supervised process needs its own service directory. There will be as many service directories as they are instances. (Some components of a template service directory can of course be reused.) So there's no concurrency issue; however, the instance management tool I'm thinking of could adopt various updating methods depending on what you want. Best effort? Clean shutdown, service replacement, then firing up of the new service's instances? Rolling upgrade across the instances? These policies all have their uses. I'm not sure how complex the supervision itself is - however I would love to solve the problem without doing supervision on my own. I thought about your approach as well but it really depends how resilient an update process is. It will definitely be resilient, but there are several ways to implement it, see above. -- Laurent
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
- we need an scanning component for the desired state of running instances (something like 'find /etc/openvpn -name "*conf"') - we need an scanning component for the current state in process list - we need a diffing component - we need a state applier component That sounds very much like what is planned for s6-rc v1, so I think you will like it when it drops - but it won't be in the near future. However, I have some ideas for new s6 tools that wouldn't follow this model directly but would make it easy for users to create and delete new instance models, and add/remove instances - so your components could be implemented over these tools by simple shell scripts. I'll try to work on that soon. -- Laurent
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
I'm looking for a pattern to solve a problem, where you have to discover dynamically the services you have to start. Examples could be VPN configurations, where you discover the configuration files and start for every file an instance of the VPN service. Hi Oliver, Dynamic instantiation is a real pain point - it's an often requested feature, but it's surprisingly hard to make it work correctly and safely in a supervision scheme. Supervision works very well in static environments, but dynamic discovery is at odds with the architecture. I have a few ideas to mitigate that and help people create instanced services. Instantiation is a planned feature of the future s6-rc v1 but it's still a ways away; I am also thinking of adding tools to help people handle instances with regular s6, amd they may come in the near future, but there are currently no such helpers, sorry. -- Laurent
Re: Be prepared for the fall of systemd
What do we as a community need to do to get S6 into a "corporate friendly" state? What can I do to help? "Corporate-friendly" is not really the problem here. The problem is more "distro-friendly". Distributions like integrated systems. Integrated systems make their lives easier, because they reduce the work of gluing software pieces together (which is what distros do). Additionally, they like stuff like systemd or openrc because they come with predefined boot scripts that, more or less, work out of the box. There are two missing pieces in the s6 ecosystem before it can be embraced by distributions: 1. A service manager. That's what's also missing from runit. Process supervisors are good, but they're not service managers. You can read why here[1]. In case you missed it, here is the call for sponsors I wrote last year, explaining the need for a service manager for s6: [2]. It has been answered, and I'm now working on it. It's going very slowly, because I have a lot of (easier, more immediately solvable) stuff to do on the side, and the s6-rc v1 project is an order of magnitude more complex than what I've ever attempted before, so it's a bit scary and needs me to learn new work habits. But I'm on it. 2. A high-level, user-friendly interface, which I call "s6-frontend". Distros, and most users, like the file-based configuration of systemd, and like the one-stop-shop aspect of systemctl. s6 is lacking this, because it's made of several pieces (s6, s6-linux-init, s6-rc, ...) and more automation-friendly than human-friendly (directory-based config instead of file-based). I plan to write this as well, but it can only be done once s6-rc v1 is released. Once these pieces are done, integration into distributions will be *much* easier, and when a couple distros have adopted it, the rest will, slowly but surely, follow suit. Getting in is the hard part, and I believe in getting in by actually addressing needs and doing good technical work more than by complaining about other systems - yes, current systems are terrible, but they have the merit of existing, so if I think I can do better, I'd rather stfu and do better. Here are some ideas: - easier access to the VCS (git, pijul, etc) The git repositories are public: [3] They even have mirrors on github. All the URLs are linked in the documentation. I don't see how much easier I can make it. Note that the fact that it's not as easy to submit MRs or patches as it is with tools like gitlab or github is intentional. I don't want to be dealing with an influx of context-free MRs. Instead, if people want to change something, I'd like *design discussions* to happen on the ML, between human beings, and when we've reached an agreement, I can either implement the change or accept a patch that I then trust will be correctly written. It may sound dictatorial, but I've learned that authoritarian maintainership is essential to keeping both a project's vision and its code readability. - Issue tracking system The supervision ML has been working well so far. When bigger parts of the project (s6-rc v1 and s6-frontend) are done, there may be a higher volume of issues, if only because of a higher volume of users, so a real BTS may become an asset more than a hindrance at some point. We'll cross that bridge when we get to it. - CI/CD build chain (being careful not to make it too painful to use) Would that really be useful? The current development model is sound, I think: the latest numbered release is stable, the latest git head is development. The s6 ecosystem can be built with a basic configure/make/make install invocation, is it really an obstacle to adoption? I understand the need for CI/CD where huge projects are concerned, people don't have the time or resources to build these. I don't think the s6 ecosystem qualifies as a huge project. It won't even be "huge" by any reasonable metric when everything is done. It needs to be buildable on a potato-powered system! - "idiot proof" website - quick start / getting started guide - easier access (better?) Documentation I file these three under the same entry, which is: the need for community tutorials. And I agree: the s6 documentation is more of a reference manual, it's good for people who already know how it all works but has a very steep learning curve, and beginner-to-intermediate tutorials are severely lacking. If the community could find the time to write these, it would be a huge help. Several people, myself included, have been asking for them for years. (For obvious reasons, I can't be the one writing them.) Turns out it's easier to point out a need than to fulfill it. It's the exact same thing as the s6 man pages. People can whine and bitch and moan for ages saying that some work needs to be done, but when asked whether they'll do it, suddenly the room is deathly silent. For the man pages, one person eventually stepped up and did the work[4] and I'm forever grateful to them; I h
Re: [DNG] Be prepared for the fall of systemd
I find it symptomatic of the fact that a guy wrote some Rube Goldberg code and a corporation decided it would be a great idea to spend millions getting the Rube Goldberg code into many major distros. As far as us running our of road with the Unix API, systemd solves no problem and offers no improvement that couldn't have been solved or improved in a dozen different ways, almost all of which would have been more modular and less prone to vendor lock in. We know all of this. Please keep these posts out of the supervision ML. It's not that we don't like systemd-bashing, it's that it takes a lot of space and makes a lot of noise, and I'd like the supervision ML to be laser-focused on solving technical issues with supervision systems, not be yet another place where we complain about systemd day in day out. Thanks. -- Laurent
Re: runsvdir does not run runsv recursively, as is documented.
I'm trying to set up services, which are in subdirectories of other services. This is supported, according to the second paragraph of the runsvdir man page: runsvdir starts a runsv(8) process for each subdirectory, or symlink to a directory, in the services directory dir, up to a limit of 1000 subdirectories, In my directory service/ I have: Foo Foo/run Foo/bar Foo/bar/run That's not what the man page means. The "subdirectories" are subdirectories *of dir*. runsvdir will understand service/Foo, and service/bar, but it will not understand service/Foo/bar. The exception is if you have a service/Foo/log directory, but that's handled by runsv, not runsvdir: service/Foo/log is a logger for service/Foo. -- Laurent
[announce] skarnet.org Summer 2022 release
Hello, New versions of some skarnet.org packages are available. skalibs has undergone a major update, mostly to yet again revamp librandom. This time I am happy with the API and implementation: I believe it finally addresses all the cases in a satisfying way, providing cross- platform failure-free pseudorandom number generation with options to choose between waiting until the entropy pool has been initialized and possibly getting less cryptographically secure data if the entropy pool is too shallow. It wasn't easy to design; it's here at last. Compatibility with previous skalibs version is not assured, but apart from librandom, and one additional function, no other interface has been modified, so the compatibility breaks are minimal and a lot of software will still build with this version without needing any modification. Most of the rest of the skarnet.org software stack has undergone at least a release bump, in order to build with the new skalibs; a large part of it has also received some changes and fixes. Some packages did not need changing at all: no release is provided for these, they should keep building with the new stack. execline comes with a quality-of-life parser change: backslashes at the end of lines are now ignored, which makes it possible to directly copy some multiline commands from shell scripts. s6-linux-utils comes with a new utility, rngseed, which is an original implementation of Jason Donenfeld's seedrng[1]. This is the work that made it necessary to get librandom right once and for all. With rngseed, no Linux system should ever have uninitialized entropy pool problems ever again. The new versions are the following: skalibs-2.12.0.0 (major) utmps-0.1.2.0 (minor) execline-2.9.0.0 (major) s6-2.11.1.1 (release) s6-rc-0.5.3.2 (release) s6-linux-init-1.0.8.0 (minor) s6-portable-utils-2.2.5.0 (minor) s6-linux-utils-2.6.0.0(major) s6-dns-2.3.5.4(release) s6-networking-2.5.1.1 (release) mdevd-0.1.5.2 (release) dnsfunnel-0.0.1.4 (release) Details of some of these package changes follow. * skalibs-2.12.0.0 - librandom rewritten. random_init and random_finish functions removed. The new random_buf function, which replaces random_strin), never fails. It blocks if the entropy pool is not initialized; the new random_buf_early function is the same, but does not block. random_devurandom is now exported, but should not be needed except in very specific cases (rngseed). - New functions added: waitn_posix and waitn_reap_posix, openc*_at. - readnclose is now exported. - openreadnclose_at() now returns an ssize_t, aligning with openreadnclose(). You should check your code for any use of openreadnclose_at(), and adapt it to the new API. (Previously it returned a size_t and the user was supposed to assume an error if it didn't fill the entire length of the buffer. Now errors are reported with -1.) - Endianness conversion primitives reworked. The nonportability of endian.h and bswap has always been a pain point; the new portable functions in skalibs should now be just as efficient as the system-dependent endian.h functions. - Added an implementation of the blake2s hash. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * utmps-0.1.2.0 - - Nothing to do with the new skalibs; utmps-0.1.2.0 has been available for a while, but was never properly announced. The main feature is that utmps-wtmpd can now take an argument naming its database file. This is useful for implementing btmp, one of the numerous idiosyncrasies of historic Linux software. https://skarnet.org/software/utmps/ git://git.skarnet.org/utmps * execline-2.9.0.0 - Bugfixes. - The execlineb parser has been rewritten and its transition table is now documented. - The wait command can now wait for *one* of the listed processes, in addition to its original capability of waiting for *all* of them. It can also stop waiting after a timeout. The new features can be used even when wait is used in posix mode. https://skarnet.org/software/execline/ git://git.skarnet.org/execline * s6-linux-init-1.0.8.0 - - The system scandir is now configurable at compile-time via the --scandir configure option. It is a relative path under the tmpfsdir. The default is still "service", for a /run/service default scandir. https://skarnet.org/software/s6-linux-init/ git://git.skarnet.org/s6-linux-init * s6-portable-utils-2.2.5.0 - - s6-test now understands the =~ operator, matching its left argument against an extended regular expression given as its right argument (this is originally a GNU bash extension to test). https://skarnet.org/software/s6-portable-utils/ git://git.skarnet.org/s6-portable-utils * s6-linux-utils-2.6.0.0 -- - New command: rngsee
Re: Unprivileged Shutdown?
I have been using simple privilege escalation to poweroff the machine, but looking through the source code for s6-linux-init-shutdownd and friends, it appears the only constraint on interacting with the daemon is the permissions on run-image/service/s6-linux-init-shutdownd/fifo. The default appears to be: 600 root root I've changed it on my system to be: 620 root power and added my user to the power group. This seems like the cleanest way to implement unprivileged poweroff/reboot, but I'm concerned that it's not possible by default. Is there a better way, or is it just meant to be done manually? No, you are correct that it is the right mechanism. Allowing unprivileged shutdown is a dangerous operation and should only be done is very specific circumstances (i.e. when a normal user has complete seat and console access), so it's not the default and the mechanism is fairly hidden. If there's demand, I can probably write a s6-linux-init-shutdown-perms program in a future version that would let you specify the user/group allowed to shutdown, rather than having you manually tinker with the fifo. -- Laurent
Re: s6 xinit replacement?
Is the purpose of executing setsid() in s6-supervise to allow for the services to continue beyond the termination of the supervision tree? It's actually the opposite: it's to protect the supervision tree against misbehaved services. :) setsid() makes sure the service is isolated, and a killpg() or equivalent won't affect anything outside of it. Of course, it also protects *other* services running under the same supervision tree. If that's the case, could there possibly be a flag to disable that, with the understanding that something like nohup or even s6-setsid would be necessary to replicate that behavior? That would enable a non-root Xorg to be managed by s6. The direction s6 has taken is really the opposite: there was such a flag in earlier versions, but it was causing a lot of special cases and problems I definitely did not want to deal with. The real issue is that a supervision tree should not be run with a controlling terminal. It's not meant to be run from a logged-in user process, but as a background infrastructure that's always there. The whole point of s6 is to make your services *more reliable*; and there are few things less reliable than a whole tree of processes that can die on an accidental ^C. Because users insisted a lot, there are still accommodations for killing a whole supervision tree with ^C when s6-svscan has been launched in a terminal. It is a nice feature to have (and although it was by design that services persisted beyond the ^C, it was unintuitive to most users, so from a pure UI standpoint, killing the entire tree in one go was better). However, I'm not going back on the "each service runs in its own session" thing, because if there's a case for allowing the user who controls s6-svscan to kill the whole tree at once, there is just no case for allowing a service running under the tree to impact other services and the tree itself. Despite this, you're right that the pattern of xinit is similar to what s6 does, and it *is* possible to run Xorg under s6; several users are doing so, and I hope they will post their setup. (You may have more luck by asking in the IRC channel, but it would be nice for the setup to be documented on the mailing-list.) It does not involve running s6-svscan from your own VT; it involves having a supervision tree already running (as your user), and starting the Xorg service, e.g. with a s6-svc -u command, on a given VT, possibly passed to the run script via a file that your xinit emulation script would fill in with the output of `tty`. If you insist on doing hacky things, you could even probably get away with something that looks like: xinit emulation: #!/bin/sh tty | sed s:/dev/tty:: > /home/me/.screen-number s6-supervise /home/me/Xorg & # small race here that disappears when s6-supervise has already run once s6-svwait -U /home/me/Xorg your-X-client s6-svc -dx /home/me/Xorg /home/me/Xorg/notification-fd: 3 /home/me/Xorg/run: #!/bin/execlineb -P backtick -E screen { cat /home/me/.screen-number } export DISPLAY :$screen X -displayfd 3 :$screen vt$screen -displayfd is used as a notification mechanism, unblocking s6-svwait when the X server is ready. Hope this helps (and I hope users who actually have done something similar will share their experience), -- Laurent
Re: Supervision on the BSD's
In searching, I found some messages on the Skaware lists about running s6 as PID 1 on FreeBSD; has that work been published anywhere? I'm not sure if I want to go so far as replacing PID 1 right out of the gate, but having some existing service directories would be nice. I have done some experiments and my conclusion was that to replace pid 1 on FreeBSD, a real s6-freebsd-init package was needed, because the way the BSDs organize their init and shutdown is radically different from the way Linux does it, and the conversion is far from obvious. However, you don't need to replace pid 1 to run s6 on a BSD. As mentioned in https://skarnet.org/software/s6/s6-svscan-not-1.html , you can start a supervision tree from /etc/ttys, and run your services under it. It will work like on any other system. Quite a few people on the #s6 channel on IRC (OFTC network) are using s6 on a BSD, so if you're looking for example service directories, and various tips and tricks, I suggest you join the channel and ask them. ;) Have I correctly understood how daemons/services work on the BSD's? If not, what am I missing? Are the daemons included with the distributions so incredibly stable that they don't need supervision in order to keep the system functional? The BSDs are tightly integrated systems, more than "distributions", and especially with OpenBSD, daemons are carefully audited and patched so they are indeed super stable. Which is a very good thing - but because of that, the BSD community tends to look down on supervision, without understanding that it has other benefits than auto-restarting crashed daemons. Finally, if you wanted to create a router that you could (metaphorically) put in a closet and forget about for 5 years, what approach would you take? My initial thought was OpenBSD + s6, but I worry now that there could be an impedance mismatch between these systems. OpenBSD + s6 will definitely work. Just make sure not to get in the way of how OpenBSD does things; run an s6 supervision tree at boot time and start your services under it as you see fit, independently from OpenBSD's rc. Since the BSDs don't care for supervision, though, depending on upstreams it may be difficult to find options for your packaged daemons that stop autobackgrounding and that are not debugging options. Just a small practical hurdle, but when it happens it can be infuriating. -- Laurent
Re: s6-linux-init and ttys in lxc containers
s6-linux-init: warning: unable to ttyname stdout: No such device I suspect this is due to the mechanism described on https://github.com/lxc/lxd/issues/1724#issuecomment-194412831, although I’m not using LXD, only lxc (which does not have a daemon running as root). You're right, it's the exact same issue. If /proc/1/fd/1 does not point to a valid pts in the container, then ttyname() will be unable to resolve it and you will get that warning message - and you also will be unable to transmit your controlling terminal to stage 2 (which is not a problem in your case). The only solution, I'm afraid, is to make sure your pseudo-filesystems are correctly prepared before you launch your container, which you may not be able to do without root privileges. Docker can do it; if you were running lxd there would probably be a way to do it as well. But if you have no use for a controlling terminal, you can forget about all this and just ignore the warning. -- Laurent
[announce] Small skarnet.org update
Hello, New versions of some skarnet.org packages are available. The changes are minor, mostly quality-of-life and small additions driven by the new version of s6-overlay. There are bugfixes all around, so users are encouraged to upgrade even if they're not using s6-overlay. The new versions are the following: skalibs-2.11.2.0(minor) execline-2.8.3.0(minor) s6-2.11.1.0 (minor) s6-portable-utils-2.2.4.0 (minor) s6-linux-init-1.0.7.3 (release) - skalibs features better sysdep autodetection when custom compilation flags are provided, and adds an option to envdir_internal() for unlimited-size variable reading. - execline adds the -P and -p options to getpid, to get the parent pid or force the default behaviour. - s6 features world-usable s6-applyuidgid and s6-setuidgid, as well as a new -L option to s6-envdir for unlimited-size variable reading. - s6-portable-utils adds a -N option to s6-dumpenv (add a newline after dumping a variable) for easier reading back via s6-envdir. - s6-linux-init fixes corner cases when used in containers. Enjoy, Bug-reports welcome. -- Laurent
Re: s6-svscan shutdown notification
What's the cleanest way to wait on s6-svscan to shut down after issuing of a SIGTERM (say s6 via-svscanctl -t)? Be its parent, and wait for it. :) On SIGTERM, s6-svscan will not exit until the supervision tree is entirely down, so that will work. If you're not the parent, then you'll have to wait for a notification somehow, but that's easy: When s6-svscan wants to exit, it doesn't exit right away, but tries to exec into the .s6-svscan/finish script. So you have a clear indicator here: when .s6-svscan/finish runs, it means the supervision tree is down. So, for instance, make a finish script that writes a byte in a fifo, and have your jail shutdown script read on that fifo. Something like: .s6-svscan/finish: #!/bin/sh exec echo > /run/blah/fifo shutdown script: #!/bin/sh ... rm -f /run/blah/fifo mkfifo /run/blah/fifo read < /run/blah/fifo & s6-svscanctl -t /run/service wait ... (read on the fifo before running s6-svscanctl, to avoid the small race condition.) Looking at the documentation, my only option appears to be to check if the return code of s6-svscanctl is 100, or maybe to monitor for the existence of .s6-svscan/control (not sure if it's removed on exit). Are there any other ways to monitor s6-svscan? Ew. Don't poll. Use .s6-svscan/finish to do anything you want to do at s6-svscan death time. -- Laurent
Re: s6-log weirdness
I noticed that in some cases s6-log exits cleanly but does not log anything. What's worse, it depends on the message content. Hi Vallo, That's the difference between '!zstd -q' and '!zstd' -q ;) When -q isn't a part of your processor command, but a part of the s6-log command line, it is interpreted as a selection directive, and will filter anything that contains a 'q' character. -- Laurent
Re: [announce] skarnet.org Winter 2021-2022 release
I think trying to explain s6-linux-init + s6-rc through the lens of runit's stages isn't a good idea. Carlos is correct - both here and in his explanation of s6-linux-init stages. When designing s6-linux-init, I kept runit's "stage" terminology because at the time it was a useful framework to see where things should go; but in retrospect, it was probably a mistake, prone to confusion, as exemplified by Steve's message. Even though, functionally, the "stages" in runit and s6-l-i have similarities, the way they're implemented and work under the hood is fundamentally different. runit's stage 1 and stage 2 are similar, their difference is only conventional: traditionally, runit runs all the oneshots in stage 1, and stage 2 is just an execution of runsvdir - but a setup where /etc/runit/1 is empty and all the oneshots are performed in /etc/runit/2 prior to the execution of runsvdir would work as well. s6-linux-init's stage 1 and stage 2 do not have the same similarity. Stage 1 is the early init, running as pid 1; this is only the /sbin/init script produced by s6-linux-init-maker, which executes into the s6-linux-init binary (which sets up the system and execs into s6-svscan); and users are not supposed to do anything with it. Stage 2, on the other hand, is the real boot sequence, running as not-pid-1; it is only run when stage 1 has completed, which means the system has an adequate long-running pid 1 process, a supervision tree, and a catch-all logger - all the basic infrastructure is in place and the services can be started. With s6-linux-init, stage 2 is where all the real work happens; and when the system's boot sequence is complete, the stage 2 script simply exits and the system keeps running until a shutdown command is issued. I want to keep the "stages" framing for s6-linux-init, because I think it is useful: these are qualitatively different parts of the init process. (There's a "stage 3" and a "stage 4" as well, at shutdown time: they're handled by the s6-linux-init-shutdownd daemon.) The framing is actually *more* useful here than in runit, where "stages" are only sequence points and the only real hardcoded meaning is that the system shuts down when /etc/runit/3 exits. The preceding is the best interpretation I could put together from https://skarnet.org/software/s6-rc/overview.html, https://skarnet.org/software/s6-rc/s6-rc.html, and discussions with you. What do I need to do to make the preceding sequence accurate? I don't know, honestly, Steve. At this point I cannot tell whether you're acting in good or bad faith. You seem to be talking about s6 and s6-linux-init, yet only mention the s6-rc documentation. You do not seem to have read the https://skarnet.org/software/s6-linux-init/quickstart.html page, which explains that s6-linux-init-maker is run offline, or the https://skarnet.org/software/s6-linux-init/s6-linux-init.html page, where the "Early preparation" part explains how stage 1 works. You do not seem to have watched my FOSDEM 2017 video at https://archive.fosdem.org/2017/schedule/event/s6_supervision/ where I describe the various duties of an init system and how the components in the s6 suite fill the boxes. Despite your claims to be interested, you have not put in s6 the tenth of the effort you put in runit. It's been going on for ages. You say you haven't paid much attention to the progress of s6, but over the years the fundamentals have not changed, they've been the same for a while now; the truth is that you have never paid much attention to s6 at all. You come to the list once in a while and ask a question that shows you are still lacking a basic understanding of s6, an understanding that comes from two hours of experimenting while having a browser open with 3-4 tabs to the documentation. And then you seem to ignore the answers, and go away until the next time when you come back just as helpless. You are clearly not dumb. So either you are a troll, or you need to get a grip and realize that if you're really interested, you can do the work of looking up and finding the relevant documentation, experimenting, and finally getting the reward of feeling the pieces of the puzzle fall into place, and acquiring the understanding that has eluded you for so long and that you seem to crave. As a technical writer, that is *your job*, so you can make the process easier for other people. s6 is not as complex as you seem to think it is, far from it. There are real, living people who understand how it works, and they're not all acne-ridden nerds living in a basement. The documentation may not be perfect, but it seems to be adequate. It lacks tutorials, yes, but I expect tutorials to be written by *people like you*, who could do a much better job of it than I ever would, if only they'd stop acting like damsels in distress at the first mention of a Unix pipe. And if you're not interested, or simply not enough to really get into it, that's okay too; you just need to own it and s
[announce] skarnet.org Winter 2021-2022 release
Hello, New versions of all the skarnet.org packages are available. The changes are, for the most part, minimal: essentially, the new versions fix a bug in the build system that made cross-building under slashpackage more difficult than intended. Very few people should have been impacted by this bug. Some packages had a few more bugfixes; and some packages have additional functionality. No major update; no compatibility break. The new versions are the following: skalibs-2.11.1.0 (minor) nsss-0.2.0.1 (release) utmps-0.1.1.0(minor) execline-2.8.2.0 (minor) s6-2.11.0.1 (release) s6-rc-0.5.3.0(minor) s6-portable-utils-2.2.3.4(release) s6-linux-utils-2.5.1.7 (release) s6-linux-init-1.0.7.0(minor) s6-dns-2.3.5.3 (release) s6-networking-2.5.1.0(minor) mdevd-0.1.5.1(release) bcnm-0.0.1.5 (release) dnsfunnel-0.0.1.3(release) smtpd-starttls-proxy-0.0.1.1 (release) Dependencies have all been updated to the latest versions. They are not strict: libraries and binaries may build with older releases of their dependencies, although this is not guaranteed. You do not need to recompile your s6-rc service databases. To make use of the new s6-linux-init functionality, however, you will have to recreate your run-image. You do not need to restart your supervision tree, unless you're deleting your old s6 binaries. Details of minor package changes follow. * skalibs-2.11.1.0 - New function: opendir_at() * utmps-0.1.1.0 - New binary: utmps-write, a generic utmp client that can write user-crafted records to the utmp and/or wtmp databases. * execline-2.8.2.0 - New -s option to the case binary, enabling fnmatch() (shell) expression matching instead of regular expression matching. * s6-rc-0.5.3.0 - - Bundle contents are now read in a "contents.d/" subdirectory, one file per content, instead of one per line in a "contents" file. In the same way, service dependencies are now read in a "dependencies.d/" subdirectory, one file per dependency. Old "contents" and "dependencies" files are still supported, but deprecated. This change allows better integration of s6-rc service definitions with package managers. * s6-linux-init-1.0.7.0 - - New -S option to s6-linux-init-maker, forcing a sync on halt even in a container. * s6-networking-2.5.1.0 - - SNI wildcarding is implemented, as well as a workaround for a bearssl bug causing errors on certificate signatures in certain cases. Enjoy, Bug-reports welcome. And happy holidays to you all! -- Laurent
Re: A program that can get exactly the log of a supervised process?
Why not have the grepper listen on the log file directly? You'll need to have a timestamp in the log and know where the log is, but those can be known at the time of writing the service script. There's no such thing as "the log file". There's the log backendS, which can be one or more automatically rotated log files in a directory, status files, network backends, etc. The only way to access the log stream in a reliable and generic way is to read it on the way to the backends. You have no control over what happens later on. -- Laurent
Re: A program that can get exactly the log of a supervised process?
Well, I do realise the lifespan issue of the loggrep program, which is why I asked the question in the first place. But I really never thought of directly inserting loggrep into the logging chain as a new node; instead, what I have thought is making loggrep a program "attachable" to the logger. That is, the logger is extended with a listener which at least one client can connect to, and which upon connection tees the log to the client. I do not know whether you have similar ideas. Well in theory you could have something like skabus-dyntee ( https://skarnet.org/software/skabus/skabus-dyntee.html ) as your logger, and have your "real" logger run as a skabus-dyntee client. Then you could add a loggrep as a second skabus-dyntee client, and it would just disappear when it has finished its work. It would be a little unreliable as is, because skabus-dyntee doesn't care if it has no client at all, so if the real logger dies, it won't block the log stream until the real logger has restarted, so you may lose logs. But apart from that, it would work. A really reliable solution would be having a skabus-dyntee equivalent that has one permanent output and blocks as long as that output isn't there. As you say, it would be a logger extended with a listener. Another question is how to piggyback loggrep into the notification mechanism: if loggrep is tied to the logger and not to the service, it doesn't have native access to the notification pipe. That means a specific mechanism is needed to give it cross-service access. That's definitely a lot of code and a lot of architectural convolutions to accommodate what is ultimately a daemon misdesign. But it's probably the least bad way to do it, so I might think about it more and add something like that to s6 in the distant future. -- Laurent
Re: A program that can get exactly the log of a supervised process?
Any idea on how the log "teeing" may be done cleanly (and portably if possible; something akin to `tail -f' seems unsuitable because of potential log rotation), and perhaps any flaw or redundancy in the design above? The obstacle I have always bumped against when trying to do similar things is that the teeing program always has to remain there, even after it has done its job (it has read the readiness line and informed the supervisor). And so, instead of "service | logger", your data flow permanently becomes "service | loggrep | logger" before readiness, and "service | cat | logger" after readiness (the best loggrep can do is exec into cat, or reimplement its functionality, once it has read its readiness line). That wouldn't be a huge performance problem, especially if "cat" can do zero-copy data, but it is definitely a reliability problem: - loggrep must die when the service dies, so a new loggrep can be run when the service runs again. So loggrep cannot be run as a separate supervised service in the same service pipeline. (If loggrep were to restart independently from service, it would need to check whether service is ready, and run as 'cat' if it is. This is doable, but more complex.) - That means that either the pipe between service and loggrep cannot be held, or loggrep must have an additional notification that service died. This is, again, doable, but more complex. - If loggrep isn't supervised, and the pipe isn't being held, then killing loggrep will incur a broken pipe, which means a service restart with a lost line of logs, which supervision aims to avoid. So basically, either loggrep is a simple tee-like program but you weaken the supervision properties of the service, or the functionality needs to be embedded in the supervision architecture, with loggrep being a consumer for service and a producer for logger (which is easy with s6-rc but not so much with pure s6) and loggrep always watching the state of service (which is not so easy with s6-rc, where you don't know the full path to another service directory). In short: practical issues. It's impossible to do that in a clean, satisfying way. And it entirely makes sense that it's so difficult, because the very idea is to use the *data flow* to inform the *control flow*, and that is inherently dangerous and not planned for in supervision architectures. Making your control flow depend on your data flow is not a good pattern at all, and I really wish daemons would stop doing that. I'm sorry I don't have a better answer. -- Laurent
Re: logging services with shell interaction
we have a fair number of services which allow (and occasionally require) user interaction via a (built-in) shell. All the shell interaction is supposed to be logged, in addition to all the messages that are issued spontaneously by the process. So we cannot directly use a logger attached to the stdout/stderr of the process. I don't understand the consequence relationship here. - If you control your services / builtin shells, the services could have an option to log the IO of their shells to stderr, as well as their own messages. - Even if you cannot make the services log the shell IO, you can add a small data dumper in front of the service's shell, which transmits full-duplex everything it gets but also writes it to its own stdout or stderr; if that stdout/err is the same pipe as the stdout/err of your service, then all the IO from the shell will be logged to the same place (and log lines won't be mixed unless they're more than PIPE_BUF bytes long, which shouldn't happen in practice). So with that solution you could definitely make your services log to multilog. procServ is a process supervisor adapted to such situations. It allows an external process (conserver in our case) to attach to the service's shell via a TCP or UNIX domain socket. procServ supports logging everything it sees (input and output) to a file or stdout. That works too. IOC=$1 /usr/bin/procServ -f -L- --logstamp --timefmt="$TIMEFMT" \ -q -n %i --ignore=^D^C^] -P "unix:$RUNDIR/$IOC" -c "$BOOTDIR" "./$STCMD" \ | /usr/bin/multilog "s$LOGSIZE" "n$LOGNUM" "$LOGDIR/$IOC" ``` So far this seems to do the job, but I have two questions: 1. Is there anything "bad" about this approach? Most supervision tools have this sort of thing as a built-in feature and I suspect there may be a reason for that other than mere convenience. It's not *bad*, it's just not as airtight as supervision suites make it. The reasons why it's a built-in feature in daemontools/runit/s6/others are: - it allows the logger process to be supervised as well - it maintains open the pipe to the logger, so service and logger can be restarted independently at will, without risk of losing logs. As is, you can't send signals to multilog (useful if you want to force a rotation) without knowing its pid. And if multilog dies, it broken pipes procServ, and it (and your service) is probably forced to restart, and you lose the data that it wanted to write. A supervision architecture with integrated logging protects from this. 2. Do any of the existing process supervision tools support what procServ gives us wrt interactive shell access from outside? Not that I know of, because that need is pretty specific to your service architecture. However, unless there are more details you have omitted, I still believe you could obtain the same functionality with a daemontools/etc. infrastructure and a program recording the IO from/to the shell. Since you don't seem opposed to using old djb programs, you could probably even directly reuse "recordio" from ucspi-tcp for this. :) -- Laurent
Re: Service watchdog
Yes, in my usecase this would be used at the place where sd_notify() is used if the service runs under systemd. Then periodically executed watchdog could check the service makes progress and react if it doesn't. The question is how to implement the watchdog then - it could be either a global service or another executable in service directory, which would be started periodically by runsv. If a single notification step is enough for you, i.e. the service goes from a "preparing" state to a "ready" state and remains ready until the process dies, then what you want is implemented in the s6 process supervisor: https://skarnet.org/software/s6/notifywhenup.html Then you can synchronously wait for service readiness (s6-svwait $service) or, if you have a watchdog service, periodically poll for readiness (s6-svstat -r $service). But that's only valid if your service can only change states once (from "not ready" to "ready"). If you need anything more complex, s6 won't support it intrinsically. The reason why there isn't more advanced support for this in any supervision suite (save systemd but even there it's pretty minimal) is that service states other than "not ready yet" and "ready" are very much service-dependent and it's impossible for a generic process supervisor to support enough states for every possible existing service. Daemons that need complex states usually come with their own monitoring software that handles their specific states, with integrated health checks etc. So my advice would be: - if what you need is just readiness notification, switch to s6. It's very similar to runit and I think you'll find it has other benefits as well. The drawback, obviously, is that it's not in busybox and the required effort to switch may not be worth it. - if you need anything more complex, you can stick to runit, but you will kinda need to write your own monitor for your daemon, because that's what everyone does. Depending on the details of the monitoring you need, the monitoring software can be implemented as another service (e.g. to receive heartbeats from your daemon), or as a polling client (e.g. to do periodic health checks). Both approaches are valid. Don't hack on runit, especially the control pipe thing. It will not end well. (runit's control pipe feature is super dangerous, because it allows a service to hijack the control flow of its supervisor, which endangers the supervisor's safety. That's why s6 does not implement it; it provides similar - albeit slightly less powerful - control features via ways that never give the service any power over the supervisor.) -- Laurent
Re: Readiness notification using signals
Hi Carlos, I'm supervising an instance of an X Server using s6. X.Org has a built-in readiness notification mechanism: it sends a USR1 to its parent process once it's ready. From what I know, it would be s6-supervise. This... 😱 this is appalling. This mechanism is a terrible way to notify readiness. 1. It forces the program reading readiness information to be the direct parent of the daemon. 2. This is not a concern for X, but if the daemon has dropped privileges, it is very possible that it does not have the rights to signal its parent anymore. 3. It forces the parent to use the siginfo_t structure, and more complex signal management, in order to detect what process is sending the signal. This can work, but is by no means a common use of signals. 4. Signals can be lost if several signals of the same value are received by a process. So if two processes are notifying readiness at the same time to a unique supervisor via SIGUSR1, it is possible that the supervisor only gets one of them. However, after reading the documentation, there doesn't seem to be a way to set up custom USR1 handlers for s6-supervise. Indeed, there isn't. Signals are meant to be a control mechanism from above, not from below: the administrator is supposed to be able to send signals to s6-supervise, but the supervised process *definitely is not*. As far as I know, this leaves me with two, not quite ideal solutions: make the run script spawn X and do readiness notification on its behalf (which lengthens the supervision chain), or poll it (which is what I've been doing). Is there a way to not let the information from this USR1 not "go to waste"? Let this "information" go to waste: it should never be sent in the first place, and if patching were to happen, it should be to delete that chunk of code from the Xorg server. We should consider ourselves lucky that s6-supervise is not using SIGUSR1 for its own purposes and is happily ignoring it as is. Fortunately, there is a different solution for you: the -displayfd option to the X command will print the display number, followed by a newline, to the (argument of -displayfd) file descriptor, when X has found a display. This only happens when X is ready - unless it only starts listening to its socket later, but the window should be super small; and it has the correct format for an s6 notification. So, using displayfd as your notification fd will work. For instance, if you have 3 in your notification-fd file, starting X -displayfd 3 should properly (or *almost* properly) notify readiness. Some people already use that mechanism, and are happy with it. :) Good luck, -- Laurent
Re: [announce] skarnet.org Fall 2021 release
just a minor thing. You probably didn't push to s6-rc repository. I don't see the new v0.5.2.3 version commit and tag. Weird. I must have missed a case with my script. Thanks for the report; it should be fixed now. :) -- Laurent
[announce] skarnet.org Fall 2021 release
Hello, New versions of all the skarnet.org packages are available. skalibs has undergone a major update, with a few APIs having disappeared, and others having changed. Compatibility with previous versions is *not* assured. Consequently, all the rest of the skarnet.org software has undergone at least a release bump, in order to build with the new skalibs. But some packages also have new functionality added (hence, a minor bump), and others also have their own incompatible changes (hence, a major bump). The new versions are the following: skalibs-2.11.0.0 (major) nsss-0.2.0.0 (major) utmps-0.1.0.3 (release) execline-2.8.1.0 (minor) s6-2.11.0.0 (major) s6-rc-0.5.2.3 (release) s6-portable-utils-2.2.3.3 (release) s6-linux-utils-2.5.1.6(release) s6-linux-init-1.0.6.4 (release) s6-dns-2.3.5.2(release) s6-networking-2.5.0.0 (major) mdevd-0.1.5.0 (minor) bcnm-0.0.1.4 (release) dnsfunnel-0.0.1.2 (release) Additionally, a new package has been released: smtpd-starttls-proxy-0.0.1.0 Dependencies have all been updated to the latest versions. They are, this time, partially strict: libraries and binaries may build with older releases of their dependencies, but not across major version bumps. The safest approach is to upgrade everything at the same time. You do not need to recompile your s6-rc service databases or recreate your s6-linux-init run-images. You should restart your supervision tree after upgrading skalibs and s6, as soon as is convenient for you. Details of major and minor package changes follow. * skalibs-2.11.0.0 - A lot of obsolete or useless functionality has been removed: libbiguint, rc4, md5, iobuffer, skasigaction, environ.h and getpeereid.h headers, various functions that have not proven their value in a while. - Some functions changed signatures or changed names, or both. - All custom types ending in _t have been renamed, to avoid treading on POSIX namespace. (The same change has not been done yet in other packages, but skalibs was the biggest offender by far.) - Signal functions have been deeply reworked. - cdb has been reworked, the API is now more user-friendly. - New functions have been added. The deletion of significant portions of code has made skalibs leaner. libskarnet.so has dropped under 190 kB on x86_64. The cdb rewrite on its own has helped reduce an important amount of boilerplate in cdb-using code. All in all, code linked against the new skalibs should be slightly smaller and use a tiny bit less RAM. https://skarnet.org/software/skalibs/ git://git.skarnet.org/skalibs * nsss-0.2.0.0 - Bugfixes. - nsss-switch wire protocol slightly modified, which is enough to warrant a major version bump. - _r functions are now entirely thread-safe. - Spawned nsssd programs are now persistent and only expire after a timeout on non-enumeration queries. This saves a lot of forking with applications that can call primitives such as getpwnam() repeatedly, as e.g. mdevd does when initially parsing its configuration file. - New nsssd-switch program, implementing real nsswitch functionality by dispatching queries to various backends according to a script. It does not dlopen a single library or read a single config file. https://skarnet.org/software/nsss/ git://git.skarnet.org/nsss * execline-2.8.1.0 - Bugfixes. - New binary: case. It compares a value against a series of regular expressions, executing into another command line on the first match. https://skarnet.org/software/execline/ git://git.skarnet.org/execline * s6-2.11.0.0 --- - Bugfixes. - Some libs6 header names have been simplified. - s6-svwait now accepts -r and -R options. - s6-supervise now reads an optional lock-fd file in the service directory; if it finds one, the first action of the service is to take a blocking lock. This prevents confusion when a controller process dies while still leaving workers holding resources; it also prevents log spamming on user mistakes (autobackgrounding services, notably). - New binaries: s6-socklog, s6-svlink, s6-svunlink. The former is a rewrite of smarden.org's socklog program, in order to implement a fully functional syslogd with only s6 programs. The latter are tools that start and stop services by symlinking/unlinking service directories from a scan directory, in order to make it easier to integrate s6-style services in boot scripts for sequential service managers such as OpenRC. https://skarnet.org/software/s6/ git://git.skarnet.org/s6 * s6-networking-2.5.0.0 - - Bugfixes. - minidentd has been removed. It was an old and somehow still buggy piece of code that was only hanging around for nostalgia reasons. - Full support for client certificates. Details of the client certificate are transmitted to the application via e
Re: First time caller to the show - am I understanding the fifo trick correctly?
Forgiving privilege separation failures and minor grammatical mistakes, does it look as if I understand the fifo trick's application in practice? Hi Ellenor, Yes, I think you have the right idea. The goal here is to redirect s6-svscan's own stdout and stderr to the stdin of the catch-all logger process, so that the supervision tree's messages, and the messages from every service that lacks a dedicated logger, go to the catch-all logger instead of /dev/console. (Because /dev/console is a terrible default place to send logs and should only be used for very critical messages such as kernel panics, or, in our userland case, for catch-all logger failures.) The problem is that we want the catch-all logger to run as a service under the supervision tree, so the s6-log process does not exist yet when we exec into s6-svscan: it will be spawned later as a grandchild of s6-svscan (with an s6-supervise intermediary). So we cannot use an anonymous pipe for this. We use a fifo instead: we can redirect init's stdout and stderr to a fifo, and later on, when the catch-all logger starts, we can instruct it (in its run script) to read from the fifo. But the Unix fifo semantics say that we *cannot* open a fifo for writing while there is no reader: open() would either block (default flags) or return -1 ENXIO (with O_NONBLOCK). So the "fifo trick" is: 1. open the fifo for reading 2. open it for writing, which now works 3. close the reading end At this point, any write() to the fifo will fail with -1 EPIPE. That is not a problem per se, except it will also generate a SIGPIPE, so in order to avoid crashing and burning, it is important to ignore SIGPIPE at the very least - or, better, to make sure that no process writes to the fifo until the catch-all logger is up. This is the case for s6-svscan and s6-supervise, so our system structure is safe; but we need to make sure that no other process starts before the catch-all logger is up, else they will just eat a SIGPIPE and die. In the s6-l-i model, s6-svscan is executed as soon as possible, on a very minimal supervision tree that only contains the catch-all logger and a few other essential "early services" (such as the shutdown daemon and an early getty). All the rest of the initialization is done in "stage 2 init", which is a script run as a child of s6-l-i's. So the end of the "fifo trick" uses the Unix fifo semantics as a synchronization mechanism: 4. fork 5. In the child, close our fd to the fifo 6. In the child, open the fifo for writing once again, *without* O_NONBLOCK. This last open() will block until the fifo has a reader. That ensures the child will only resume once the parent has completed its work and executed into s6-svscan, and the supervision tree has started and the catch-all logger is running. Then the child can exec into stage 2 init and perform the rest of the work with the guarantee that the supervision tree is operational and all the stdout and stderr messages go to the catch-all logger by default. To see exactly how to implement stage 1 init and the fifo trick as an execline script, you can checkout (or download) any version of s6-l-i *prior to* 1.0.0.0; try version 0.4.0.1, downloadable from skarnet.org if you type the URL by hand, and accessible via the v0.4.0.1 tag in git. It is very different from what it is now, as in there is no sysv compatibility at all, but stage 1 should be understandable. A few months ago, I tried adding a few conditional compilation options to s6-l-i to make it work under FreeBSD, but unfortunately the organization of the FreeBSD init is so different from Linux's, especially shutdown-wise, that my attempt only succeeded in turning the package into an unholy plate of spaghetti. At some point in the future, however, a similar-but-separate s6-freebsd-init package may make sense. -- Laurent
Re: s6-rc-init verbose equivalent messages?
# /usr/local/bin/s6-rc-init -c /s/comp -l /s/run /s/scan s6-rc-init: fatal: unable to supervise service directories in /s/run/servicedirs: No such file or directory I've completed a disk-disk copy, as I need to integrate s6 into hardenedbsd. Do you have a s6-svscan process running on /s/scan ? You need one before you can run s6-rc-init. If you have one, please get a strace of s6-rc-init, if you can: that will tell exactly what's happening. But this error generally points to a supervision tree that's not set up properly. -- Laurent
Re: S6 Queries
Thanks Laurent for the detailed explanations. We did a bootup speed comparison between S6 and systemd. S6 is able to boot up slightly faster than systemd. Actual result is 4-4.5% faster but we were expecting something near to 20%. Ours is a bit complex setup with more than 140 services (includes a lot of long run services and a lot of dependencies). The main advantage in systemd is, it starts many critical processes very quickly since it has no dependency to logging services. We collect the logs from journalctl and store it in log files. Whereas in S6, the critical services start up is a bit delayed since it has to depend on logging services which in turn depends on other services (responsible for backing up the previous logs). Thank you for these numbers! Indeed they confirm my intuition: booting via s6 is a little faster than systemd, but nothing extraordinary, because s6-rc emphasizes reliability over speed. Your critical services may be slightly delayed, but you have the guarantee that you will not lose logs - whereas with systemd, if a piece of your logging system fails to start, your critical services may already have produced some data that will vanish into the aether. Whether or not that's an acceptable risk is up to you. -- Laurent
Re: S6 Queries
Do you think this is any better? = #!/bin/sh test_for_myrequirement || exit 1 exec mydaemon -myarg1 -myarg2 = This does not accomplish the same thing at all: it does not ensure that myrequirement is at least attempted before mydaemon runs. Instead, it conditions the readiness of mydaemon to that of myrequirement. So, it is "better" in the sense that it does not control another service from a run script, but it is even further from what the OP wants. Any reference to another service in a run script is going to be quirky at best. Managing all kinds of dependencies between services is really best done *outside* of run scripts, which is why s6-rc exists. It does not currently have all the expressive power of systemd for dependencies, but in the future, it will. -- Laurent
Re: S6 Queries
I thought the way to do what the OP asked is: = #!/bin/sh s6-svc -u myrequirement || exit 1 exec mydaemon -myarg1 -myarg2 = This is not a good idea in a s6-rc installation, because it sends raw s6 commands, which may mess with the service state as viewed by s6-rc. Also, it sends control commands to a service from another service's run script, which is bad form in general: it is unintuitive that starting the mydaemon service also causes the myrequirement service to be started. Dependencies should be handled at the service manager level, not at the process supervision level. Of course, you can do that in a pinch for small s6-only installations, where you have no proper dependency engine, but that does not seem to be what the OP is asking. And even then, this does not implement After= because s6-svc -u returns instantly. This only implements Wants=. To implement After=, you would need something like s6-svc -uwU instead, which is not good because it adds the myrequirement readiness delay to the mydaemon readiness delay, so if mydaemon has strict readiness timeouts, it can make it fail. All in all, it's better to avoid controlling another service in a run script: there's always an annoying corner case somewhere. Dependency management is best handled by the external tool that is explicitly supposed to do it: the service manager. -- Laurent
Re: S6 Queries
1. In systemd, the services are grouped as targets and each target depends on another target as well. They start as targets. [ex: Reached local-fs.target, Reached network.target, Reached UI target,...]. Is there any way in S6 to start the init system based on bundles? Yes, that is what bundles are for. In your stage 2 boot script (typically /etc/s6-linux-init/current/scripts/rc.init), you should invoke s6-rc as: s6-rc change top if "top" is the name of your default bundle, i.e. the bundle that contains all the services you want to start at boot time. You can basically convert the contents of your systemd targets directly into contents of your s6-rc bundles; and you decide which one will be brought up at boot time via the s6-rc invocation in your stage 2 init script. 2. Are there any ways to have loosely coupling dependencies? In systemd, we have After=. After option will help the current service to start after the mentioned service (in after). And the current service will anyway start even if the mentioned service in After fails to start. Do we have such loosely coupled dependency facility in S6? Not at the moment, no. The next version of s6-rc will allow more types of dependencies, with clearer semantics than the systemd ones (After=, Requires= and Wants= are not orthogonal, which is unintuitive and causes misuse); but it is still in early development. For now, s6-rc only provides one type of dependency, which is the equivalent of Requires+After. I realize this is not flexible enough for a lot of real use cases, which is one of the reasons why another version is in development. :) 3. Is there any tool available in S6 to measure the time taken by each service to start? We can manually measure it from the logs, but still looking for a tool which can provide accurate data. Honestly, if you use the -v2 option to your s6-rc invocation, as in s6-rc -v2 change top and you ask the catch-all logger to timestamp its lines (which should be the default, but you can change the timestamp style via the -t option to s6-linux-init-maker) then the difference of timestamps between the lines: s6-rc: info: service foo: starting and s6-rc: info: service foo successfully started will give you a pretty accurate measurement of the time it took service foo to start. These lines are written by s6-rc exactly as the "starting" or "completed" event occurs, and they are timestamped by s6-log immediately; the code path is the same for both events, so the delays cancel out, and the only inaccuracy left is randomness due to scheduling, which should not be statistically significant. At the moment, the s6-rc log is the easiest place to get this data from. You could probably hack something with the "time" shell command and s6-svwait, such as s6-svwait -u /run/service/foo ; time s6-svwait -U /run/service/foo which would give you the time it took for foo to become ready; but I doubt it would be any more accurate than using the timestamps in the s6-rc logs, and it's really not convenient to set up. 4. Does the S6 init system provide better boot up performance compared to systemd ? One of our main motives is to attain better bootup performance. Is our expectation correct? The boot up performance should be more or less *similar* to systemd. The code paths used by the s6 ecosystem are much shorter than the ones used by systemd, so _in theory_ you should get faster boots with s6. However, systemd cheats, by starting services before their dependencies are ready. For instance, it will start services before any kind of logging is ready, which is pretty dangerous for several reasons. As a part of its "socket activation" thing, it will also claim readiness on some sockets before even attempting to run the actual serving processes (which may fail, in which case readiness was a lie.) Because of that, when everything goes well, systemd cuts corners that s6 does not, and may gain some advantage. So all in all, I expect that depending on your system, the difference in speed will not be remarkable. On very simple setups (just a few services), systemd's overhead may be noticeable and you may see real improvements with s6. On complex setups with lots of dependencies, s6 might still have the speed advantage but I don't think it will be anything amazing. The real benefit of s6 is that it achieves roughly the same speed as systemd *while being more reliable and predictable*. If you actually make bootup speed comparisons between systemd and s6, please share them! I am interested in that kind of benchmarks, and I'm sure the community would like to see numbers as well. -- Laurent
Re: Query on S6 system shutdown
I believe the finish script is not being called by s6-svc. When I run it manually , the finish script runs and kills the process and graceful shutdown is happening as expected. What may be the cause for not triggering the finish script of critical service. The finish script, which is entirely optional, is not supposed to kill the process. It is not called by s6-svc, and does not run when the process is still alive. It is *not* equivalent to the "down" script of a oneshot. (There is a subtle hint in the fact that the scripts are named "run" and "finish" instead of "up" and "down".) If it exists, the finish script is run by s6-supervise *after* the service dies. The point is to perform any potentially needed cleanups that are not automatically performed at service death. Your current finish script is entirely incorrect and you should delete it. In order to kill your service, you need to send it a signal. Daemons normally have a signal that tells them to gracefully exit; for most daemons, it is SIGTERM, which is what svc -d sends by default. If your critical.service does not die on s6-svc -d, it means that it ignores SIGTERM. Then you need to find out *what* signal it interprets as a request for a graceful shutdown, and put the name of this signal in the "down-signal" file in your service definition directory. Then, when you send s6-svc -d, critical.service will receive the signal you indicated. down-signal can be SIGKILL - this is what you're using at the moment - but if your service is that critical, chances are it's not the best way to kill it and you should look for a gentler signal to send it first. If there are cases where a graceful exit request is ignored or takes too long, and you want to make sure the service dies at some point, you can create a "timeout-kill" file, containing a number of milliseconds - if the process is still alive after that duration after receiving its down-signal, it will be sent a SIGKILL. -- Laurent
Re: Query on s6-log and s6-supervise
I have checked the Private_Dirty memory in "smaps" of a s6-supervise process and I don't see any consuming above 8kB. Just posting it here for reference. Indeed, each mapping is small, but you have *a lot* of them. The sum of all the Private_Dirty in your mappings, that should be shown in smaps_rollup, is 96 kB. 24 pages! That is _huge_. In this list, the mappings that are really used by s6-supervise (i.e. the incompressible amount of unshareable memory) are the following: - the /bin/s6-supervise section: this is static data, s6-supervise needs a little, but it should not take more than one page. - the [heap] section: this is dynamically allocated memory, and for s6-supervise it should not be bigger than 4 kB. s6-supervise does not allocate dynamic memory itself, the presence of a heap section is due to opendir() which needs dynamic buffers; the size of the buffer is determined by the libc, and anything more than one page is wasteful. ( - anonymous mappings are really memory dynamically allocated for internal libc purposes; they do not show up in [heap] because they're not obtained via malloc(). No function used by s6-supervise should ever need those; any anonymous mapping you see is libc shenanigans and counts as overhead. ) - the [stack] section: this is difficult to control because the amount of stack a process uses depends a lot on the compiler, the compilation flags, etc. When built with -O2, s6-supervise should not use more than 2-3 pages of stack. This includes a one-page buffer to read from notification-fd; I can probably reduce the size of this buffer and make sure the amount of needed stack pages never goes above 2. So in total, the incompressible amount of private mappings is 4 to 5 pages (16 to 20 kB). All the other mappings are libc overhead. - the libpthread-2.31.so mapping uses 8 kB - the librt-2.31.so mapping uses 8 kB - the libc-2.31.so mapping uses 16 kB - the libskarnet.so mapping uses 12 kB - ld.so, the dynamic linker itself, uses 16 kB - there are 16 kB of anonymous mappings This is some serious waste; unfortunately, it's pretty much to be expected from glibc, which suffers from decades of misdesign and tunnel vision especially where dynamic linking is concerned. We are, unfortunately, experiencing the consequences of technical debt. Linking against the static version of skalibs (--enable-allstatic) should save you at least 12 kB (and probably 16) per instance of s6-supervise. You should have noticed the improvement; your amount of private memory should have dropped by at least 1.5MB when you switched to --enable-allstatic. But I understand it is not enough. Unfortunately, once you have removed the libskarnet.so mappings, it's basically down to the libc, and to achieve further improvements I have no other suggestions than to change libcs. If possible, can you please share us a reference smap and ps_mem data on s6-supervise. That would really help. I don't use ps_mem, but here are the details of a s6-supervise process on the skarnet.org server. s6 is linked statically against the musl libc, which means: - the text segments are bigger (drawback of static linking) - there are fewer mappings (advantage of static linking, but even when you're linking dynamically against musl it maps as little as it can) - the mappings have little libc overhead (advantage of musl) # cat smaps_rollup 0040-7ffd53096000 ---p 00:00 0 [rollup] Rss: 64 kB Pss: 36 kB Pss_Anon: 20 kB Pss_File: 16 kB Pss_Shmem: 0 kB Shared_Clean: 40 kB Shared_Dirty: 0 kB Private_Clean: 8 kB Private_Dirty:16 kB Referenced: 64 kB Anonymous:20 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped:0 kB FilePmdMapped: 0 kB Shared_Hugetlb:0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked:0 kB You can see 40kB of shared, 16kB of Private_Dirty, and 8kB of Private_Clean - apparently there's one Private_Clean page of static data and one of stack; I have no idea what this corresponds to in the code, I will need to investigate and see if it can be trimmed down. # grep -E '[[:space:]](-|r)(-|w)(-|x)(-|p)[[:space:]]|^Private_Dirty:' smaps 0040-00409000 r-xp ca:00 659178 /command/s6-supervise Private_Dirty: 0 kB 00609000-0060b000 rw-p 9000 ca:00 659178 /command/s6-supervise Private_Dirty: 4 kB 02462000-02463000 ---p 00:00 0 [heap] Private_Dirty: 0 kB 02463000-02464000 rw-p 00:00 0 [heap] Private_Dirty: 4 kB 7ffd53036000-7ffd53057000 rw-p 00:00 0 [stack] Private_Dirty: 8 kB 7ffd5309-7ffd53094000 r--p 00:00 0 [vvar] Private_Dirty: 0 kB 7ffd53094000-7ffd53096000 r-xp 00:00 0 [vdso] Private_Dirty: 0 kB One page of static data, one page of
Re: Suspend s6-log writing to log file and resume it back after sometime
Any pointers on how I can go about this? Is there any hack or tricks that could be done in s6-log to achieve this? Sorry, but no, nothing comes to mind - s6-log was not designed for this. I don't think expecting services to keep running while not logging to disk, whether or not in standby/sleep mode, is reasonable: if logs keep coming up, memory fills up. What do you do if the machine doesn't wake up before the memory is full? The logger will die and you will lose all your carefully accumulated logs. Ideally, you would have dynamic verbosity in the service, and switch it to zero when going into standby/sleep mode, so it would stop producing logs, so you'd never wake up the disk. Of course, unless it's the wakeup event listener, the concept of still having a service running when in standby mode is weird: it defeats the very purpose of standby mode, which is saving energy. The best way to not have your disk spin is to have nothing to make it spin in the first place. :P s6-svc -p all your services when entering standby mode! (Except the wakeup event listener.) :D Sorry for being unhelpful, and good luck, -- Laurent
Re: Query on s6-log and s6-supervise
1. Why do we need to have separate supervisors for producer and consumer long run services? Is it possible to have one supervisor for both producer and consumer, because anyhow the consumer service need not to run when the producer is down. I can understand that s6 supervisor is meant to monitor only one service, but why not monitor a couple of services when it is logically valid if I am not wrong. Hi Arjun, The logic of the supervisor is already complex enough when it has to monitor one process. It would be quadratically as complex if it had to monitor two. In all likeliness, the first impact of such a change would be more bugs, because the logic would be a lot more difficult to understand and maintain. The amount of memory used by the s6 logic itself would not change (or would *increase* somewhat) if the code was organized in a different way in order to reduce the amount of processes, and you would see an overall decrease in code quality. Worsening the design to offset operational costs is not a good trade-off - it is not "logically valid", as you put it. I would not do it even if the high amount of memory consumed by your processes was due to s6 itself. But it is not the case: your operational costs are due to something else. See below. 2. Is it possible to have a single supervisor for a bundle of services? Like, one supervisor for a bundle (consisting of few services)? Again, there would be no engineering benefit to that. You would likely see operational benefits, yes, but s6 is the wrong place to try and get those benefits, because it is not the cause of your operational costs. 3. Generally how many instances of s6-supervise can run? We are running into a problem where we have 129 instances of s6-supervise that leads to higher memory consumption. We are migrating from systemd to s6 init system considering the light weight, but we have a lot of s6-log and s6-supervise instances that results in higher memory usage compared to systemd. Is it fine to have this many number of s6-supervise instances?ps_mem data - 5.5 MiB s6-log (46) , 14.3 MiB s6-supervise (129) It is normally totally fine to have this many number of s6-supervise instances (and of s6-log instances), and it is the intended usage. The skarnet.org server only has 256 MB of RAM, and currently sports 93 instances of s6-supervise (and 44 instances of s6-log) without any trouble. It could triple that amount without breaking a sweat. The real problem here is that your instances appear to use so much memory: *that* is not normal. Every s6-supervise process should use at most 4 pages (16k) of private dirty memory, so for 129 processes I would expect the memory usage to be around 2.1 MB. Your reported total shows 7 times as much, which sounds totally out of bounds to me, and even accounting for normal operational overhead, a factor of 7 is *completely bonkers*. There are two possible explanations here: - Either ps_mem is not accurately tallying the memory used by a given set of processes; - Or you are using a libc with an incredible amount of overhead, and your libc (and in particular, I suspect, dynamic linking management in your libc) is the culprit for the insane amount of memory that the s6-supervise processes seem to be eating. The easiest way to understand what's going on is to find a s6-supervise process's pid, and to perform # cat /proc/$pid/smaps_rollup That will tell you what's going on for the chosen s6-supervise process (they're all similar, so the number for the other s6-supervise processes won't be far off). In particular, look at the Private_Dirty line: that is the "real" amount of uncompressible memory used by that process. It should be around 16k, tops. Anything over that is overhead from your libc. If the value is not too much over 16k, then ps_mem is simply lying to you and there is nothing to worry about, except that you should use another tool to tally memory usage. But if the value is much higher, then it is time to diagnose deeper: # cat /proc/$pid/smaps That will show you all the mappings performed by your libc, and the amount of memory that each of these mappings uses. Again, the most important lines are the Private_Dirty ones - these are the values that add up for every s6-supervise instance. My hunch is that you will see *a lot* of mappings, each using 4k or 8k, or even in some cases 12k, of Private_Dirty memory. If it is the case, unfortunately there is nothing I can do about it, because that overhead is entirely caused by your libc. However, there is something *you* can do about it: - If "ldd /bin/s6-supervise" gives you a line mentioning libs6.so or libskarnet.so, try recompiling s6 with --enable-allstatic. This will link against the static version of libs6 and libskarnet, which will alleviate the costs of dynamic linking. (The price is that the *text* of s6-supervise will be a little bigger, but it doesn't ma
Re: [PATCH] s6-rc-compile: Fix setting of flag-essential
Don't set the bitposition (which is 0 for 'flag-essential') to the flags, but the bit at the position. Ha, nice catch. Applied, thanks! -- Laurent
[announce] #s6 is moving from Freenode to OFTC
Hello, As some of you are aware of, last week, the Freenode IRC network was subjected to a forceful change of its operational control. The history and details of the change are deeply political in nature and very much off-topic for our lists, and I really do not wish this event to be discussed here; there is plenty of information and discussion to be had in other corners of the Internet. The fact remains, however, that Freenode is no longer a safe IRC network for free and open source software communities. As of now, the #s6 channel (and its associated off-topic channel) has moved to the OFTC network: https://www.oftc.net/ The skarnet.org documentation has been updated accordingly. Please update your pointers, and if you're currently hanging on Freenode, please come to OFTC instead. The #s6 channel on Freenode will cease to be maintained, except for a /topic pointing to the new hosting network. At some point, it will be moderated, i.e. you will not be able to write in it anymore. In the same way, there will be an empty #s6 channel on the libera.chat network, pointing to the OFTC network as well. I know that some of you would have preferred to move to libera.chat, the "official" successor of freenode, instead of OFTC. It was not an easy choice to make, and no matter the decision, some people would have ended up unhappy anyway. Please fight the undoubtedly overwhelming urge to explain why the decision was wrong and the other one would have been better; instead, feel free to address all complaints to Andrew Lee, Crown Prince of Korea and originator of all this shitshow. Thank you. I hope to meet you on the other side. -- Laurent
[announce] New hypermail archives of skarnet.org mailing lists
Hello, ezmlm-cgi, the web interface to the archives of the skarnet.org mailing-lists, has been broken for... forever, resulting in an inability to display certain messages. I tried debugging it, but could not find out what was happening within a reasonable amount of time dedicated to it. A new web interface to these archives is now available, and this one appears to work better (there are still a few quirks with utf-8 display but we'll sort them out). https://skarnet.org/lists/skaware/ https://skarnet.org/lists/supervision/ If you are using web links to ML archived messages, please update your links to point to the hypermail archive instead of the ezmlm one. At some point in the (distant) future, the ezmlm archive will be deprecated. Thanks. Enjoy. -- Laurent
Re: S6 logging
I am trying to log the prints coming from daemons like dropbear using s6-log, but couldn't make it. Am I missing something? You really need to improve on your way of asking for help. "couldn't make it" is not an actionable report. You need to say: - exactly what you did (you did that, good, but you did not mention that you were using s6-rc, which may or may not be relevant) - exactly what you expected (you didn't do that, but it is easy to figure out) - exactly what happened (you didn't do that at all and we're left to guess). Going out on a limb, and reading the manual page (I haven't used dropbear in a little while), I'm guessing that the problem is this: /usr/sbin/dropbear -F You also need the -E option in order to tell dropbear to send its error messages to stderr instead of syslog. But if it still doesn't work with -F -E, then we'll need more precise information. -- Laurent
Re: Path monitoring support in s6-services
Since it requires individual instances of inotifyd for each service(s) [which depends on multiple files/paths modifications) to get started] Have you tried it and noticed an impact on your boot times? (AKA: "Profile, don't speculate.") -- Laurent
Re: Path monitoring support in s6-services
inotifyd (or something similar) + s6-svc (or s6-rc)? Thought of the same but I have many such services;Just thinking of cpu overhead during the initial boot up. What makes you think this would have a noticeable impact on your CPU load? -- Laurent
Re: Some suggestions on old-fashioned usage with s6 2.10.x
(Apologies for the broken threading, I originally sent my answer with the incorrect From: and it was rightfully rejected.)
Re: Some suggestions on old-fashioned usage with s6 2.10.x
I do not really understand their excuse here. CLI incompatibility is trivially solvable by creating links (or so) for `halt' / `poweroff' / `reboot', and even the `shutdown' command can be a wrapper for an `atd' based mechanism. The options! The options need to be all compatible. :) And for "shutdown", they would never implement a wrapper themselves, I would have to do it for them - which is exactly what I did, although it's a C program that actually implements shutdown, not a wrapper around an atd program I can't assume will be present on the system. I'm not defending distros here, but it *is* true that a drop-in replacement, in general, is a lot easier to deal with than a drop-in- most-of-the-time-maybe-but-not-with-that-option replacement. Anyone who has tried to replace GNU coreutils with busybox can relate. In case they complain about the implementation of the CLI, the actual interface to `shutdownd' is not that similar to the `telinit' interface (at least to the one I think it is) either. Which is why s6-l-i also comes with a runleveld service, for people who need the telinit interface. shutdownd is only for the actual stages 3 and 4, not service management (which telinit is a now obsolete forerunner of). If I understand it correctly, letting `s6-svscan' exec() stage 3 also achieves immunity to `kill -KILL -1'. I also find this "old-fashioned" approach conceptually and implementationally simpler than an army of `s6-supervise' restarting only to be killed again What army? By the time the final kill happens, the service manager has brought everything down, and shutdownd has cleaned up the scandir, only leaving it with what *should* be restarted. You seem to think I haven't given these basic things the two minutes of attention they deserve. Conceptually, the "old-fashioned" approach may be simpler, yes. Implementationally, I disagree that it is, and I'll give you a very simple example to illustrate it, but it's not the only thing that implementations must pay attention to, there are a few other quirks that I've stumbled upon and that disappear when s6-svscan remains pid 1 until the very end. You're going to kill every process. The zombies need to be reapt, else you won't be able to unmount the filesystems. So your pid 1 needs to be able to wait for children it doesn't know it has (foreground does not) and guarantee that it doesn't try unmounting the filesystems before having reapt everything (a shell does not give ordering guarantees when it gets a SIGCHLD, even though it works in practice). So for this specific use I had to add a special case to execline's wait command, "wait { }", that waits on *everything*, and also make sure that wait doesn't die because it's going to run as pid 1, even very briefly. And after that, you need to make sure to unmount filesystems immediately, because if you spawn other processes, you would first have to wait on them as well. For every process that may run as pid 1, you need extra special care. Using an interpreter program as pid 1 means your interpreter needs to have been designed for it. Using execline means every execline binary that may run as pid 1 needs to be vetted for it. If your shutdown sequence is e.g. written in Lisp, and your Lisp interpreter handles pid 1 duties correctly, okay, that's fair, but that's *two* programs that need to do it, when one would be enough. s6-svscan has already been designed for that and provides all the guarantees you need. When s6-svscan is running as pid 1, it takes away a lot of mental burden off the shutdown sequence. and a `shutdownd' restarting to execute the halting procedure (see some kind of "state" here? Functional programmers do not hate it for nothing). Yes, there is one bit of state involved. I think our feeble human minds, and a fortiori computers, can handle one bit of state. I know this seems less recoverable than the `shutdownd' approach, but does that count as a reason strong enough to warrant the latter approach, if the halting procedure has already been distilled to its bare essentials and is virtually immune to all non-fatal problems (that is, excluding something as severe as the absence of a `reboot -f' implementation)? My point is that making the halting procedure virtually immune to all non-fatal problems is *more difficult* when you tear down the supervision tree early. I am more confident in the shutdownd approach, because it is less fragile, more forgiving. If there's a bug in it, it will be easy to fix. I understand that the barebones approach is intellectually more satisfying - it's more minimalistic, more symmetrical, etc. But shutting down a machine is *not* symmetrical to booting it. When you boot, you start with nothing and need a precise sequence of instructions in order to build up to a functional system. When you shutdown, you have a fully functional system already, that has proven to be working, and you just need to clean up and make sure you don