socket-activate is published [was: Re: listen(1)]
Hi folks-- On Wed 2016-08-10 11:58:55 -0400, Daniel Kahn Gillmor wrote: > The tool is a binary called "listen", which tries to open a series of > sockets and then exec()'s another command with those file descriptors > already open. It adopts a convention for identifying those file > descriptors from systemd's socket activation features, so that it can be > used to supervise any daemon that follows the same convention. Just wanted to let folks on this list know that if you are interested in this, i've renamed "listen" to "socket-activate", and finally gotten around to publishing an implementation for broader general consumption. It's at https://gitlab.com/dkg/socket-activate, and it's a very simple python3 implementation of the supervisor's side of the sd_listen_fds(3) convention. It depends only on python3's stdlib, so while it's not as minimal as a C implementation would be, it's still pretty lean. In discussions over on https://bugs.debian.org/922353, Guillem Jover suggested that start-stop-daemon might also be willing to accept comparable functionality to support the same convention, to enable support for daemons that use it. Happy hacking, --dkg signature.asc Description: PGP signature
Re: Mailing-lists, failing probes, unsubscriptions
On Thu 2017-11-30 22:53:55 +, Laurent Bercot wrote: > In the past year, or sometimes at an older time than that, a lot of > addresses have apparently failed enough probes and been unsubscribed, > even still working addresses. Only today was it brought to my > attention. This kind of mass unsubscription is often due to DMARC. please see: https://wiki.list.org/DEV/DMARC for a description of how the mailman folks describe the problem and how they deal with it. I think the skarnet lists are probably running ezmlm, i don't know that toolchain well enough to suggest a way to resolve the issue. --dkg signature.asc Description: PGP signature
Re: Vim `.swp' files in nosh source tarballs
On Sun 2017-04-16 20:30:24 +0100, Jonathan de Boyne Pollard wrote:
> I purge the directories of these from time to time, as I have just done again.
> But they tend to very slowly accrue, for one reason or another. It's not a
> major worry. Anyway, better that the archive have everything including some
> harmless text editor temporary files than that it have one vital file missing
> because of a wildcard exclusion/deletion, which I have experienced in the
> past.
I recommend generating release tarballs directly from the code that is
committed to the revision control system. This provides confidence that
the released tarball contains exactly what has been kept track of in
other more detailed ways.
Also, since i'm sure you don't commit editor tmpfiles to your revision
control system, you'll never have this problem again.
if you use git, it's just something like:
versiontag=1.2.3
git archive --format=tar --prefix=nosh-${versiontag}/ ${versiontag} | gzip
-n9 > nosh-${versiontag}.tgz
hth,
--dkg
Re: s6, listen(8), etc.
On Mon 2016-09-05 02:48:54 -0400, Jonathan de Boyne Pollard wrote: > Daniel Kahn Gillmor: > >> #!/bin/sh >> mkdir -p /run/kresd/workdir && \ >> setfacl -m u:kresd:rwx /run/kresd/workdir && \ >> cd /run/kresd/workdir && \ >> exec listen -udp::53 \ >> -tcp::53 \ >> -tcp:label=tls:853 \ >> -unix:label=control,mode=0600:/run/kresd/control \ >> chpst -u kresd -p 1 \ >> /usr/sbin/kresd > > start: > > #!/bin/sh -e > install -d -m 0755 -o kresd /run/kresd/workdir > > stop: > > #!/bin/sh -e > rm -r /run/kresd/ > > run: > > #!/bin/nosh > udp-socket-listen --systemd-compatibility --combine4and6 :: domain > tcp-socket-listen --systemd-compatibility --combine4and6 --backlog 2 > :: domain > local-datagram-socket-listen --systemd-compatibility --mode 0666 > /run/kresd/query.socket > local-stream-socket-listen --systemd-compatibility --mode 0600 > /run/kresd/control > ./service > > service: > > #!/bin/nosh > chdir /run/kresd/workdir > softlimit -p 1 > setuidgid kresd > kresd Thanks, this looks pretty cool. a couple caveats: * i don't see where the label for the sockets is passed in. how is the daemon supposed to know whether to listen on the tcp socket using TLS or not? How is it supposed to know whether the unix domain socket is for control or for recursive DNS queries? * it looks like this will result in a chain of proceses: nosh → nosh → kresd, rather than a single process due to the lack of exec. or am i misunderstanding nosh? I see you've got debian packages available for nosh, but they aren't in debian proper. have you considered putting them into debian? --dkg signature.asc Description: PGP signature
Re: s6, listen(8), etc.
I think we might be approaching diminishing returns on this thread, so please don't take offense if i don't reply too much beyond this. I do appreciate your taking the time to document your thought process here, though. thanks! A couple minor comments: On Thu 2016-09-01 08:34:18 -0400, Laurent Bercot wrote: > I agree with this principle entirely. This is good design. The question > now becomes: how does the daemon know what fd corresponds to what use? > > The simplest, natural answer, used for instance by UCSPI clients and > servers, is: the fd numbers are hardcoded, they're a part of the daemon > interface and that should be documented. > So, in your example, the kresd documentation should say something like: > > - fd 3 must be a datagram socket, that's where kresd will read its > UDP DNS queries from. > - fds 4 and 5 must be listening stream sockets, that's where kresd will > read its TCP DNS queries from. kresd use two sockets for this in order to > allow one of them to be set up over TLS. > - fd 6 must be a listening Unix domain stream socket, that's where kresd > will read its control messages from. This doesn't allow for the creation of multiple different sockets e.g. for a daemon expected to listen on distinct IPv6 and IPv4 addresses, but not on all available addresses. > LISTEN_FD_MAP=udp=3,tcp=4,tls=5,control=6 I'd want the map the other way around: from number to use case, if i was going to go for a re-design like this, though i suppose it's roughly equivalent to have udp=3,udp=4,tcp=5 as long as you don't mind having repeated keys. but meh, this is basically equivalent. > Or does it? The API still constrains you: even if you can provide labels, > you still don't send a map, you send a number of fds and a list of labels. > So you still have to make sure the fds you send are 3, 4, 5, 6 ! And > if your "listen" program happens to have fd 3 open when it's launched > (and there may be very good reasons to have some fd already open when you > run a daemon), well, it has to overwrite it with the udp socket it opens, > because that's what the API forces you to do. If I want the daemon to run > with some open fd that's not explicitly passed via LISTEN_FDS (for instance > if the daemon doesn't have to know it has it open) then I have to make sure > that my fd is at least 3+$LISTEN_FDS. Talk about unwieldy. this seems no more unwieldy (probably less unwieldy) than the convention to always use FD 6 and FD 7. what if i wanted to have some other fd open on those numbers for some other reason? i'd have to move it out of the way to use the s6 convention. Note that i'm not claiming this is terrible, it's only a minor hassle. but it's the same minor hassle as what you're describing here, no? > And from the daemon's side? it has to parse the contents of $LISTEN_FDNAMES, > which is a colon-separated list of labels; that's not really easier than > parsing my suggested LISTEN_FD_MAP string. right, it's equivalent. > So, LISTEN_FDS doesn't know if it wants to pass a list of hardcoded fd > numbers, or a map with labels; it ends up doing a bit of both, badly, and > getting the worst of both worlds. eh? on the contrary: if your daemon doesn't need to distinguish between any of its sockets -- if all sockets are standard listening sockets (it can still inspect the socket itself to distinguish between datagram and stream) then it doesn't care about anything but LISTEN_FDs. If the daemon receives some sockets (like a local control socket) that need to be treated differently, then it needs to look at the map. Your proposed LISTEN_FD_MAP looks precisely equivalent to LISTEN_FDNAMES for this purpose, with only minor syntactic differences. The non-syntactic difference, of course, is that multiple daemons written and running today actually already know how to parse LISTEN_FDNAMES, whether because they link to libsystemd or do the simple parsing themselves. if LISTEN_FD_MAP had gotten there first, i would have made listen(8) implement that labeling scheme. > Oh, and I haven't entirely given up my argument of political agenda. > The only reason you could have for designing such a whimsical interface, > apart from obvious lack of experience in software design, is to encourage > daemon authors to use the sd_listen_fds_with_names() interface (and so, > link against libsystemd), because it's too painful to fully implement it > yourself as a daemon author - you have to handle the case when > LISTEN_FD_NAMES is set, the case when it's not set, etc. etc. (i don't think that baseless slights against the authors of other software advance your cause here) on the technical merits, i really don't see it as difficult as you're making it out to be. When this convention is in use, your daemon already knows whether it needs to parse LISTEN_FDNAMES, or whether it can get away with just looking at LISTEN_FDS. in any case, both will be set. And there is absolutely no need to link against libsystemd.
Re: s6, listen(8), etc. (was: [PATCH 1/3] correct typo)
On Tue 2016-08-30 12:25:37 -0400, Laurent Bercot wrote: > On 30/08/2016 15:20, Daniel Kahn Gillmor wrote: >> i assume you mean http://www.skarnet.org/software/s6/ . thanks for the >> pointer. I'm looking at that, and it looks like s6's preferred form of >> user contact is via github (i've just submitted a trivial pull request). > > Um, not at all. If the homepage gave you that impression, then it's not > clear enough and I should add a note to it. I only use github as a mirror > for source browsing and downloading. Well, your earlier e-mail suggested that you don't like reading patches on mailing lists, and there were two other pull requests that had already been closed so i made the (apparently wrong) inference. sorry for the confusion. > The preferred form of user contact for s6 is through this list - you're > in the right place. :) What did you want to send a PR about? it's a bugfix to the web page for s6-ioconnect: https://github.com/skarnet/s6/pull/3 > Please don't use the term "socket activation": it is a systemd buzzword > with vague and confusing semantics. See > http://skarnet.org/software/s6/socket-activation.html Thanks, once i make it past the angry words, this page has some interesting ideas to work from. The timing and privilege separation aspects are still not clear to me yet, but that'll probably come in time. > Daemons that use shared state in RAM generally open their sockets > themselves. well, sure. they also generally daemonize themselves, which is (quite reasonably) discouraged under a runit or s6 style architecture, though. And ones that listen on "privileged" ports (below 1024), or who need to open listening sockets in places that they themselves wouldn't be able to create sockets generally start as root and then drop privileges after opening sockets. some of them don't always drop privileges very successfully, either :/ I'm looking for supervisors that help me avoid some of that stuff. systemd is one; runit is another; so (potentially) is s6. > In any case, I'll probably update s6-ipcserver-socketbinder so it's > usable with more types of sockets. cool, thanks! > I'm very interested in sharing thoughts about that, but you'll > probably find that I'm *extremely* reluctant to yield an inch when > negotiating with ideas and protocols that come from systemd i'm sorry to hear you're so inflexible about this, but that's your call, i guess. > while being open to ideas for improving daemon management under Unix; > and most of those ideas have been floating around - and sometimes > implemented in supervision suites - since before systemd was a > thing. ;) yep, we're all learning from each other, hopefully. At any rate the LISTEN_FDS convention of how to pass labeled file descriptors via a small number of environment variables requires no systemd code whatsoever, and it's described in a relatively simple document. It's a reasonable improvement in flexibility over the convention of one-pipe-per-process, I/O on 6+7 and 0+1, and it's not particularly hard to implement. So i'm hoping that it'll find a taker in one of these more toolkit-style supervisor suites. thanks for the discussion, --dkg
Re: [PATCH 1/3] correct typo
On Tue 2016-08-30 07:15:20 -0400, Laurent Bercot wrote: > Oh, yes, sorry, you even wrote a message (about a listen(1) command) > not too long ago :) Yes, indeed. That program is now halfway done, and i am hoping to share it with people who might be interested. > (Shameless plug: in the meantime, you could take a look at runit's > cousin, s6. ;)) i assume you mean http://www.skarnet.org/software/s6/ . thanks for the pointer. I'm looking at that, and it looks like s6's preferred form of user contact is via github (i've just submitted a trivial pull request). It doesn't seem to do what i'm proposing with listen(8), though, and listen doesn't quite fit there. For one thing, the socket activation it provides doesn't seem to provide for any shared state between concurrent connections without using the filesystem. Maybe that's a plus, but i don't know of many daemons built to work that way -- they often have shared state directly in RAM, where it's arguably more straightforward to synchronize complex data structures (i'm sure you could do something clever with mmapped files and/or ipc, but i'd have to think about the locking/synchronization stuff there a lot more). Anyway, thanks for the pointers. it looks to me like my proposed listen(8) seems to bridge the gap between the daemontools family and the standard daemons that people will want to deploy (which are being updated for systemd) slightly more closely with runit than with s6, but if you're interested in thinking that kind of gap-bridging through with me, i'm certainly interested. Regards, --dkg signature.asc Description: PGP signature
Re: [PATCH 1/3] correct typo
On Tue 2016-08-30 04:55:19 -0400, Laurent Bercot wrote:
> Please avoid sending patches to this list with git-send-email, or at
> least please provide a 0/n introductory message to give context. This
> list is read by humans.
I'm a human, and i read this list :)
I will be happy to send a 0/n introductory message (git send-email
--compose) for any future proposed series, i was not aware that was the
custom here. Indeed, i wrote this series because i could find no clear
instructions for how to propose improvements to runit's codebase.
This series is a set of minor changes to improve the web presence for
runit and to make it clearer how to contribute code.
Alternately, if there's a better way to propose changes than sending
mail to this list, i'd be happy to rewrite code.html to document it, but
i don't know what it is.
Perhaps an additional patch to the proposed doc/code.html to add the
--compose argument to git send-email would make that clearer.
If i were to do that, would you prefer i send a new series of patches
(rebased against master so that they apply directly) or a new follow-on
patch to this series ("4/3", as it were) that just adds the --compose
argument to the example of how to send patches?
regards,
--dkg
signature.asc
Description: PGP signature
[PATCH 2/3] document how to get the source code and send changes
--- doc/code.html | 47 +++ doc/index.html | 1 + 2 files changed, 48 insertions(+) create mode 100644 doc/code.html diff --git a/doc/code.html b/doc/code.html new file mode 100644 index 000..ca3b32e --- /dev/null +++ b/doc/code.html @@ -0,0 +1,47 @@ + + + +runit - Hacking on runit + + +http://smarden.org/pape/;>G. Pape +runit + +runit - Hacking on runit + +The source code for runit is currently tracked in +https://git-scm.com/;>the git revision control system, +which is likely available for your operating system. When you +have git installed, you can check out the development version +of the code with: + + $ git clone http://smarden.org/git/runit.git + +If you have made changes that you want to contribute back +to runit, please send them to +mailto:supervision@list.skarnet.org;>the project mailing list. + +If you want to use git send-email to send your changes to the +list, please: + + http://skarnet.org/lists/#supervision;>subscribe to the list, + https://git-scm.com/docs/git-send-email;>configure git + send-email, and + from within your runit/ source directory, do: + + $ git config sendemail.to supervision@list.skarnet.org + + +When you have committed patches to your local repository that you want +to propose for inclusion, you can now send them to the list with: + + $ git send-email origin/master + + +Happy hacking! + +mailto:p...@smarden.org;> +Gerrit Pape p...@smarden.org + + + diff --git a/doc/index.html b/doc/index.html index 85bb01b..4f401b9 100644 --- a/doc/index.html +++ b/doc/index.html @@ -10,6 +10,7 @@ How to install runit Upgrading from previous versions of runit +Hacking on runit Benefits How to replace init -- 2.9.3
[PATCH 3/3] move http links to https
--- doc/benefits.html| 6 +++--- doc/chpst.8.html | 2 +- doc/index.html | 12 ++-- doc/install.html | 2 +- doc/replaceinit.html | 6 +++--- doc/runscripts.html | 2 +- man/chpst.8 | 2 +- package/CHANGES | 6 +++--- 8 files changed, 19 insertions(+), 19 deletions(-) diff --git a/doc/benefits.html b/doc/benefits.html index b1687d9..24f667b 100644 --- a/doc/benefits.html +++ b/doc/benefits.html @@ -148,12 +148,12 @@ They normally are shell scripts, and an operating system distribution with software package management should adapt these scripts if they need support for their package management. The -http://packages.debian.org/unstable/admin/runit-run.html;> +https://packages.debian.org/unstable/admin/runit-run.html;> runit-run Debian package is an attempt to integrate runit into -http://www.debian.org/;>Debian GNU/Linux as an alternative to +https://www.debian.org/;>Debian GNU/Linux as an alternative to the default -http://packages.debian.org/unstable/base/sysvinit.html;> +https://packages.debian.org/unstable/base/sysvinit.html;> sysvinit. Stage 2 is packaging friendly: diff --git a/doc/chpst.8.html b/doc/chpst.8.html index 97d90f2..ebc7944 100644 --- a/doc/chpst.8.html +++ b/doc/chpst.8.html @@ -131,7 +131,7 @@ package respectively. sv(8), runsv(8), setsid(2), runit(8), runit-init(8), runsvdir(8), runsvchdir(8) http://smarden.org/runit/ - http://cr.yp.to/daemontools.html + https://cr.yp.to/daemontools.html Author Gerrit Pape p...@smarden.org diff --git a/doc/index.html b/doc/index.html index 4f401b9..d6a9acb 100644 --- a/doc/index.html +++ b/doc/index.html @@ -128,17 +128,17 @@ I recommend doing this; for instructions, see The following distributions are known to include or package runit: -http://packages.qa.debian.org/runit;> +https://packages.qa.debian.org/runit;> Debian GNU/Linux (as alternative init scheme) -http://www.freshports.org/sysutils/runit/;> +https://www.freshports.org/sysutils/runit/;> FreeBSD -http://www.openbsd.org/cgi-bin/cvsweb/ports/sysutils/runit/;> +https://www.openbsd.org/cgi-bin/cvsweb/ports/sysutils/runit/;> OpenBSD http://pkgsrc.se/wip/runit/;> NetBSD http://packages.ubuntu.com/runit;> Ubuntu (as alternative init scheme) -http://packages.gentoo.org/package/sys-process/runit;> +https://packages.gentoo.org/package/sys-process/runit;> Gentoo https://code.google.com/p/runit-for-lfs/;> Linux from Scratch @@ -245,11 +245,11 @@ minit - a small yet feature-complete init svscan as process 1 - by Paul Jarc ftp://ftp.cistron.nl/pub/people/miquels/sysvinit/;> sysvinit - source code -http://www.freebsd.org/cgi/cvsweb.cgi/src/sbin/init/;> +https://www.freebsd.org/cgi/cvsweb.cgi/src/sbin/init/;> FreeBSD's init - CVS repository http://cvsweb.netbsd.org/bsdweb.cgi/src/sbin/init/;> NetBSD's init - CVS repository -http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/init/;> +https://www.openbsd.org/cgi-bin/cvsweb/src/sbin/init/;> OpenBSD's init - CVS repository http://www.atnf.csiro.au/~rgooch/linux/boot-scripts/;> Linux Boot Scripts - by Richard Gooch diff --git a/doc/install.html b/doc/install.html index fb3b2c4..f6b7824 100644 --- a/doc/install.html +++ b/doc/install.html @@ -10,7 +10,7 @@ runit - installation runit installs into -http://cr.yp.to/slashpackage.html;>/package. +https://cr.yp.to/slashpackage.html;>/package. If you don't have a /package directory, create it now: # mkdir -p /package diff --git a/doc/replaceinit.html b/doc/replaceinit.html index 70f6f7d..5e4c3f7 100644 --- a/doc/replaceinit.html +++ b/doc/replaceinit.html @@ -16,7 +16,7 @@ Replacing sysvinit (GNU/Linux) Follow these steps to migrate from sysvinit to runit on -http://www.debian.org/releases/woody/;>Debian GNU/Linux (woody). +https://www.debian.org/releases/woody/;>Debian GNU/Linux (woody). The /sbin/init binary is not replaced until step 6, runit is the default Unix process no 1 after step 7. @@ -128,8 +128,8 @@ To report success: Replacing init (*BSD) Follow these steps to migrate from init to runit on -http://www.openbsd.org/;>OpenBSD 2.9 or -http://www.freebsd.org/;>FreeBSD 4.4. +https://www.openbsd.org/;>OpenBSD 2.9 or +https://www.freebsd.org/;>FreeBSD 4.4. The /sbin/init binary is not replaced until step 4. Step 1: The three stages runit looks for the three stages implementing the system's diff --git a/doc/runscripts.html b/doc/runscripts.html index f832b63..9cc4329 100644 --- a/doc/runscripts.html +++ b/doc/runscripts.html @@ -12,7 +12,7 @@ This is a collection of run scripts for popular services to have them monitored by the -http://cr.yp.to/daemontools.html;>daemontools or +https://cr.yp.to/daemontools.html;>daemontools or http://smarden.org/runit/;>runit package. If you have additional run scripts or one of these run scripts works for you on an operating system not stated here, please diff --git a/man/chpst.8 b/man/chpst.8 index 43c0b8d..3402708 100644 --- a/man/chpst.8 +++ b/man/chpst.8
listen(1): proposed addition to the runit suite
Hi Gerrit and other runit folks-- I'm working on a new tool that i think would fit well with runit, and i was wondering if there is any interest in it as a contribution to the runit suite. The tool is a binary called "listen", which tries to open a series of sockets and then exec()'s another command with those file descriptors already open. It adopts a convention for identifying those file descriptors from systemd's socket activation features, so that it can be used to supervise any daemon that follows the same convention. I've attached a rough draft of the documentation for the proposed utility -- my code doesn't implement the whole thing yet, and it's not yet styled for inclusion in runit. But if you think it'd be useful as part of the runit suite, i'd be happy to try to implement it following the runit conventions. Also: is there a public revision control system for runit? I've fetched the tarballs but some of the info in the tarballs looks like it might be generated from git (e.g. "chpst -V" produces something that looks like a git commit id). I'd be happy to supply changes as patches against a canonical git repo, if there is one. The draft of listen(1)'s man page (in markdown format) follows. Feedback welcome, --dkg listen == SYNOPSIS listen [socketinfo ...] prog where *socketinfo* is of the form: -family:[label]:address[,option=value...] DESCRIPTION --- The `listen` program opens the requested sockets and then execs *prog* with those file descriptors open. The first requested socket will be opened on file descriptor 3, and subsequent ones follow in order. *prog* is the first argument that does not start with a `-`. *family* can be one of several different values. For the following values of *family*, if *address* is a single decimal number, it is treated as a port number and listen will bind to all available addresses on that port. If *address* contains a /, the part before the / is interpreted as a network address of the requested family, and the part after the / is treated as a port number. tcp: TCP on IPv4 and/or IPv6 udp: UDP on IPv4 and/or IPv6 For the following values of *family*, *address* is interpreted as a filesystem path. unix: Unix-domain streaming socket unix-dgram: Unix-domain datagram socket OPTIONS --- Each *socketinfo* can have 0 or more options appended to it, separated by the comma character (,) with their value appended after an equals sign (=). Available options are: backlog: Maximum queue for pending connections (see listen(2) -- defaults to SOMAXCONN). mode: octal mode for unix-domain socket (defaults to relying on umask(2)) user: a numeric User ID value for unix-domain socket ownership group: a numeric Group ID value for unix-domain socket ownership EXIT CODES -- `listen` exits 100 when called with unparseable options. It prints an error message to stderr and exits 11 if it cannot successfully bind all of the requested sockets. Otherwise, its exit code is the same as that of *prog*. ENVIRONMENT VARIABLES - `listen` stores the number of opened file descriptors in the environment variable `$LISTEN_FDS`, sets `$LISTEN_PID` to getpid(2), and stores the list of colon-separated labels in `$LISTEN_FDNAMES`. These environment variables follow the convention understood by libsystemd (see sd_listen_fds(3)) so it should be usable with any socket-activated daemon that expects this behavior. If '$VERBOSE' is set, listen will write details of what it is doing to stderr. EXAMPLES Run a webserver as a non-privileged user, using privileged ports, and indicating to the webserver which ports should offer HTTPS: listen -tcp::80 -tcp:tls:443 chpst -u httpd webserver Run a foregrounded (supervised) pulseaudio daemon: listen -unix::/run/user/1000/pulse/native,backlog=5 pulseaudio --daemonize=no Run a network daemon only on the loopback: listen -tcp::127.0.0.1/ network-daemon Run a public-facing network daemon with a local socket for the local administrator to control it: listen -tcp::443 -unix:control:/run/xyz/control,user=1000,mode=0600 xyzd BUGS AND LIMITATIONS `listen` does not do any sort of name-to-number resolution: network addresses, user ids, group ids, and port numbers must all be specified numerically. Hostnames, user names, group names, service names are not interpreted. `listen` cannot create unix-domain socket filenames that contain a comma (,). `listen` produces its error messages in english and is not internationalized. SEE ALSO listen(2), chpst(8), sd_listen_fds(3), umask(2) AUTHOR -- Daniel Kahn Gillmor <d...@fifthhorseman.net> signature.asc Description: PGP signature
