Using runit-init on debian/Jessie in place of sysvinit/systemd
Luke Diamand: Are there any plans to create a debian package that would just put all the bits in the right place, so you can simply install the package and have it then use runit-init as /sbin/init ? It's called runit-run, created over a decade ago, and it was a Debian package for some years until other people filed bugs that discouraged M. Pape and led him to believe that it wasn't wanted. https://tracker.debian.org/pkg/runit-run Note that people are now filing bugs against runit, some of which have been causing M. Pape grief, because they involve things like the disappearance of /etc/inittab pulling the rug out from underneath the maintainer scripts. https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=runit
Re: thoughts on rudimentary dependency handling
Laurent Bercot: I'm of the opinion that packagers will naturally go towards what gives them the less work, and the reason why supervision frameworks have trouble getting in is that they require different scripting and organization, so supporting them would give packagers a lot of work; whereas sticking to the SysV way allows them not to change what they already have. Especially with systemd around, which they already kinda have to convert services to, I don't see them bothering with another out of the way packaging design. So, to me, the really important work that you are doing is the run script collection and standardization. If you provide packagers with already made run scripts, you are helping them tremendously by reducing the amount of work that supervision support needs, and they'll be more likely to adopt it. nosh 1.12 comes with a collection of some 177 pre-built service bundles. As I said to the FreeBSD people, I have a goal of making the 155 service bundles that should replace most of FreeBSD rc.d . (There are a load of non-FreeBSD bundles in there, including ones for VirtualBox services, OpenStack, RabbitMQ, and so forth. This is why I haven't reached 155 even though I've made 177.) It also comes with a tool for importing system service and socket units into service bundles. And the nosh Guide chapter on creating service bundles has pointers to the run file collections by Gerrit Pape, Wayne Marshall, Kevin J. DeGraaf, and Glenn Strauss. xdg-open /usr/local/share/doc/nosh/creating-bundles.html Incidental note: I just added another service bundle, for nagios, to version 1.13 because of this: http://unix.stackexchange.com/questions/179798#179798 Enjoy this, too: http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/run-scripts-and-service-units-side-by-side.html
Re: thoughts on rudimentary dependency handling
Avery Payne: But from a practical perspective there isn't anything right now that handles dependencies at a global level. Now you know that there is. The nosh design is, as you have seen, one that separates policy and mechanism. The service-manager provides a way of loading and unloading services, and starting and stopping them. The dependency system is layered entirely on top of that. Whilst one can use svscan/service-dt-scanner and svc/service-control and have things run the daemontools way, one can alternatively use the start/stop/enable/disable subcommands of system-control and have service dependency management instead. The trick is that dependency management is calculated by the system-control program. When you tell it systemctl start workstation.target it follows all of the wants/, required-by/, and conflicts/ symbolic links recursively to construct a set of start/stop jobs for the relevant services. Then it follows the after/ and before/ symbolic links to turn that set into a ordered graph of jobs. Finally, it iterates through the graph repeatedly, sending start and stop commands to the service manager for the relevant services and polling their statuses, until all jobs have been enacted. conflicts/ is actually easy, although it took me two tries. If A/wants/B exists, then a start job for A creates a start job for B. If A/conflicts/B exists then a start job for A creates a stop job for B. The important point is that the service manager is entirely ignorant of this. It is just told to start and stop individual services, and it knows nothing at all of dependencies or orderings. (A dependency is not an ordering relationship, of course, but the manual page that I said to read has already explained that. (-:) It's all worked out outwith the service manager. Which means, of course, that it is alterable without changing the service manager. And indeed, nosh demonstrates such alteration in action. The dependency-based system that one gets with system-control is one of two alternative policies that come in the box; the other being an old daemontools-style system with a /service directory and ./log relationships, as implemented by the service-dt-scanner (a.k.a. svscan) daemon. Again, the trick is that the daemontools-style stuff is all worked out in svscan, and the service manager proper has no dealings in it. The service manager provides a flexible plumbing layer. Higher layers decide how they want that plumbing put together.
Re: thoughts on rudimentary dependency handling
John Albietz: I wonder if this will help address a common situation for me where I install a package and realize that at the end of the installation the daemon is started using upstart or sysv. At that point, to 'supervise' the app, I first have to stop the current daemon and then start it up using runit or another process manager. On Debian, for one, they aren't started using upstart or sysv (whatever that is). Maintainer scripts enable them with update-rc.d and start them with invoke-rc.d. You are expected to have update-rc.d and invoke-rc.d tools that are appropriate to your init system, as well as the respective control files of course. openrc comes with update-rc.d and invoke-rc.d that understand openrc scripts. The sysv-rc package comes with update-rc.d and invoke-rc.d that understand systemd units, upstart jobs, and System 5 rc.d scripts. Ironically, the systemd and upstart packages do not come with their own update-rc.d and invoke-rc.d commands, relying instead upon the sysv-rc package to supply them. This is all a bit shakey and rickety, though. One well-known fly in the ointment is that what may be a single init.d script for System 5 rc may be multiple service and socket units for systemd, and stopping a socket-activated service for package upgrade might not do the right thing as the socket activation might activate the service unit mid-upgrade. Last year, I gave Debian Developer Russ Allbery a patch for an improved version of the Debian Policy Manual that sets this out more clearly than the current one. You might want to get it off him. The sections (of the patched document) that you are interested in are 9.3.1.2, 9.3.6, 9.3.6.1, 9.3.6.2, and 9.3.6.3.
Good news for BSD
Laurent Bercot: Readiness notification is hard: it needs support from the daemon itself, and most daemons are not written that way. Ah yes. That reminds me. I said a while back that I have to give some of you some good news. https://lists.debian.org/debian-devel/2014/10/msg00709.html You might be interested in the readiness discussion, too.
Re: dependant services
Buck Evan: For example, I'd like to encode the fact that I don't expect service A to be able to come up before service B. In nosh, the filesystem is the database. This is an ordering, not a dependency. One can separately encode in nosh (a) that start of service B will cause the start of service A, and (b) that start of service A has to be scheduled after the start of service B rather than in parallel. The latter is an ordering. And it's a symbolic link, either A/after/B pointing to B or B/before/A pointing to A. The former is a symbolic link B/wants/A pointing to A. This is a very brief precis. There's a detailed explanation of service bundles and all of the subdirectories in them in the manual. Start with man system-control. Or even man manual/system-control.1 if you haven't been brave enough to actually export the software into /usr/local. (-: The most widespread general purpose practice for breaking (i.e. avoiding) this kind of ordering is of course opening server sockets early. Client and server then don't need to be so strongly ordered.
Re: Arch Linux derivative using s6?
Avery Payne: There's already a project for adding definitions for various daemons. http://bitbucket.org/avery_payne/supervision-scripts There are even more than that. I mentioned back in January that the nosh Guide chapter on creating service bundles has pointers to the run file collections by Gerrit Pape, Wayne Marshall, Kevin J. DeGraaf, and Glenn Strauss. I also pointed out that nosh came with some 177 pre-built service bundles. That figure has since risen to some 230-odd (not including log services). Of course, I do sometimes get to just take systemd units and convert them. * http://unix.stackexchange.com/a/200281/5132 * http://askubuntu.com/a/617822/43344
Re: dependant services
Laurent Bercot wrote: If all this fuss is about socket activation, then you can simply forget it altogether. Jonathan was simply mentioning socket activation as an alternative to real dependency management, as in that's what some people do. I don't think he implied it was a good idea. Only Lennart says it's a good idea. Or people who blindly repeat what Lennart says. Actually, I carefully wrote opening server sockets early, talking about the specific mechanism that is employed to weaken client ordering dependencies upon servers. As to whether opening server sockets early is a good idea: I'm not in a hurry to naysay. It achieves the stated effect. Arguably, indeed, it can be described as *what the system already does* if one has a lot of daemontools-style services spawned through UCSPI toolsets. They all start up early and in parallel, opening the sockets very first thing with something like tcpserver or tcp-socket-listen and *then* progressing to starting the main server program, thereby allowing clients to connect and block (rather than fail to connect and abend over and over) in parallel. So it would be possibly a bit rich for me to agree that this is a Lennartism. Especially given that I have machines that do this and have had since before systemd was a twinkle in upstart's eye. (-:
Re: nosh on GNU/Linux: missing sys/event.h
Guess who didn't spot that it was Cc:ed to a mailing list? (-:
Some of the salient points that others may be interested in from what I
sent privately, plus some extra general-audience stuff:
Yes, kqueue. Being in the very first bullet point right along with the
other important API dependencies is not really not mentioning it, in
fairness. (-:
(I might have mentioned this before, but it bears repeating: If you are
writing a service manager, kevent() on the BSDs is well worth
investigating. One can track orphaned daemon children with NOTE_TRACK
on EVFILT_PROC. The Linux library version has a rather nasty open file
descriptor security vulnerability, and a sneaky doco change to cover up
a difference in signal semantics, both of which are the cause of the
majority of the conditional compilation in nosh. But you'll notice that
I've not needed to write a fghack equivalent. The nosh
service-manager is quite happy with the
old you-have-to-turn-an-undocumented-debug-option-on-to-stop-it
forking Vixie cron on FreeBSD.)
No, redo isn't entering a loop. It simply traverses multiple dependency
paths to end at the same failing leaves. There are several paths to
some object files, and of course several leaves to end up at in the
dependency tree that require the kqueue API. It does stop eventually.
Turn on the --verbose option (with REDOFLAGS is easiest) to see.
In early versions, one had to build from source. There were no binary
packages. In some ways that's the BSD influence at work, where
downloading something from the ports collection and building it from
source is the norm. I hadn't even looked at the BSD binary packaging
system until quite recently.
The downside of building from source is that you won't reap the benefit
of all of the work that I've recently put into maintainer scripts to
automatically do all of the service bundle setup. package/export is not
the whole story, as the source package page warns. But I'm sure that
the package/debian/*.{post,pre}inst scripts won't prove that cryptic to
non-Debian Linux people. (BSD people can enjoy the
package/bsd/*.{post,pre}-* scripts.) There are some maintainer script
bugfixes coming in version 1.17, note.
If you are planning to run under systemd, the Debian maintainer scripts
will show you the way. If you are planning to run fully
nosh-system-manager-manged, note that packaging that up is still on the
roadmap. At the moment you'll have to do some things by hand. There are
no prepackaged services yet for loading sysctl settings and loading
kernel modules; and running udev and a system-wide dbus are slightly
shaky. And how you arrange for init=/bin/system-manager and running
something like mdev or s6-devd instead of udev is up to you. (-:
In version 1.17 I hope to have ready the tool for auto-creating the
mount@*, fsck@*, ttylogin@*, dump@*, and swap@* service bundles from
/etc/ttys and /etc/fstab, as also mentioned on the roadmap. I already
have some of it. Of course, you can already run the underlying
convert-fstab-services command and do things longhand, but a nice set of
redo scripts to do it from a simple redo all (and thus only regenerate
things when the configuration files change) seems to be turning out to
be a good idea in practice. As a consequence preset has had to gain a
--fstab option which you will see in version 1.17. A kmod@* bundle
generator has yet to be written, though. But you can always
convert-systemd-units a new kmod@wibble.service service, or copy one of
the existing kernel module service bundles that are provided.
nosh version 1.16
nosh is now up to version 1.16 * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html As you'll see, the WWW pages have expanded a bit. In part this is because of the Big News, which is the arrival of FreeBSD packages, bringing FreeBSD up to par with Debian. The old box down the right-hand side of the page was starting to make the thing look lop-sided. (-: * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/freebsd-binary-packages.html * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/debian-binary-packages.html More big news on the package front is the reorganization into a main bundles package and a group of -run packages. Lesser news is the addition of packages for enabling/running various further groups of services. Comparatively small news are things like the change to the output of system-control status and service-status, which now uses long form that displays more information. svstat retains its 1 line short form, however. There's also a system-control cat command, for dumping out service bundle configuration files. The new emergency-login fills the gap where FreeBSD lacks a sulogin (because it's hardwired into the old init and thus unusable separately), and also means that there's no need to rely upon the old System 5 utilities/Linux utilities for sulogin on Linux. There is also a new roadmap WWW page. The Nosh Guide has also gained several new pages dealing with logging and the import of external stuff.
start: Unknown job: runsvdir
An anonymous person has a problem with runit under upstart on Ubuntu Linux version 14: http://askubuntu.com/questions/630166/ So too do others on the hyperlinked Launchpad bug discussion. You might like to help them all.
pflog(8) manual page bug
Jonathan de Boyne Pollard: There's the steady chipping away at that list of 157 rc.d scripts, which has just lost off its list a few pf services and savecore amongst other things. Yet another bug report that has come out of this is that the pflogd(8) manual page has some gibberish under the -i option. The original OpenBSD update ... * http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/pflogd/pflogd.8.diff?r1=1.35r2=1.36f=h ... was not accurately brought over to FreeBSD ... * https://svnweb.freebsd.org/base/head/contrib/pf/pflogd/pflogd.8?r1=171172r2=223637 ... and was missing two lines that make the gibberish make sense. Of course ... * http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/pflogd/pflogd.8.diff?r1=1.37r2=1.38f=h ... that option doesn't even exist any more. (-:
Re: nosh: service-dt-scanner gets repeatedly killed by SIGABRT
Guillermo: Have you ever considered just writing a BSD backend using kqueue / kevent, and a Linux backend using epoll / inotify or whatever native Linux mechanism is available to suit nosh's needs? I believe you mentioned there's quite a few conditional compilation already to work around libkqueue issues, and you also already have `uname` = FreeBSD and `uname` = Linux tests in the .do files. That's really the reverse of the direction that I'd like to go. I'd like to have less of this particular conditional compilation, not more. The aim is for the libkqueue bugs to be fixed, so that I can then take out the idiosyncratic Linux code.
nosh version 1.22
The nosh package is now up to version 1.22 .
* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
There are several things in this release:
* a new binary package for FreeBSD
* improvements to the user-mode virtual terminal subsystem
* changes and additions to UCSPI tools
* log export to remote servers
* kqueue on Linux
* miscellany
a new binary package for FreeBSD
There's now a debian-shims binary package for FreeBSD. This contains the
heretofore not packaged invoke-rc.d and update-rc.d shim programs. I
decided to make this separate from the systemv-shims package because
these areless general-purpose than those shims are.
The haltsys, fasthalt, fastpoweroff, and fastboot shims are now
packaged, also.
improvements to the user-mode virtual terminal subsystem
The console-fb-realizer now displays a mouse pointer sprite on the
display, to reflect the position of the mouse, when an application has
turned it on with the relevant control sequences. Mouse support via the
evdev input subsystem on Linux is thus now fully implemented, including
support for tablets that use absolute rather than relative positioning.
On the FreeBSD side, you can use sysmouse devices. But this only
permits relative positioning. This is a limitation of sysmouse itself,
as far as I can see. A lot has to change, including the kernel, the
protocol, and moused, to enable absolute positioned devices via
sysmouse. Absolute positioning devices will therefore be supported using
uhid devices. Some of that is done already, but it's not complete yet.
Keyboard maps are now generated by the external configuration import
subsystem from whatever one has in /usr/share/vt/keymaps , rather than
being hardwired to a fixed set of countries. In the absence of this
directory (as will usually be the case on Linux operating systems),
fallback U.S.A. and U.K. keyboard maps are generated.
This generation is worthy of note, as it exemplifies the mechanism that
allows multiple BSD keyboard maps to be overlaid to make one generated
map. The fallback U.K. keyboard map is generated by taking the built-in
U.S.A. keyboard map and applying a "us_to_uk" overlay map on top of it
that only has the few differences between the U.S.A. International
layout and the U.K. one. (This currently produces the basic U.K.
layout. "U.K. Extended" should be a simple matter of another overlay
that does the various Option+A -> a-acute mappings, but that's
somethingfor the future.) Similarly, versions of existing maps that
swap Caps Lock and Control are generated by adding a simple overlay that
does solely that. Likewise, generated maps have an overlay applied that
sets the Backspace key to the application-programmable DEC VT behaviour
that console-terminal-emulator supports, that out-of-the-box BSD keymaps
don't know anything about.
changes and additions to UCSPI tools
For consistency, the UCSPI tools that supported a single --numeric
option now support --numeric-host and --numeric-port options, for
separately determining whether hosts and ports are taken to be names or
just numbers.
There are now client-side tcp-socket-connect and udp-socket-connect
tools, that open client sockets, connect them to servers, and then
chain. These adhere to the UCSPI conventions for inherited open file
descriptors in client-side tools.
log export to remote servers
The new UCSPI clients were motivated by the new export-to-rsyslog
command. This is a daemon that expects to be invoked as a UCSPI client,
connected to a remote RFC 5424/RFC 5426 ("rsyslog") server. It
maintains a set of "log cursors" that point to daemontools-stylelog
directories. Tracking its position in the logs using those cursors, it
sends new log information to the connected server. In the usual nosh
fashion, the filesystem is the database, and the "cursors" are just
files and symbolic links. The details are on the manual page.
In conjunction with the UCSPI clients, export-to-rsyslog thus makes a
log remote export service.
This isn't intended to be the last word in such things. RFC 5426is
unreliable, and RFC 5424 loses the microsecond and nanosecond
information of TAI64N. But it demonstrates the idea and shows that this
can be done in the daemontools world. One can indeed export
daemontools-stylelogsif one has (say) a suite of servers whose log data
should be copied over, on the fly, to a centralized rsyslog server.
There's room here for someone to take this idea and run further with it
using something like RELP.
miscellany
--
The several miscellaneous items include OpenLDAP services in the
autoconfiguration subsystem and some tweaks to the /etc/fstab conversion
on Linux to
Re: nosh: service-dt-scanner gets repeatedly killed by SIGABRT
On 2015-11-05 01:29, Guillermo wrote: So problem solved. Thank you, this is a major improvement. Brill. Thank you for the confirmation.
Re: nosh: service-dt-scanner gets repeatedly killed by SIGABRT
Guillermo: Jonathan de Boyne Pollard: If there's no error output, crank up strace and see what the last few system calls are. It's probably worthwhile doing that anyway, in fact. [...] a read() call on the file descriptor returned by the inotify_init() that produces an EINVAL error, followed rt_sigprocmask() with a SIG_UNBLOCK argument, and the tgkill() that sends the SIGABRT. Remember that I said that my immediate suspicion is a (fourth) libkqueue bug? It's a fourth libkqueue bug. And it's here: * https://github.com/mheily/libkqueue/blob/master/src/linux/vnode.c#l70 As the inotify(7) manual page says, if an event is larger than the buffer size given to read(), it fails with EINVAL. And events can be larger than sizeof(struct notify_event). libkqueue doesn't deal with this failure properly, leading to a call to abort(): * https://github.com/mheily/libkqueue/blob/master/src/linux/platform.c#l181 nosh code never calls abort(), never calls raise(SIGABRT), and would have printed some kind of message if an unhandled exception had led to an abort being raised by the C++ library. The output that you are seeing from service-dt-scanner is because of a spurious wakeup. * https://github.com/mheily/libkqueue/blob/master/src/linux/platform.c#l199 You can turn these debug messages on with the KQUEUE_DEBUG=1 environment variable (and compiling the library in debug mode), apparently. * https://github.com/mheily/libkqueue/blob/master/src/common/kqueue.c#l68 libkqueue is receiving events from inotify that the caller of kevent() isn't actually interested in, resulting in a spurious wakeup from the call to kevent() with no actual event to report. The output to standard error is a minor bug in service-dt-scanner, because it assumes that every time that it is woken up and kevent() returns successfully there will be at least one event. It's finding nonsense in the event buffer and printing out a debug message when it ignores the nonsense. This is fixed in version 1.18, but this isn't really the cause of your problem here. It's just distracting log noise. The problem here is that inotify is waking kevent() up because you listed the directory. I suspect this change in your version of libkqueue, at first glance: * https://github.com/mheily/libkqueue/commit/e41cc259a0318b0e7925521d0fe3bc7433971ace After the spurious wakeup, there is another second event enqueued by the kernel, that is bigger than sizeof(struct notify_event). Whether that's an uninteresting event too, and whether it is also caused by your listing the directory, is unknown. libkqueue isn't passing a buffer big enough to read it so that we see what it is, and is abort()ing when the kernel returns an error because the read buffer is too small. This will be a tricky one for the libkqueue people to fix, since libkqueue isn't currently geared up to process multiple events from inotify at once, which it would have to be prepared for if it were to start using a bigger buffer. But it is a libkqueue problem to be fixed. All that service-dt-scanner is doing is registering just one event of interest, and calling kevent() in a fairly tight loop that's in fact doing nothing else (apart from dumping the value of the spurious event).
Re: nosh
Guillermo: Colin Booth: As it stands, nosh is playing in the same space as upstart, launchd, and systemd. If you don't look too closely, you could also say nosh without the userspace terminal and login services stuff more or less covers the same ground as s6, s6-rc and a small subset of execline. If someone does want to look closely, this page is now a lot more complete than last time I checked (or as I remembered it being): http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html Your memory isn't playing tricks on you. I gave it some long-needed attention a while back. (-: There's now this as well: http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html
Re: nosh
Colin Booth: I mean't more in the "one stop low-level system management" sense. Also in the design of the nosh unit files (which may be a case of parallel evolution, my knowledge of the history of nosh is limited at best). Pholosophicaly it's definitely coming from the daemontools world. Yes, and they aren't really "nosh unit files". The native mechanism for nosh is the service bundle, comprising supervise/ subdirectory, service/ subdirectory, and various others. The systemd unit file is something that can easily be turned into a service bundle. It's not parallel evolution. It's simple recognition that the world is handing around systemd unit files with its softwares, and providing an easy migration path for that. There's no reason, in principle, that someone couldn't come along and write a tool that parsed a launchd plist and made a native nosh service bundle. Translate EnvironmentVariables into setenv; SoftResourceLimits into softlimit; and so forth.
Re: nosh version 1.19
Guillermo: Hi, I had some issues with the source package I thought I should mention: As I said I intended to a while back, I incorporated fixes for these into version 1.20. This is now out.
nosh version 1.21
The nosh package is now up to version 1.21 . * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html console-terminal-emulator now has a mouse input event protocol, and speaks both the DEC VT Locator protocol and the xterm Private Mode 1006 protocol over the terminal interface to applications. These are the protocols that you get with ttymouse=dec and ttymouse=sgr in vim. Since support for the 1006 protocol is fairly widespread in the relevant places nowadays, it seemed not worthwhile implementing the inferior Private Mode 1005 and Private Mode 1015 protocols. Moreover, console-terminal-emulator is UTF-8 and the Private Mode 1005 protocol has ambiguities once one introduces UTF-8. console-fb-realizer as yet only talks to real mouse input devices on Linux, but handling FreeBSD/PC-BSD mouse input devices is on the roadmap. This has necessitated a change in the command-line syntax of console-fb-realizer, and concomitant changes in the pre-supplied realizer service bundle, which will need to be updated in tandem if you are using user-space virtual terminals. This change allows the mouse input device to be specified in addition to the keyboard input event device. It also slightly regularizes display-only mode, which is signified now by the simple lack of any mouse or keyboard device specifications, rather than by an explicit option. Yes, I am aware that there's no mouse cursor sprite drawn yet. Acutely so. FreeBSD improvements include the completion of geli and gbde import that the sharp-eyed might have noticed quietly part-done in version 1.20. /etc/fstab entries for volumes using these should now be translated into appropriate interdependent mount@*, gbde@*, and geli@* service bundles. This is rather difficult for me to test, though, as noted on the roadmap. The big PC-BSD improvement is jails support, which has lurked at the bottom of the roadmap page for a while. Both PC-BSD Warden and FreeBSD 9 jails are now recognized by the external configuration import subsystem, and converted into appropriate service bundles. The mechanism here is fairly straightforward: The jails themselves are one service bundle, and the programs that run in the jails are another. The latter service is after/ and wants/ the former service. The jexec command is a chain loading tool that modifies process state in the same vein as setuidgid, softlimit, and envdir, and one can simply employ it as such. The jail command can be used analogously, with jail -c and jail -r, to how the mount and umount commands in mount@* service bundles are used. Those are what the import subsystem does. Importing Warden Linux jails isn't available yet; and some of the more esoteric FreeBSD 9 rc.conf and PC-BSD Warden METADIR/* jail options are not yet imported. Enabling jails to be autostarted at bootstrap is via the "jails" and "warden" targets, by the way. The rc.d conversion project has progressed, with a few more things wiped off. As mentioned in the version 1.20 message, all assistance in wiping the final 40-odd FreeBSD rc.d scripts off the list, to be found on the roadmap page, is welcome. And if any PC-BSD people have ideas on how to turn things like /usr/local/etc/rc.d/pc-samba into service bundles, those are welcome too. (Note that pre-supplied service bundles already exist for the Linux flavours of some of these, which may or may not be a starting point.)
Re: [announce] s6-rc: a s6-based service manager for Unix systems
Laurent Bercot: s6-rc is a service manager for Unix systems, running on top of a s6 supervision tree. ... which according to https://news.ycombinator.com/item?id=10277123 is configured using proprietary binary blobs.
nosh version 1.20
Joe Maloney: do you have a source code repository somewhere for nosh? * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/source-package.html The source package and how to build from source are here.
nosh version 1.20
The nosh package is now up to version 1.20 . * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html It's worth noting that the WWW site has gained some more pages, an installation how-to and a quick look at user-space virtual terminals. * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/user-vt-screenshots.html The command and tool list page, which was woefully out of date, has had some attention, too. It is rather longer than it was. * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html You might notice a couple of new BSD packages, as well. FreeBSD/PC-BSD binary packaging is now up to parity with Debian Linux. One can create a fully-nosh-managed system on both just by installing some binary packages. This wipes another to-do item off the roadmap page. The list of remaining rc.d items on the roadmap has shrunk, also. As always, assistance in wiping those remaining rc.d items off the list is welcome. If someone feels up to tackling /etc/rc.d/bluetooth, perhaps looking at what Iain Hibbert has apparently already done, for example ... In addition to having yet more service bundles, this release irons out some wrinkles in startup and shutdown. The sysinit phase of bootstrap was causing undesirable mounts in emergency mode. That has been restructured. Some ordering problems in shutdown relating to unmounting filesystems have also been fixed. And the System 5/BSD compatibility reboot, halt, and poweroff shims no longer rely upon some other toolset's (not necessarily even present) shutdown command. There are now -run packages for four different Debian Linux plug-and-play managers, with vdev and suckless mdev now added.
Re: Some suggestions about s6 and s6-rc
Colin Booth: The distinction is specifically thus: ./up is what fires when the service is brought up, ./down is what fires when the service is brought down, ./run is what fires when a non-running service is supposed to be running, and ./finish is when a running service stops. Just because oneshots don't support run or finish, or that longruns don't support up or down, doesn't mean that the separation of duties is any different. Just for comparison, since you are hunting for names: In nosh, which re-uses the daemontools-encore state model: ./start is what runs in the starting state when the service is started from the stopped state; ./stop is what runs in the stopping state when the service is stopped from the running state; ./run is what runs in the running state; and ./restart is what runs in the failed state, before either going back to running or going through stopping to stopped. systemd's "Type=oneshot" translates to a service with its meat in ./start with ./run either executing pause or true. (See http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/unix-daemon-readiness-protocol-problems.html for the readiness protocol that this is replicating.)
nosh version 1.23
The nosh package is now up to version 1.23 . * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project There is one major item in this release. * I've adjusted console-fb-realizer's keyboard handling on BSD to use USB directly. There are a few more minor changes. * I've upgraded the version of clang++ that is used to build the binary packages to 3.8.0. This should have no visible effect whatsoever. (-: * The handling of the DECDA2 control sequence by console-terminal-emulator now copes with what vim sends. (What vim sends isn't what my DEC VT tests had been checking.) * convert-systemd-units now inserts uses of the ionice and chrt chain-loading commands on Linux into the generated service bundles. Mostly this is a clearing the decks release in the hope that I will be able to do some more work on the remaining few FreeBSD conversions before the new year. USB keyboard support The keyboard handling is a change to using the USB HID devices (/dev/uhid*) on FreeBSD in preference to (but not forcibly instead of) the ATA keyboard protocol. In part this is in order to handle the "consumer" keys that USB has. In part this is in order to handle the extra keys that one finds on 106-key, 107-key, and 109-key keyboards and on some numeric keypads (such as the ABNT2 thousands-separator key). In part it's to remove an extra layer of the user-space virtual terminal system that can be outwith the kernel. In part it's to match the USB mouse capability from version 1.22 of the toolset. Please note that the structure of kbdmap files has changed slightly, to accomodate mappings for "consumer" keys, to reposition the entries for some of the 106/107/109-key keyboards' extra keys, and to cover all of the function key gymnastics that vim can accept. The /etc/system-control/convert/ system should automatically re-convert your VT kbd files into the new format.As part of this, I've moved the mapping for the Euro symbol in the fallback U.K. layout (as generated on Linux in the absence of VT kbd files). It used to be level 3 shift on the [eE] key in prior versions of the toolset. Almost all real U.K. keyboards nowadays have it engraved as level 3 shift on the [4$] key, and that's where it now is. Also note that I'm still working on this. There might be further changes in 1.24. I've found a U.K. keyboard with two [#~] keys (at A00 and C12), and I need to check out whether this actually employs what I had thought to be an error in the USB HID usage tables (distinct usages for "\|" and "Europe1") and had corrected, or whether this is a quite mad keyboard that simply has two "Europe1" keys (or two "\|" keys). Also, I've ordered an ABNT2 and a Japanese USB keyboard, and hope to do some testing with them, which may prompt further tweaks. (I really wanted to buy a Leadership 4530 keyboard. They seem to be out of stock in a lot of places.)
Re: nosh version 1.23
Guillermo: It seems that the 'unload-when-stopped' and 'version' subcommands of system-control are not mentioned in the documentation. They both should be. I've added the latter to the doco ready for the next version. The former I'm wavering about. I've had a to-do item for some time to turn that into "service-control --exit" (a.k.a. "svc -x"), changing it to use the supervise/control API instead of the service manager's control socket. The service manager would do the same thing. Just the API and the control utilities would change. The problems are (a) I haven't got around to that part of the to-do list yet and (b) the upgrade path is a little tricky and I think would involve more than one release.
Re: nosh version 1.23
Guillermo: So I looked at the source/targets/virtualbox-host.target file from which the bundle is created, and, IMHO, there are some issues with it. That's not surprising. Running as a virtualbox host is not something that is well tested. As you can see from the roadmap WWW page, there are a few things that are hard to test because I simply don't have them in use anywhere, like GELI and AppleTalk. Another of those is a machine hosting virtualbox VMs. Guillermo: But on a more conceptual level, I don't really understand why these 'Wants=' lines are even there. They might well be entirely redundant, as you say. I'll add looking at them afresh to the to-do list. It may well be simply the case that they can be deleted outright from the target, and enabled by presets. That's certainly the design on the running-as-guest side. (There seems to be a slight bug in 80-virtualbox-guest.preset, considering this. Hmmm.)
Re: logging and mutualise outputs
Laurent Bercot: for logdir in `cat logdir_list` ; do (cd $logdir && cat *.s *.u current) ; done | sort > logfile find `cat logdir_list` -maxdepth 1 -type f \( -name '*.[su]' -o -name current \) -print0|xargs -0 sort -m -- > logfile sort can do a merge sort directly from the original files, which are of course already sorted. It's better when one's logs are *big*.
Re: logging and mutualise outputs
Vincent de RIBOU: Is it possible to perform the same mechanism but having concatenated file along all the others at all times ? There's always tail with the -F option on a list of "current" files. But you're then limited to just the "current" log data, it's tricky to track which output comes from which log, and it's tricky to track over missed log rotations when tail wasn't running. One way to track things better when one knows that one is specifically looking at a TAI64N log directory (as produced by cyclog/multilog/s6-log) is exemplified by Russ Allbery's multilog-watch (https://www.eyrie.org/~eagle/software/multilog-watch/) and my export-to-rsyslog (http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/). It is the idea of log cursors. multilog-watch maintains a single cursor because it only watches one log directory. export-to-rsyslog has a suite of maintained log cursors which it holds in a cursor directory, because it can watch multiple log directories. A cursor is roughly the TAI64N stamp of the last log entry to be processed. (Read the export-to-rsyslog manual page for details.) Watching a TAI64N log directory with cursors has the advantage over some other mechanisms that it entirely de-couples the log watching from the logging daemon. There's no back pressure on the running service (back down a pipeline) if the log watcher is very behind. One can have multiple quite independent log watchers with their own sets of cursors. One can run the log watcher when the logging daemon isn't running, and vice versa. The caveats of (say) making sure that the reading end of a FIFO never goes away never arise. multilog-watch mails logs, which isn't really what you want. export-to-rsyslog writes log entries in RFC 5424 form to a (datagram) socket. Its major usage is providing near real time export of multiple TAI64N log directories to something like logstash, with the individual log cursor names encoded in the RFC 5424 data. You are not required to point it at logstash, though, and can point it at something simpler. However, RFC 5424 is still a lossy protocol, not least because it doesn't have full TAI64N resolution. So pointing this at an RFC 5424 receiver is to introduce codec loss. Nonetheless, it's one way to obtain a centralized combined log, in a properly size limited and rotated TAI64N log directory no less, that you can manipulate with all of the log directory tools that you are used to. For those interested despite the aforementioned, the nosh service bundles package has pre-made service bundles for an RFC 5424 server over either a local datagram socket or a RFC 5426 UDP socket. The local-syslog-read service bundle is an instance of local-datagram-socket-listen opening /run/log (/dev/log on Linux operating systems) and chaining to syslog-read. The udp-syslog-read service bundle is an instance of udp-socket-listen binding to the well-known "syslog" UDP port and chaining to syslog-read. syslog-read itself is a UCSPI-style tool that reads RFC 5424 from the already-opened socket and writes it to standard log suitable for piping into cyclog/multilog/s6-log. There are accompanying cyclog@local-syslog-read and cyclog@udp-syslog-read service bundles for that very thing. There's no reason that someone cannot take the idea of log cursors and make a tool like export-to-rsyslog but with a non RFC 5424 back-end, though.
Re: Supervising a pipeline?
Laurent Bercot: You can't supervise a pipeline per se; you need to supervise both processes in the pipeline independently, and make sure the pipe isn't broken when one of them dies. So, have "exec inotifywait /dev/disk" as foobar/run, and have "exec automounter.py" as foobar/log/run. This will work with daemontools, runit and s6. (You can accomplish the same goal with perp and nosh too; the syntax will just be different.) Actually, the syntax for nosh can be exactly as described: something/run and something/log/run . It's not ideal, because of course then there's no properly separated logging of the automounter. Laurent Bercot: Alternatively, you could use s6-rc and create the "inotifywait" and "automounter" longrun services in a pipeline; your compiled database would then include instructions to set up the supervised pipeline for you. This is more complex to set up than just using the integrated pipe management in svscan and runsvdir, but it's also more powerful, because you can pipeline an arbitrary number of processes that way (this is also what nosh does). Yes: The nosh toolkit makes this extensible. The aforegiven configuration is enacted by the daemontools service scanner. The underlying service management supports arbitrarily long pipelines of services, each one's standard output and standard error feeding into the standard input of the next one along. The system-control utility looks at the "log" symbolic links of whatever collection of services it is operating upon and sends a "plumb services together" order to the service manager. So one could construct a 3-service long pipeline: inotifywait into Python program into cyclog instance.
Re: Announcing cgid v0.1.0
fREW Schmidt: To read the docs check out https://github.com/frioux/cgid/releases/download/v0.1.0/cgid-v0.1.0, where you can see examples of how to run under `nosh` or `s6`. That URL returned an XML error document from Amazon to me. https://github.com/frioux/cgid/releases/tag/v0.1.0 was a more productive URL. https://github.com/frioux/cgid took me right to the aforesaid doco, though. (-:
nosh version 1.25
The nosh package is now up to version 1.25 . * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project As you may have noticed from discussions elsewhere, a new oom-kill-protect utility has snuck in at the last moment. This takes Linux-style OOM Killer score adjustments (an integer between -1000 and 1000), BSD-style binary YES/NO settings, or a special setting for querying the "oomprotect" environment variable; and tries to do the closest matching thing for each platform. Details are in the manual, of course. With this, the OOMScoreAdjust setting is now converted by the convert-systemd-units utility. The local-syslogd, udp-syslogd, and syslogd service bundles make use of oom-kill-protect with the special environment variable setting in their run programs. So FreeBSD bug #204741 is addressed in a more general fashion that can be easily used in other service bundles. "rcctl set syslogd oomprotect YES" and "rcctl set syslogd oomprotect NO" can be used to turn OOM Killer protection on and off. Other things in this version include: * More configuration import utilities, covering ip6addrctl, webcamd, and NFS settings. * A fix for a problem with configuration import on Linux in version 1.24. * Two minor utilities for querying the fstab database, get-mount-what and get-mount-where, needed by the configuration import for mdconfig (but generally usable). * New binary "run-" packages for OpenSSH server, syslog on a local socket, and klog. * The new syslog and klog packages provide the Debian package manager's virtual package names "linux-kernel-log-daemon" and "system-log-daemon" (per Debian Bug #67604). As can be seen from the roadmap, we are nearing the end of the rc.d conversion for FreeBSD. Additions in this release include nfsserver, gptboot, rtadvd, virecover, and pcdm. Almost all of mdconfig is actually done, bar some after/before orderings. * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d
Re: How to compile nosh?
I was going to warn you that you're about to kick yourself, but I see that Guillermo has already replied, so you're probably already kicking yourself. (-: fREW Schmidt: First off, there's no README or anything, so I sorta have to guess. [...] Again there's no README for redo. As Guillermo said, the instructions for building from source are right there on the WWW pages whence one downloads the source package. * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/source-package.html * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/redo.html Both can be built either in the Debian way into installable packages, with the Debian dpkg-buildpackage tool (with directories prepared with package/debian/prepare); or as a self-contained package, slashpackage style, with the conventional slashpackage commands. FreeBSD/PC-BSD doesn't have Debian toolsets, but there's a bsd/rules mechanism (prepared with package/bsd/prepare) available for building BSD package files with a "Debian-like" system. Or one can build as a self-contained package on BSD, too. As Guillermo said, what you were fiddling with were actually internals of the build system, that it sets up for you. All that you have to remember, whether you choose self-contained package or Debian-esque build, is one command line. It's *the same* command across nosh and redo, moreover. (It is indeed the same command in daemontools, UCSPI-SSL, ftpcopy, and others. I've also employed it in my own source packages that I use for djbdns and publicfile.) * http://www.superscript.com/ucspi-ssl/install.html * https://ohse.de/uwe/ftpcopy/install.html * http://www.fehcom.de/ipnet/ucspi-ssl.html As the (nosh and redo) source package download and build instructions pages say, slashpackage is a whole subject in its own right. One day I might put up a FGA about it. But if one looks to other systems and spots hierarchical package naming, self-contained build-trees, "current version" symbolic links, atomic upgrades, and "index"/"alternatives" directories one can see those all in slashpackage. * https://news.ycombinator.com/item?id=10505897 It is not mine, but the invention of Some Bloke On Another Continent: * http://cr.yp.to/slashpackage.html He didn't really document the build process too well. It's the build process that mainly concerns us, here. Some Other Bloke On A Different Continent To The First Bloke wrote about that part in 2002, though: * https://web.archive.org/web/20021011214224/http://www.skarnet.org/software/compile.html A Third Bloke On A Third Continent (at the time) also wrote about it in 2002: * http://thedjbway.b0llix.net/builddjb.html#slashpackage A Fourth Bloke even got in on the act: * http://code.dogmap.org/slashpackage/
Re: system-control unload-when-stopped (was: nosh version 1.23)
Guillermo: And as a consequence, the Guide ends up never actually explaining what the unload-when-stopped subcommand does (AFAICS, make service-manager unload the named service after its 'stop' file finishes executing, whatever the reason for running it was, so that a subsequent service-status on the service's directory says that no supervisor is running). I really don't want to have to change the manual back, so I might have to get on and make the change. (-: I'll see what I can do for 1.29. 1.28 is pretty much frozen now, and its release is awaiting my writing a WWW page about the MySQL/MariaDB changes. Enjoy a peek at one of the new pages already written: * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/italics-in-manuals.html
Re: nosh: convert-systemd-units' treatment of Standardxxx=tty directives
Guillermo: So, the convert-systemd-units tool recognizes the systemd 'StandardInput=tty' unit file directive, and translates it to a vc-get-tty + open-controlling-tty sequence in the 'run' file of the generated bundle directory. Which does more than just redirecting stdin to the appropriate character device file. Out of curiosity, why is the controlling terminal allocation done there? Is it just to respect systemd's behaviour, or is there a more fundamental reason? It is to provide largely equivalent behaviour, yes. In 1.28, forthcoming soon, you'll see that I spent a little time on this. Quirks mode remains much the same. But in ideal mode there are some tweaks with respect to the defaults, as well as a StandardError=log override. Guillermo: I also note that, unlike the StandardInput case, 'StandardOutput=tty' and 'StandardError=tty' directives are not supported by the conversion tool and ignored with a warning, when it appears that in the absence of a 'StandardInput=tty' directive they could just be translated to an fdredir (or fdredir + fdmove --copy sequence) naming the device file. No controlling terminal required in this case according to the systemd documentation. And when 'StandardInput=tty' is present, 'StandardOutput=tty' (or 'inherit') and 'StandardError=tty' (or 'inherit') look already covered to me by the open-controlling-tty invocation, since that utility redirects stdout and stderr to the terminal, along with stdin. So, again out of curiosity, is there a reason for ignoring these uses of StandardOutput and StandardError? It's an unusual case, that I haven't seen in the wild. The 1.28 changes were partly to make ideal mode more normal but also partly in response to things that I have seen. That, though, I have not. In practice, service unit writers either want the like-a-ttylogin-service scenario -- setting the controlling terminal, allowing input, and all -- or they push logs out to a log service. See http://askubuntu.com/questions/770673/ for just one of many examples of the former use case.
nosh version 1.27
The nosh package is now up to version 1.27 . * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project In fact, it is soon to be version 1.28. This is a somewhat delayed notice for 1.27, because I forgot to send out the notices for versions 1.27 and 1.26 after updating the WWW site. As can be seen from the roadmap, we are at the point in rc.d conversion for FreeBSD/PC-BSD where it's actually easier to count the things that remain unconverted. Discounting the PC-BSD Active Directory services and a handful of suspect FreeBSD services (such as growfs, which doesn't apply to ZFS in the first place) the remaining things to be converted can be counted on the fingers of one hand. The external configuration import mechanism has gained the ability to handle stf, atabridge, mdconfig, and a few others. There are also a whole bunch more service bundles: cross-platform, for Linux, and for BSD. * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d The OOM Killer avoidance measures from version 1.25 are now employed in the PostgreSQL service bundle as well. The systemd service unit conversion tool has gained a whole load of NUMA-related extensions: NUMAInterleave, NUMAMemBind, NUMACPUNodeBind, NUMAPhysCPUBind, NUMALocalAlloc, and NUMAPreferred. These it translates into the equivalent invocations of the numactl chain-loading utility. It has also gained a couple of minor fixes and tweaks. The %m substitution now works, and service bundles comprising FIFOs or AF_LOCAL sockets are now created so that they are ordered after any relevant filesystem mount services. By request, the nosh Guide has gained a whole chapter of cheatsheets, giving quick one-liner pointers to some common tasks. The chapter is divided into three sections: chain loading, logging, and service management. The service management division is subdivided into daemontools-style commands, systemd-style commands, OpenBSD-style commands, SMF-style commands, and common commands. The chain loading division gives a number of the more common commands used in chain-loading run scripts (and whereever else one might want to use them). There have been improvements in static network setup, including fixes for some bugs in static_arp and static_ndp and a more cross-platform replacement for the static-networking service. The nosh-bundles package now supplies several aliases for services, which are just plain old symbolic links. So (for example) one can address the CUPS service as either org.cups.cupsd or just plain cupsd. Things to look forward to in version 1.28 already include: more service bundles; another chain-loading utility; a major revision to MySQL and MariaDB service bundling, to reflect the pushes by their own developers to obviate their rc scripts and the mysql-safe command and just run mysqld directly under service management using the tools provided by the service management system; and a change relevant to the all-important linux_logo command. (-:
nosh version 1.28
The nosh package is now up to version 1.28 . * https://jdebp.eu./Softwares/nosh/ * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project * http://jdebp.info./Softwares/nosh/ There's a lot in this one: MySQL and MariaDB changes; more prophylaxis for Desktop Bus bus activation; improvements to systemd unit conversion; support for the old svc -x; machineenv; improvements to service management; fixes for the per-user manager; improvements to the console terminal emulator; BSD boot mode changes; the ability to pass more open sockets to connection-accepting programs; cron; and OpenBSD. Italics and colour == * https://jdebp.eu./Softwares/nosh/italics-in-manuals.html This isn't a toolset change, per se. But the WWW site now has a guide to seeing actual italic text in manual pages. The nosh toolset's user-space virtual terminals support true italics (if one has the fonts) or obliquing, and this works with them. MySQL and MariaDB changes = * https://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html New in version 1.28 is a different and up-to-date way of managing MySQL and MariaDB server services — where "new" translates to finally getting rid of that unnecessary mysql_safe wrapper and doing things the way that daemontools-family toolset users have wanted to do them since the turn of the century. There's a lengthy exposition on the WWW site, q.v.. The major visible effect is that your "mysql" or "mariadb" service is now an alias, for something like a "mysql@" or "mysql@01" (if you have [mysql01] in your my.cnf) service. The configuration file import mechanism tries to construct/update mariadb@NN and mysql@NN service bundles for you, based upon your MariaDB and MySQL configuration files. Further prophylaxis for Desktop Bus bus activation == * https://jdebp.eu./Softwares/nosh/avoid-dbus-bus-activation.html The nosh toolset now comes with a dbus-daemon-launch-helper replacement. The purpose of this is to sit in your /usr/local/etc/dbus-1/system.conf (or equivalent) and redirect to service management attempts, by the Desktop Bus broker daemon, to demand-start services. It is slightly fiddly to install, requiring manual setup by the system administrator, there being no simple way to add overrides to /usr/local/etc/dbus-1/system.conf and it requiring that you allow the "messagebus" user the necessary access for starting and stopping services (but not necessarily *superuser* access — rembember ACLs). To assist with this, several popular Desktop Bus "services" now exist as alias names for service management services. These are just symbolic links to the service bundle directories, of course. So, for example: With the helper in place, Desktop Bus bus activation will try to demand-start a service named "org.freedesktop.PackageKit" using service management. This is just an alias for the "packagekit" service. Improvements to systemd unit conversion === Ideal mode is now closer to the daemontools-family mainstream, defaulting to the daemontools-family norm of always restarting services. Quirks mode, conversely, now implements more of the non-daemontools redirection semantics for standard I/O, in particular with regard to listening socket units. Some more Linuxisms have been added. Limits (where applicable) can now take SI and IEC suffixes (so you can, say, express limits in kiloseconds). This latter is actually an augmentation to the underlying softlimit command. Passing more open sockets to connection-accepting programs == The improvements to systemd unit conversion also allow passing more than one listen()ing socket to connection-accepting programs. You can use, say, ListenStream and ListenDatagram and the conversion utility will translate this into an appropriate chain of multiple invocations of udp-socket-listen and tcp-socket-listen. It will do local-stream-socket-listen, local-datagram-socket-listen, netlink-datagram-socket-listen, and fifo-listen too. The motivator for this was Daniel J. Bernstein's dnscache. I have modified versions of tinydns, dnscache, and taiclockd that understand the LISTEN_FDS protocol for their being told about listening sockets that have been opened for them, and don't open their own sockets in that case. dnscache, in particular, takes a UDP socket and a listening TCP socket. The UCSPI tools in this version of the toolset can now provide these two to a dnscache process. One simply chains through udp-socket-listen and tcp-socket-listen to dnscache, using the --systemd-compatibility flag. The sharp-eyed will notice that the tinydns and dnscache services are following in the footsteps of the mariadb and mysql services, being instantiated for relevant IP addresses by the configuration import
Re: nosh version 1.28
Guillermo: OpenBSD === [...] There are an awful lot of limitations to OpenBSD, [...] How funny it is that this summary and the WWW page echo the sentiments in skarnet.org packages' source files comments and commit messages :D We didn't collaborate. (-: I don't actually know what M. Bercot has said on the matter. It's not unexpected that two projects sharing several design principles will hit the same problems with OpenBSD, though. The more interesting things to consider are other operating systems. For starters: Ubuntu on Windows NT would possibly be a less problematic port than OpenBSD. Whilst it, too, has obstacles with pseudo-terminals, framebuffers, and the system manager; what it doesn't have, that OpenBSD has, is the difficulty with the package management. Ubuntu on Windows NT has APT like Debian, of course. I've said before, elsewhere, that one could probably successfully get nosh service management, UCSPI support, and log management working on Ubuntu on Windows NT; although obtaining an actual daemon context is still problematic. (https://news.ycombinator.com/item?id=11416376) Moreover: UbuntuBSD and Debian FreeBSD shouldn't have the obstacles with the pseudo-terminals, framebuffers, and the system manager; these, after all, being things that the FreeBSD operating system kernel provides in largely suitable form. (https://news.ycombinator.com/item?id=11326457) Incidentally: I wrote a while ago that UbuntuBSD probably wouldn't use Mewburn rc. It doesn't. UbuntuBSD 16.04, released this month, uses BusyBox init and (the Debian port of) OpenRC rc.
Re: nosh version 1.28
> > I'm going to be very naughty and patch 1.28 post-release. It's a minor > change. > I've changed my mind. I'm going to point the two of you at a potential version 1.29 and see how you get on. This is because I have ended up doing slightly more than a 2-line script patch.
Re: register runsvdir as subreaper
Laurent Bercot: You want runsvdir to be your reaper, so you'd just run "local-reaper runsvdir scandir" instead of "runsvdir scandir". Actually you'd run > local-reaper true runsvdir scandir
Re: register runsvdir as subreaper
Kamil Cholewiński: Reaping orphaned children should be the duty of PID 1. * http://unix.stackexchange.com/a/197472/5132 * http://unix.stackexchange.com/a/177361/5132 There is no objective basis for such a claim, this not actually being a minimal requirement of process #1. Welcome to the future. Your service manager does not have to be process #1. Your interactive logins are ordinary services controlled by your service manager. Orphaned child processes are reparented to your service manager or to some other process that is even closer to them.
Re: nosh version 1.31
Guillermo: One could instead attach controllers to the hierarchy rooted in /sys/fs/cgroup/systemd, for example by mounting it with 'mount -t cgroup -o cpu,name=systemd cgroup /sys/fs/cgroup/systemd' (not with system-manager as process 1), and things would appear to be OK: It's tempting to do that, and make the adjustments to the current control group location code. But the harsh reality I suspect is that version 1 control groups are on the way out. Let's put it this way: I am targetting Debian Linux, notorious for being behind the times, and even that has a kernel (in Debian 8 backports, admittedly) with version 2 control groups now. I'll save the idea up. But there's other, higher priority stuff to do. I've just finally turned unload-when-stopped into a simple service-control --exit wrapper, for example. (-:
nosh packaged for Arch Linux
Someone: I am trying to create one or multiple packages for Archlinux to install nosh This has come up several times with multiple people, now. I have no Arch Linux machines, and I am entirely dependent from Arch Linux people for the packaging part. As I already told one person back in August 2016, this is your opportunity to shine. (-: My second redo is already packaged for Arch, I am told: * https://aur.archlinux.org/packages/redo-jdebp/ * https://git.parabola.nu/abslibre.git/plain/pcr/jdebp-redo/ Here's what I know. The redo packaging runs package/compile followed by package/export. This is not the way to approach packaging the nosh toolset. It's not one giant lump, as you can see from the FreeBSD/TrueOS, Debian, and OpenBSD packages that I have done. What you Arch Linux people with the packaging experience have to provide is a way, on Arch, for one source package to build multiple binaries packages. This should not be hard. It's a very common thing to do, after all. It is, indeed, pretty much the norm in the worlds of Debian and Ubuntu. As you can see from the WWW pages about the FreeBSD/TrueOS, Debian, and OpenBSD packages, there are doco packages, service bundle packages, multiple toolset packages, and multiple -run packages. The way that this works on Debian and the BSDs is largely driven by package/debian/rules and package/bsd/rules, which are in turn determined by Debian's package build system, in particular its dpkg-buildpackage tool. Notice what the rules system does. It runs package/export to populate the ./debian/tmp/ tree, and then package/stage to populate the individual binary package trees (with links) from that in the layout that is actually used within the packages. The Debian Maintainers' Guide and the FreeBSD Porters' Handbook have more on the concept of "staging" when building binary packages. * https://www.debian.org/doc/manuals/maint-guide/build.en.html#completebuild * https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/special.html#staging Coming up with how something like this works on Arch is another part of your opportunity to shine, Arch packaging people. Note that I currently plan to take package/export out of the build process and have package/stage work directly from (a copy of) the original slashpackage tree, i.e. command/ manual/ config/ guide/ et al.. Rearranging everything from slashpackage into another layout under ./debian/tmp/ only in order to rearrange it again in the staging areas seems like an unnecessary complexity. pax -r -w of the original slashpackage layout into ./debian/tmp/ or ./bsd/tmp/ seems a better idea. There has been for a fair while now an important note at http://jdebp.eu./Softwares/nosh/source-package.html that the package/export command may change. It does both too much and too little as a tool for installing from source. It exports one giant lump, and it doesn't do all of the things that the package "maintenance scripts" do. In pursuit of that latter, back in 1.29 the BSD side of package creation was modified to make it possible to auto-generate installation/upgrade/deinstallation scripts for the old and new BSD package tools. This is, as you can see, done with package/bsd/gencontrol, which makes (the very different) +MANIFEST files for both the old and new BSD package tools. This is on the cards for Debian as well. For Arch, the build system will need to process a similar set of files to the various package/bsd/nosh-*.p files, in some way that is appropriate to how such maintenance scripts are handled on Arch, as well as do more specific things for likes of nosh-run-system-manager. Yes, this is vague. I've refactored this into a way of making different maintenance scripts from effectively non-imperative lists of service bundles and user accounts. I'll press on with getting you a set of package/debian/nosh-*.p files, which at least will list the Linux service bundle suite not the BSD one, which can then be patched into arch/*.p files. You have to work out how to then join these lists up with however Arch does installation/upgrade/deinstallation scripts in packages, however. This is another part of your opportunity to shine, Arch packaging people.
nosh with Debian's sysstat package
Someone:
I haven't installed much else yet on the system but I tried the
sysstat package which gives me the following error:
preset: ERROR: sysstat: No such file or directory
I haven't yet investigated this problem. Sysstat seems to be part of
the Debconf enable/disable system, I'm not quite sure how that
interacts with nosh.
This is a good example for general consumption.
The maintenance script for the package is trying to enable the "sysstat"
service using the "update-rc.d" command. You've installed the shim for
this command from the nosh-debian-shims package, so the maintenance
script is actually ending up trying to preset the "sysstat" service
bundle using "system-control preset". You don't have a "sysstat"
service bundle. Yet.
Ironically, if the systemd support in Debian's "sysstat" were as good as
the author's own, you could just make one. The origin package comes
with a systemd service unit, built by passing this through a macro
processor to turn things like @SA_LIB_DIR@ into "/usr/lib/sysstat":
* https://github.com/sysstat/sysstat/blob/master/sysstat.service.in
Debian, however, only builds and packages up a Debian-supplied van
Smoorenburg rc file. It doesn't actually package up the systemd support
that comes from the author. It does provide the /usr/lib/sysstat/sa1
script that is referenced by the systemd service unit, however. So you
could take the sysstat.service.in, manually make a sysstat.service out
of it, and pass that through convert-systemd-units to make a service
bundle that would invoke /usr/lib/sysstat/sa1 .
However, we are heading into systemd House of Horror territory here, as
Debian also provides a "Debianized" version of the sa1 script as
/usr/lib/sysstat/debian-sa1 that does the stuff that Debian's van
Smoorenburg rc script does. So using what's in the box we would have
sysstat.service which sets up settings the systemd way, running the
debian-sa1 script that sets up things the Debian way (reading
/etc/defaults/sysstat), running the sa1 script that sets things up the
RedHat/SuSE way (/etc/sysconfig/sysstat). It's a mess of nested shell
scripts and overlapping configuration mechanisms. And that's
overlooking the surprise secret second service disable mechanism.
The systemd people don't like surprise secret second service disable
mechanisms, and the modern Debian practice is to not have them. The
Debian sysstat package has more than one thing to improve. Moreover
there's no real need for all of these configuration mechanisms,
especially since the underlying command has only two knobs to twiddle in
the first place. So start with a more ideal-world version of what
sysstat should have for systemd: a simple service unit that has 1
configuration mechanism, and cuts out all of the daft middle-men layers
of shell scripting entirely.
1. Take this service unit instead. Call it sysstat.service .
2. Use convert-systemd-units to make a service bundle from it. chown
everything to root if you didn't do this as root.
3. Place that in /var/local/sv/sysstat .
4. install -d -m 0755 /var/local/sv/sysstat/service/env
5. system-control set-service-env sysstat OPTIONS -D
See what happens when you install the package then.
[Unit]
Description=Insert a dummy record in sysstat's current daily data
file to indicate that the counters have restarted from 0.
[Service]
# The service is "ready" after it has run to completion.
Type=oneshot
# This enables controlling service options with rcctl set and rcctl get .
EnvironmentDirectory=env
# Two - characters, note. Also, this is specifically targetting
being converted into a nosh service bundle.
ExecStart=/usr/lib/sysstat/sadc -F -L ${OPTIONS} "${DIR:--}"
[Install]
WantedBy=workstation.target
Re: s6-linux-init: SIGUSR1 and SIGUSR2
Guillermo: > a) SIGTERM for reboot > b) SIGUSR1 for halt > c) SIGUSR2 for poweroff > d) SIGINT for a programmable CTRL + ALT + Del action > e) SIGWINCH for a programmable 'keyboard request' action > > nosh's system-manager supports e) via the 'kbrequest' target. But for > compatibility reasons, supports d) on [GNU/]Linux via the > 'secure-attention-key' target, but not on the BSDs, where SIGINT means > 'halt'. Supports a) and b) on the BSDs, but not a), because SIGTERM > means 'enter rescue mode'. And doesn't support a) to c) on > [GNU/]Linux, because it follows the systemd signal convention (signals > above SIGRTMIN) there instead. Or, to put it another way: On FreeBSD, PC-BSD, and OpenBSD, the nosh system-control and system-manager tools use (b) and (c) for compatibility with existing BSD tools. But for further compatibility with existing BSD tools, SIGINT means (orderly shutdown and) reboot and SIGTERM means bring up rescue mode. To the current FreeBSD/PC-BSD init and OpenBSD init and accompanying toolset, SIGINT means reboot (indivisibly from Ctrl+Alt+Del) and SIGTERM means single-user mode. The FreeBSD/PC-BSD and OpenBSD kernels send SIGINT but don't send SIGWINCH in the first place. If they did, though, it would be regarded as (e). On Linux, the nosh system-control and system-manager are designed to use the Linux kernel and systemd signalling conventions. (d) and (e) are Linux kernel conventions. (a), (b), and (c) are BSDisms that as Guillermo noted have never really been the case on Linux at all. (b) actually outright conflicts with a van Smoorenburg init convention (where it means "restart the initctl server"). It also conflicts with an upstart convention (where it means "re-connect to Desktop Bus"). Moreover, upstart and systemd have made the semantics of SIGTERM into a mess, where for a system-wide systemd/upstart init it means one thing and for a per-user systemd/upstart init it means something quite different. Neither meaning is reboot. Orderly shutdown and halt/reboot/poweroff are various real-time signals, instead. The systemd real-time signals for orderly shutdown and halt/reboot/poweroff are also available on FreeBSD and PC-BSD, in fact. Apart from this last, all of the aforegiven is documented in the system-control and system-manager user manuals. See https://jdebp.eu./FGA/emergency-and-rescue-mode-bootstrap.html for the evolution of single-user mode into rescue mode. The nosh system-manager can in fact do both emergency and rescue modes on FreeBSD/PC-BSD and OpenBSD as well as on Linux. The BSD bootstraps themselves don't have two flags, though.
Linuxisms in s6
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955 , Adrian Chadd: We're using s6 at work, and it works out mostly ok. Mostly once you get around the linuxisms, and the lack of sensible time code in it (its calculations for daemon run duration is based on system time, not wall clock, so if your box boots jan 1, 1970 then gets NTP, things are.. hilarious), and some of the arcane bits to get logging working right. What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have Linuxisms, obviously. But what Linuxisms does s6 have?
Is missing SysV-init support a bug?
Gerrit Pape: To me too this readiness IPC ideas and implementations look over-engineered. A good convention for service programs would be to functionally test for services it needs very early on startup, and fail if dependencies are not available. The service supervisor (any modern init scheme seems to now support this) relaunches eventually, until all dependencies are fulfilled. The problem with the thundering herd approach is twofold. Firstly, it really does matter in practice when the machine has tens if not hundreds of client processes all continually restarting whilst they wait for (say) the RabbitMQ server to come up. Secondly, these explanations never seem to take system shutdown into account. In the ordered services world, shutdown order is the reverse of startup order, and things generally work. In the thundering herd world, often the theory is just to send terminate and kill signals willy-nilly to every service on the system. This almost never works cleanly in any but the most trivial systems. (People will no doubt be thinking the classic example of NFS mounts, here. But there are all sorts of possibilities, from /var/ being unmounted before logging services are turned off to the proxy DNS server being turned off whilst other services are still doing DNS lookups.) We discussed this on the Supervision mailing list last year: http://www.mail-archive.com/supervision%40list.skarnet.org/msg00673.html
Re: s6, listen(8), etc.
Laurent Bercot: fds 6 and 7 are only used for UCSPI clients, which are a very minor subset of the programs you'd want to use that mechanism with. Laurent Bercot: I don't want the caller to tell me "here's a bunch of fds, you sort them out": that's just laziness. Just so that everyone is not operating under any more misapprehensions: One dictum is that if you're using the "LISTEN" protocol for any UCSPI use-case, you are doing it wrong. The name "LISTEN" in "LISTEN_FDS" is a big clue. It's not for accepted socket file descriptors. It's for listening ones. UCSPI deals in accepted connections, conversely. The systemd people reinvented their own, pretty poor, protocol for accepted connections. It is not the "LISTEN" protocol. In 2015 I tried to point them at UCSPI where there's years to decades of existing practice, doco, and implementation (including GNU inetd!) to gain from. See https://lists.freedesktop.org/archives/systemd-devel/2015-June/033299.html (The UCSPI FGA is now http://jdebp.eu./FGA/UCSPI.html , by the way.) If you think that "here's a bunch of file descriptors" is the protocol, then you've missed a subtlety. The protocol is that the list of file descriptors is *ordered*, by the system administrator/package writer. The systemd manual pages explain that it's the order that the various ListenXXX directives occur in the .INI file. (systemd treats .INI files as ordered in various ways, of which this is one.) In the nosh toolset, it's simply the order in which you chain together the chain-loading *-listen tools. Each tool appends a further listening file descriptor to the end of the list that it begins execution with. So whilst in the wild generally programs just scan the whole list looking for the first/last file descriptor whose type (FIFO, socket address family, and so on) they like, on the basis that they are generally only looking for one file descriptor of any type, this is an interpretation that's defined by the particular server programs concerned. They could equally well define something like "You specify my control listening FIFO first, my TCP4 service listening socket second, and my TCP6 service listening socket third in the list.". How the ordered list of file descriptors is treated is defined by the service programs themselves.
Mass bug filing: use and misuse of dbus-launch (dbus-x11)
Simon McVittie: This can already work. If you put XDG_RUNTIME_DIR in user programs' environment, and arrange for your favourite service manager to make a dbus-daemon (or something else that speaks the same protocol) listen on $XDG_RUNTIME_DIR/bus before any user process would try to connect to it, then modern versions of at least libdbus, GDBus and sd-bus will connect to it by default with no additional effort on your part. This client-side code path does not depend on systemd, does not depend on libsystemd (except obviously sd-bus which is part of libsystemd), and is compiled in for all supported Unix platforms. That's the problem. No the whole unix:runtime=yes mechanism is not. As I said, this is something that you and Joe Marcus Clarke and whomever else have to sort out with each other. I'm unfortunately stuck in the middle, here. Please do whatever it is that you need to do with each other to make your program understand address=systemd: and address=unix:runtime=yes on FreeBSD/TrueOS/OpenBSD. It does not do so. Simon McVittie: Meanwhile, if you want the relevant integration files (your favourite service manager's equivalent of systemd units) to be part of dbus (the reference implementation of D-Bus), please propose tested patches; if they follow the "user session" model[1], they could eventually go in dbus-user-session.deb, with its dependencies changed from the current systemd-sysv to "systemd-sysv | your-service-manager". Kudos for being the first project to offer integration, ever. (-: Yes, down the road it would be marvellous if people included service bundles in their own projects. Yes, I'd like to see the day when the number of service bundles in the nosh-bundles package actually starts going down, because people are taking on shipping their own service bundles for their own services, instead of going up. So yes, eventually you'll be taken up on that offer I hope. But one step at a time. Simon McVittie: As for what I would like, I'd like you (where that's plural, including Joe Marcus Clarke or whomever else) to please make either address=systemd: or address=unix:runtime=yes work in your program on FreeBSD/PC-BSD/OpenBSD. To the best of my knowledge, the listenable address "unix:runtime=yes" (as in "dbus-daemon --address=unix:runtime=yes") does work on generic Unix, and should interoperate fine with the XDG_RUNTIME_DIR/bus fallback used by clients with no DBUS_SESSION_BUS_ADDRESS. It is compiled and tested whenever DBUS_UNIX is defined (i.e. everything except Windows), and I haven't seen bug reports about that test failing. There you go, then. New knowledge. (-: It doesn't work with your program as ported to FreeBSD/TrueOS/OpenBSD. Joe Marcus Clarke is the porter for FreeBSD, according to the port information, and Antoine Jacoutot the porter for OpenBSD. This is why I am saying that it's something that you (plural, remember) need to sort out amongst yourselves. We users stuck in the middle cannot use address=systemd: and address=unix:runtime=yes with your program on these systems. As I said, it's something that I had to fix in November 2015, to stop trying to use address=systemd: on FreeBSD/TrueOS because it turned out that it didn't actually work. I thought that address=unix:runtime=yes might, but that did not either. Simon McVittie: I am *not* going to go looking for patches on display at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "beware of the leopard"; Luckily, you didn't need to. The page that I hyperlinked before pointed directly to Pierre-Yves Ritschard's and Cameron T. Norman's actual code, even down to positioning the window around the first lines of the functions. Now if one *did* want to follow the Debian way of having mention of these things buried down in the depths, in a bug report from years ago, then this is a truly excellent example of the genre that one could enjoy. One should begin with Cameron T. Norman's post here, roughly one thousand eight hundred messages down, in a bug report from 3 years ago: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#1859 (-: Simon McVittie: To be brutally honest, there is a fairly low limit to how much benefit I can create by giving new things to PC-BSD users, [...] That's not the right way to look at it. You yourself have just said several times that this is stuff that is supposed to be on "supported Unix platforms". This isn't giving new things to anyone. This is making existing things, that you yourself think are existing, work. I shouldn't dismiss PC-BSD so readily, if I were you, either. PC-BSD (now rebranded as TrueOS Desktop a few days ago -- I just got through changing a whole load of preset file and -run package names.) is the BSD that comes in the box with the desktop environments and with all of the desktop programs that use Desktop Bus. Yes, people can and do run
Re: s6, listen(8), etc.
Daniel Kahn Gillmor wrote: So i'm hoping that it'll find a taker in one of these more toolkit-style supervisor suites. http://jdebp.eu./Softwares/nosh/#Features socket services section
Re: s6, listen(8), etc.
Laurent Bercot: how does the daemon know what fd corresponds to what use? In the wild, it's generally a for() loop over the passed-in descriptors that checks each socket type. In the wild, only one of any type is often the case. "If AF_INET4 and SOCK_DGRAM, this must be my UDP4 socket." That's how I patched dnscache and tinydns to work, too. A simple set of socket_is() functions in the Bernstein socket library, and it is off and away.
Re: Linuxisms in s6
Jonathan de Boyne Pollard: What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have Linuxisms, obviously. But what Linuxisms does s6 have? Adrian Chadd: We just had a bunch of fun trying to get it to build right, [...] Such as what, specifically?
Re: Linuxisms in s6
Adrian Chadd:
[...] the uptime stuff really threw us.
It's unfair to lay such system time problems at s6's door. Systems whose
system clock jumps 46 years during system bootstrap don't get to blame
s6 for mad time gaps that appear in logs and service start time
records. There is a *lot* of the Unix and Linux worlds that depends
from time being right. It's not just s6 that is affected by such
things. You note crypto. There are a lot of other things as well that
have unstated, sometimes undocumented, and sometimes surprising
dependencies upon system time being current.
Here's one such.
For quite a while, Linux distributions had rather an odd problem at
bootstrap. They'd repeatedly fsck volumes at every bootstrap when they
need not have. And this didn't affect U.S. or U.K. people, which is in
part why it persisted for so long.
* https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/63175
* https://bugs.archlinux.org/task/17438
* http://lwn.net/Articles/264498/
The problem was that people were erroneously running their real-time
clocks in local time rather than UTC, and this triggered an odd hidden
dependency upon having the right time in the system clock in countries
where local time was in advance of UTC. The Linux method for handling
RTCs erroneously running in local time is for the system bootstrap to
make a special settimeofday() call that effectively tells the kernel
what the UTC offset is for the RTC hardware. This could happen *after*
the fsck of the root volume, however. So whilst that fsck was
happening, the kernel was assuming that UTC was the local time that it
had taken from the RTC and initialized its system clock with. In
effect, as soon as the special settimeofday() call was executed, the
system clock would jump backwards by one or more hours, to what UTC
actually was.
But the ext2/3/4 filesystem format has last checked/mounted/written
timestamps in its superblock. Part of the checking to see whether a
full fsck is needed at bootstrap is comparing them to the current time.
If they are in the future by hours or more, something is clearly wrong,
thinks fsck, and it runs the full check. At bootstrap, when the initial
fsck (of at least the root volume and sometimes other volumes as well)
is run, the system clock is not UTC yet. Comedy results.
Both systemd and the nosh system manager have to ensure that they do the
special settimeofday() system call before they start off service
management and thus run mount/fsck services, or indeed anything else
that might have a closet dependency from not stepping the system time by
hours partway through bootstrap. The nosh system-manager's manual page
has a whole section on this subject.
FreeBSD/PC-BSD has a mechanism for correctly reading a RTC that is
erroneously in local time. One sets up the RTC's offset from UTC in the
machdep.adjkerntz variable in /boot/loader.conf{,.local} and the system
clock never has to jump by hours during bootstrap. I've yet to
experience a FreeBSD/PC-BSD system where the installer actually
configures this, though.
Interestingly, FreeBSD/PC-BSD also has a fallback mechanism that uses
the latest volume mount timestamp that it can find as the initial system
time when no hardware clock device registers at bootstrap. Presumably
you have a clock device that registers but it is not battery-backed,
your volumes don't preserve (or reset) their mount timestamps, or you
are encountering the comedy situation where FreeBSD/PC-BSD is setting
the system clock to 1970-01-01 because the last time around it mounted
the filesystems before the clock was corrected.
Re: Linuxisms in s6
Adrian Chadd: Sure, but I'm looking for something more generic than just devd. Like, notifications about events like "default route is up" can be done by sniffing the rtsock, but notifications like "ntpdate has updated the date, we can now do crypto services" doesn't happen there right now. You're reinventing upstart. The lesson of upstart is that whilst the event-driven paradigm looks like the bright shiny future, once one gets down to the details it is a lot harder than it at first appears. I strongly recommended learning about upstart, and especially learning the problems that people hit with it, to anyone going down the same route. The Debian systemd Hoo-Hah had some lengthy discussion of upstart. (I regret not having bookmarked the discussion that I once came across, where someone opined that xe preferred systemd to upstart because at a Linux conference the systemd presentation had been exciting and had been put forward as the wave of the future, where upstart had been presented as old-school, traditional, and boring. Ironically, this person wasn't aware that the designs are exactly the opposite of that. upstart has the novel event-driven design where the system is configured with the information that event A triggers programs P, Q, and R, and the system starts by raising a "first event", that runs programs, that raise further events, that run further programs. Whereas it is systemd that has the conventional design, shared by Mewburn rc and others, of starting from a goal, working through a dependency tree, and doing topological sorts.) The Debian people chose to improve a non-event-driven architecture instead. It's a lesson to be learned from SMF, in fact. One can have a lot more additional abstract targets, such as "/milestone/name-services" and "/milestone/system-clock", and dependencies to and from them. The world is not 2 to 4 run levels plus "DAEMON", "NETWORKING", and "$local-fs". That said, something like this hypothetical "/milestone/system-clock" is a milestone that would need to be reached *very* early on in the bootstrap process. Fixing up the clock is something that both the nosh system manager and systemd handle themselves directly, outwith of service management. More on this in a moment.
Re: Problem with s6-softlimit -c
Jan Olszak:
Command that fails:
s6-softlimit -c 1 pwd
# strace s6-softlimit -c 204800 pwd
...
prlimit64(0, RLIMIT_CORE, NULL, {rlim_cur=RLIM64_INFINITY,
rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_CORE, {rlim_cur=200*1024, rlim_max=RLIM64_INFINITY}, NULL)
= 0
You seem to have mis-spelled "works". (-: The program has made the
system calls that you told it to, and they have succeeded, setting the
soft limit that you instructed.
Re: ruinit-rpm systemd service file
Otheus: From systemd's viewpoint, the *service *is runsvdir, and not runsvdir-start. The latter is simply a wrapper script for the former and needed because of initttab's limitations. With systemd, a service file can contain all the information in that script and more. Similarly: http://jdebp.eu./FGA/inittab-is-history.html#svscanboot
Re: Adding capability control into the `run' script comparison page
Guillermo: I suppose the interesting suprise is that as consequence, when a service definition gets 'imported' to nosh from a unit file (and this covers pretty much everything in the nosh-bundles* binary packages),the corresponding service gets placed in a cgroup of its own when launched by nosh's service manager: This also happens in per-user service management. /service-manager.slice/user-services@.service: └─user-services@jdebp.service ├─31423 per-user-manager ├─per-user-manager-log.slice │ └─31426 cyclog --max-file-size 32768 --max-total-size 1048576 . └─service-manager.slice ├─31427 service-manager ├─gvfs-daemon.service │ └─9305 /usr/lib/gvfs/gvfsd ├─at-spi-dbus-bus.service │ ├─9276 /usr/lib/at-spi2-core/at-spi-bus-launcher │ ├─9290 /usr/bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.co... │ └─9293 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session ├─dbus-servers-log.service │ └─30089 cyclog jdebp/dbus-servers/ ├─pulseaudio.service │ └─3625 pulseaudio --exit-idle-time=-1 ├─mpd.service │ └─31592 mpd --no-daemon --stderr --stdout --verbose ./mpd.conf ├─emacs.service │ ├─6088 strace -f emacs --daemon │ └─6094 emacs --daemon ├─cyclog@.service │ ├─cyclog@pulseaudio.service │ │ └─20138 cyclog jdebp/pulseaudio/ │ └─cyclog@mpd.service │ └─18028 cyclog jdebp/mpd/ ├─simple-servers-log.service │ └─6070 cyclog jdebp/simple-servers/ └─gnome-terminal-server.service ├─9408 /usr/lib/gnome-terminal/gnome-terminal-server ├─9411 gnome-pty-helper ├─9412 zsh └─9451 sleep 6000
subreapers
Martin "eto" Misuth:
I think Mr Jonathan de Boyne Pollard might be cooking, or even already
has, something similar in nosh.
Long since. (-: It was in version 1.0 .
Martin "eto" Misuth:
at some point I was interested in digging out whether systemd had
"subreapers" at it's disposal, and why it didn't use them
It has. The systemd people were responsible for getting this mechanism
into Linux in the first place.
* http://unix.stackexchange.com/a/177361/5132
Martin "eto" Misuth:
- observing behaviour of reparenting under init for thousands of times
in htop
Now observe it under not init and not htop. (-:
Here's the nosh service manager's subreaper mechanism in action. Process
31427 is a per-user service manager, which is a subreaper. Process 9408
is GNOME Terminal and process 9412 is a Z Shell running connected to
that terminal. In the shell, I ran (sleep 6000&) which resulted in an
orphaned sleep process, process 9451. As you can see, that process has
been reparented to the nearest subreaper.
per-user-manage(31423)-+-cyclog(31426)
`-service-manager(31427)-+-3(6088)
|-4(31592)-+-{decoder}(31598)
| |-{io}(31596)
| |-{output:default
}(31599)
| `-{player}(31597)
|-at-spi-bus-laun(9276)-+-dbus-daemon(9290)
|
|-{dconf worker}(9277)
|
|-{gdbus}(9279)
|
`-{gmain}(9291)
|-at-spi2-registr(9293)---{gdbus}(9296)
|-cyclog(6070)
|-cyclog(18028)
|-cyclog(20138)
|-cyclog(30089)
|-emacs(6094)---{gmain}(6095)
|-gnome-terminal-(9408)-+-gnome-pty-helpe(9411)
|
|-zsh(9412)---pstree(9482)
|
|-{dconf worker}(9409)
|
|-{gdbus}(9410)
|
`-{gmain}(9413)
|-gvfsd(9305)---{gdbus}(9306)
|-pulseaudio(3625)---{null-sink}(3626)
`-sleep(9451)
services that need DISPLAY
Daniel Kahn Gillmor: Yet surely there are some user-wide services that don't need DISPLAY at all, and would be happy to run per-user? GNOME Terminal isn't one of them. Witness the behaviour of gnome-terminal-server run as a service if it doesn't have a DISPLAY environment variable: @4000584830f10ae09b9b Unable to init server: Could not connect: Connection refused @4000584830f10ae0dbb8 Failed to parse arguments: Cannot open display:
passing the listening socket as an open file descriptor
Martin "eto" Misuth: I personally am not so "hot" about this listening socket passing stuffs. When you've had to deal with tens if not hundreds of different ways of saying "listen on this IP address and port", you'll come around to the idea of having a single tool that does this one job universally. (-:
Re: GNU Emacs now runs in foreground
Martin "eto" Misuth: On my presonal box "user level" s6 /services subtrees are in `.config/s6/host` For comparison: ~/.config/service-bundles/services/ and ~/.config/service-bundles/targets/ Martin "eto" Misuth: #!/bin/sh exec mpd --no-daemon /usr/home/eto/.config/mpd/mpd.conf Tip: In the daemontools world all services have ther own service directory, at minimum; and this directory is the working directory of the service. You can put an mpd.conf that is particular to the service in the service directory itself. Martin "eto" Misuth: Getting tmux "supervised" was bit tricky as it double forks the daemon if not running, and is thus is "escaping under init" under normal conditions. Note: This can be easily handled on FreeBSD and TrueOS. See NOTE_TRACK in EVFILT_PROC for kevent(). Martin "eto" Misuth: It would be great if tmux project allowed to spawn raw user level daemon by explicit command, One of the interesting developments over the past couple of decades is how much the world has been influenced to come around to the daemontools way of doing this. I've observed before, elsewhere, the number of daemon programs, especially in the BSD worlds, that have in that time gained -F/--foreground or similar options that tell them not to do a whole bunch of this unnecessary stuff. It's in a way somewhat saddening to see the new option to emacs that started this thread. It's welcome; but the sad part is that it's a decade and a half behind quite a lot of others. For some systems like emacs the process has been glacially slow. MySQL is another. daemontools users were asking about letting daemontools do the service management and doing away with mysqld_safe in 2002. It took thirteen years for it to happen ... in MariaDB. Oracle, where software goes to die, may or may not have done the same for MySQL. PostgreSQL lags further behind. * http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Prompt
Re: GNU Emacs now runs in foreground
Jonathan de Boyne Pollard: One of the interesting developments over the past couple of decades is how much the world has been influenced to come around to the daemontools way of doing this. I've observed before, elsewhere, the number of daemon programs, especially in the BSD worlds, that have in that time gained -F/--foreground or similar options that tell them not to do a whole bunch of this unnecessary stuff. Martin "eto" Misuth: I think this is bit related to daemon(8) program which acts as poor man's runsv, with all the "advandages" of pid files. It isn't, though. Several of the manual pages that I've seen explicitly mention daemontools. Here's winbindd's manual page, for example: -F If specified, this parameter causes the main winbindd process to not daemonize, i.e. double-fork and disassociate with the terminal. Child processes are still created as normal to service each connection request, but the main process does not exit. This operation mode is suitable for running winbindd under process supervisors such as supervise and svscan from Daniel J. Bernstein's daemontools package, or the AIX process monitor.
Re: NOTE_TRACK, EVFILT_PROC, kqueue, and subreapers
Jilles Tjoelker: This should probably be fixed. There's another more insidious bug hiding inside kevent() somewhere that causes a kernel abend complaining about a sleeping thread holding a non-sleepable lock. One needs to make fairly heavy use of kevent() in order to trigger it, I believe, as I have only seen it so far on systems that do.
Re: GNU Emacs now runs in foreground
Daniel Kahn Gillmor: dbus-user-session supports at most one graphical session concurrently ... in order to avoid people encountering the very problem of half-hearted and flawed implementations that I described. Non-half-hearted implementations are the goal, however. Read https://lists.debian.org/debian-devel/2016/08/msg00554.html and http://www.mail-archive.com/supervision@list.skarnet.org/msg01393.html .
Re: Adding capability control into the `run' script comparison page
Casper Ti. Vector: But I do think the capability argument has its validity: chainloading is, at this time, not well known to normal users, which is why many systemd supporters compulsorily identify cgroup support with systemd with few people opposing. Therefore I suggest to add some examples of capacility control (eg. one example for ulimit, plus one example for cgroup) into the comparison page, or an independent page. Such "systemd supporters" don't actually know systemd. * http://jdebp.eu./FGA/linux-control-groups-are-not-jobs.html To anyone running the service manager and bundles from nosh version 1.28 or later on Linux: You are encouraged to look at your control group hierarchy, with a tool like "systemd-cgls /", with the "cgroup" field of the ps command, or by simply listing your /sys/fs/cgroup/ hierarchy. You are in for an interesting surprise.
Re: nosh per-user service management
Jean Louis:
So, placing user daemons into system supervision may not be the best
option, due to so many customization that have to be done for the
user, especially for GNU Emacs -- as one cannot know which programming
languages and their variables are required.
I just explained that these are *not* system-wide services, but per-user
ones. A user who is setting up environment variables for xyr own needs
to run emacs as a service simply sets up environment variables for xyr
own needs against the per-user service. Indeed, I already showed how
that is done. Once again:
Adjust its environment, if desired, in the conventional way
$ system-control --user set-service-env emacs DISPLAY :15.2
or (if /usr/local/sbin is on one's path)
$ rcctl set --user emacs DISPLAY :15.2
The idea that this is somehow difficult because one might have to set an
environment variable named GUILE_LOAD_PATH in this way, is just plain
wrong. This is just an envdir in a conventional place in a service
directory. It's actually easier to manipulate than a $HOME/.{z,}profile
or a $HOME/.login_conf for setting such an environment variable so that
one could spawn the daemon in an interactive login session.
subreapers
Martin "eto" Misuth: Point of tool is simple: it always runs marking itself as "subreaper", thus any descendants who lose parent, that was running under it, will get reparented (and their attached process subtrees as well) under it's process. Once "main", the important and original, child exits, tool tries to term and then kill all remaining descendants. By the way, you should talk to this person: * http://unix.stackexchange.com/questions/329631/
Re: Adding capability control into the `run' script comparison page
Jonathan de Boyne Pollard: To anyone running the service manager and bundles from nosh version 1.28 or later on Linux: You are encouraged to look at your control group hierarchy, with a tool like "systemd-cgls /", with the "cgroup" field of the ps command, or by simply listing your /sys/fs/cgroup/ hierarchy. You are in for an interesting surprise. There are more interesting surprises in the same vein in 1.31. I've put a sneak peak of the 1.31 Guide up for you.
Re: GNU Emacs now runs in foreground
Jean Louis: emacs --new-daemon=NAME I have added a new per-user service for this to nosh, ready for version 1.30 . So one just has to start the per-user service manager # system-control start user@jlouis.target then start the emacs server $ system-control --user start emacs Adjust its environment, if desired, in the conventional way $ system-control --user set-service-env emacs DISPLAY :15.2 or (if /usr/local/sbin is on one's path) $ rcctl set --user emacs DISPLAY :15.2
Re: nosh version 1.29
Bloody Thunderbird! Here's that again, I hope without the surprise reformatting after pressing "send" this time: The nosh package is now up to version 1.29. * http://jdebp.eu./Softwares/nosh/ * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project * http://jdebp.info./Softwares/nosh/ There's been a lot going on since version 1.28 . 2016 leap second The TAI to UTC conversions know about the forthcoming leap second. service bundles --- As usual, there are several new service bundles, from powerd++ through zfsd to fwknopd. The new fs-servers target allows one to order the initialization of NFS servers before loop-to-self NFS mounts. The new multi-user-pre target is another ordering target that allows services such as the motd file updater to be ordered before TTY login services. The instantiated kdm@tty7 and kdm@ttyv6 services have been replaced with a single kdm service, with a view to dealing with display managers better in the future. I have some plans in this area. The Samba service names have been fixed. Debian calls them nmb, smb, and winbind; but the Samba doco and most places on the WWW call them nmbd, smbd, and winbindd. The latter names are used in the service bundles package, with aliases pointing to them from the Debian names. doco The doco has been improved and kept up-to-date in various places, including correct descriptions of set-service-env and print-service-env after one confused user got in touch. PC-BSD is now named as TrueOS where the reference is not historical. code review --- As a result of some code review that was offered, std::auto_ptr is now gone and a rare memory corruption bug in safe_execvp() has been fixed. Building from scratch when one doesn't have a prior daemontools or freedt toolset installed also no longer hits a bug. configuration import improvements - In an effort to clear those last few remaining items on the nosh roadmap, a whole load of configuration import (pppd, sppp, rfcomm_ppp, dhclient, wpa_supplicant, natd, and hostapd) has been consolidated under the umbrella of static-networking. I plan to expand this further in 1.31, given how much is already in 1.30. Linux kernel VTs Management of Linux kernel virtual terminals has some improvements, including setting UTF-8 canonical mode editing and keyboard composition modes, and emitting the control sequences that set up the screen saver. tai64nlocal changes --- tai64nlocal has adopted a minor but important change from the BSD and GNU C libraries: before reading the start of a line it flushes its output. This came from trying to use it as a co-process in GNU awk. To prevent deadlocks, GNU awk co-processes need to be in what is effectively line buffered output mode even though their standard inputs and outputs are not terminal devices. This is now the case for tai64nlocal and it can be used to convert TAI64N timestamps as a GNU awk co-process. FreeBSD and TrueOS packaging The largest change, however, is in the FreeBSD/TrueOS and OpenBSD packaging. This is a change that is going to happen in the Debian packaging in a later version. It's partly to simplify the package maintenance, and partly a step towards having OpenBSD packages that work. A single package description is fed to both the new pkg tool that exists on FreeBSD/TrueOS and the old pkg tool that exists on OpenBSD. It's not perfect, as there are things that are easy with the new pkg tool that are hard with the old one; and the OpenBSD packages are still not fully functional. But things are better than they were. The OpenBSD service bundles package now almost properly sets up per-service user accounts and log directories, for example. === === IMPORTANT UPGRADE NOTE FOR FreeBSD/TrueOS: === === An important consequence of the aforementioned is that the semantics of the nosh-bundles package have changed. In earlier versions, the various nosh-run-* packages were how one set services running, except for a small rump set of services that were set up by the nosh-bundles package. This is now no longer the case. The nosh-bundles package now presets and starts no services at all. *All* running of services must be achieved with the nosh-run-* packages or some other sets of scripts and presets. To this end, there are now two new packages, nosh-run-freebsd-desktop-base and nosh-run-freebsd-server-base. These parallel the already existing nosh-run-trueos-desktop-base and nosh-run-trueos-server-base packages; except that they do not start any of the services that exist in TrueOS but do not exist in FreeBSD, such as the various pc-* services. You must
Re: Adding capability control into the `run' script comparison page
Casper Ti. Vector: the docs are in tarballs on jdebp.eu * http://jdebp.eu./Softwares/nosh/guide.html
Re: Adding capability control into the `run' script comparison page
Casper Ti. Vector: one example for ulimit An irony here is that the page *already contains* two entire sets of examples that set memory resource limits, using daemontools, daemontools-encore, freedt, perp, s6, and nosh tools.
GNU Emacs now runs in foreground
Martin "eto" Misuth: First, there are two major caveats, There are actually three. They break scripting. For example: People cannot use the GNOME Editor as $VISUAL or $EDITOR because one of the things implicit in the $EDITOR/$VISUAL mechanism is that when the program that has been invoked exits, the editing is over and the file being edited has been saved in the desired form. That is not the behaviour of the "small and lightweight" GNOME Editor, however. * http://unix.stackexchange.com/questions/201900/ * http://unix.stackexchange.com/a/323700/5132 * https://news.ycombinator.com/item?id=13056252 Other "interesting" problems result from the move that the Desktop Bus and the Desktop Environment people are making away from per-session instances of the D-BUS daemon to per-user instances of the same. This causes fun with deciding what the daemon's $DISPLAY should be set to. A per-session Desktop Bus obviously has one $DISPLAY by and large. But a per-user Desktop Bus not only has to handle multiple logins from a single user, it has to handle that the per-user session management can be running when there's no X server at all. (systemd starts its per-user instances via PAM hooks that act upon every login, including logins over SSH and on terminals.) Even though some daemons try to take the approach that the daemon supports multiple $DISPLAYs, sent in from multiple clients as part of the client session, one unfortunately finds that the daemons themselves still have to have an arbitrary $DISPLAY in order to start up in their initial, not connected to any clients and their displays yet, mode. In practice, thus, the implementation of the user-wide client-server idea is half-hearted and flawed in this respect.
Re: djbwares version 4
Jonathan de Boyne Pollard: In celebration of the forthcoming leap second, djbwares is now at version 4. * http://jdebp.eu./Softwares/djbwares/ * http://jdebp.info./Softwares/djbwares/ Jean Louis: http://jdebp.info./Softwares/djbwares is not working: "access denied" and I instinctively tried that one first, as to avoid .eu (even it makes no sense). You should have just tried the URL that I gave to you, without your changing it to something different. Ironically, Bernstein publicfile is part of the package at hand, and this is the documented behaviour of publicfile, in its original Bernstein manual: > A request for http://v/f refers to the file named ./v/f inside the root directory hierarchy, if f does not end with a slash. > httpd will refuse to read a file if the file [...] is anything other than a regular file: a directory, socket, device, etc. publicfile isn't going to let you read the WWW server's directories directly with URL tricks. You attempt that in vain. (-: For *not* trying to trick the WWW server, and simply reading the blurb and the download instructions, just use the actual URL that I gave.
Re: s6 talk at FOSDEM 2017
Your goo.gl hyperlink on page 34 is to the old WWW site. (-:
Re: nosh: service-control --exit
Guillermo: for some reason 'service-control --exit' doesn't unload a service that is already in the 'stopped' state. I'll look into it for 1.31. You won't see a fix in 1.30 because that is coming out ... well ... now.
nosh version 1.30
The nosh package is now up to version 1.30 . * http://jdebp.eu./Softwares/nosh/ * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project * http://jdebp.info./Softwares/nosh/ service bundles --- As usual, there are more service bundles, including for the UWSGI "Emperor" and the new services in FreeBSD/TrueOS 11 such as ypldap. There are now services to run Sendmail in the same manner as the services that run exim. Note that this is slightly different to the old FreeBSD division of labour. There are individually controllable services for SMTP Submission, SMTP Relay, the Submission queue runner, and the Relay queue runner. doco The Guide has been extended with several new chapters, including a gazetteer of interesting directories, a chapter on log file post-processing, a chapter on logging security, a chapter on per-user service management, and some notes for individual services. The commands list has moved from the blurb into the Guide, too, as it seems like something that an administrator might find handy to have available when there's no Internet connection. * http://jdebp.eu./Softwares/nosh/guide.html service management -- There's now a hardlimit chain-loading command, analogous to softlimit. The convert-systemd-services utility now makes use of this and permits setting separate hard and soft limits, or only one or the other, with settings like LimitOFILE=32:128 and LimitNPROC=:infinity . There's now a local-reaper chain-loading command, that can turn "local reaper" status for the current process on or off. Have a care when using this, per the note on the manual page. There is a LocalReaper=true extension to systemd service units for this. netlink-datagram-socket-listen is now available on the BSDs for script compatibility. It always aborts with an address family error. There's a new hangup subcommand of system-control, equivalent to the existing -H option to svc . enhancements to system-control stop/start/reset and single-shot services This is the first big item for 1.30 : The start and stop subcommands of system-control now operate more quickly. Instead of polling once per second, they monitor the supervise/status files of each service that is in the process of being started and stopped, with kevent(). In addition, system-control now supports the notion of services that become ready when their main process has exited, marked with a new flag file in the service directory. convert-systemd-units has been modified to convert "oneshot" services to this, instead of to services that put all of the run code into the start program. Thus "oneshot" services that are running their actual main programs are reported as "running" by svstat, rather than as "starting". This takes advantage of the extended status information that service-manager has been writing to the status file since version 1.28. The sharp-eyed may have noticed that in version 1.28 the output of "svstat"/"system-control status" gained information about the exit statuses of the start, run, restart, and stop programs. This is what system-control now uses to detect whether ready-after-run services ran before they stopped. (Detection of ready-after-run services that are running with no processes, because they are "remain" services, can be and is done with just the daemontools-encore-compatible status information.) Old-style "oneshot"s will continue to work as before, as of course they become ready as soon as the run process is spawned, which is after they have run their programs as part of start. The benefit of this new style, apart from reporting a running service as actually "running", which should help with nagios monitoring and the like, is that "oneshot" services converted from systemd no longer have to be marked as RemainAfterExit=true in order to avoid a dummy "pause" process hanging around. This is the case for old-style "oneshot" services. They have to run something in run, after all, and that something has to keep running in order for the service to be considered ready and services ordered after it to be unblocked. A ready-after-run service, however, unblocks ordered-after services if it has reached the stopped state via a run, thus puts its programs in run, thus doesn't have to have a dummy pause process, and can be RemainAfterExit=false without adding to the process list. log file management --- export-to-rsyslog had a bug that caused it to skip old log files (the @.s ones) in catch-up mode. This has been corrected. There is now a follow-log-directories command that can substitute for tail -F . It knows the actual structure of log directories, operates using one or more cursors like export-to-rsyslog does, and copes correctly with cyclog/multilog log rotation (which
Re: s6 talk at FOSDEM 2017
Casper Ti. Vector: resource control and [...] cgroup support can be easily implemented with chainloading I pointed at the nosh Guide back in December. The sharp-eyed will notice the advent of a new command in the command list chapter. This is a sneak preview of 1.31.
GNU Emacs now runs in foreground
Martin "eto" Misuth: - rxvt-unicode - uberterminal - this thing can operate [...] daemon, when single process hosts all your terminals - benefits are [...] Drawbacks are that it doesn't understand receiving the listening socket as an already-open file descriptor, and by default it places the socket in $HOME/.urxvt/ rather than in /run/user/$USER/urxvt/ . Martin "eto" Misuth: - mpd - the music player daemon This can accept listening sockets passed to it. I've added to the nosh bundles package a service bundle for it as a system-wide instance, and generators in the configuration import subsystem for per-user instances. You'll see these in version 1.30.
djbwares version 5
djbwares is now at version 5. * http://jdebp.eu./Softwares/djbwares/ * http://jdebp.info./Softwares/djbwares/ This contains some long-overdue changes: ip6.int has been replaced by ip6.arpa in tinydns-data and dnscache, and rblsmtpd no longer falls back to using an RBL that has been defunct for many years. It also contains some additions: some UCSPI-SSL capability, a new gopherd UCSPI server to go alongside httpd and ftpd in publicfile, and most of the previously missing manual pages (including a few for commands which had no manuals in the original toolsets). There are no longer any placeholder manual pages for the "man" command. There are still a few manual pages that are only present in roff form, though. You can see gopherd in action: * gopher://jdebp.info./1/Repository/
Re: nosh version 1.33
Talking of Gentoo, Guillermo: The Gentoo people have been inspired by Archnosh, but it looks like they could benefit from your experience. * https://wiki.gentoo.org/wiki/Nosh
Re: Vim `.swp' files in nosh source tarballs
I purge the directories of these from time to time, as I have just done again. But they tend to very slowly accrue, for one reason or another. It's not a major worry. Anyway, better that the archive have everything including some harmless text editor temporary files than that it have one vital file missing because of a wildcard exclusion/deletion, which I have experienced in the past. (-:
Immortal: "based on daemontools & runit"
"Immortal", a tool that is a "*nix cross-platform (OS agnostic) supervisor based on daemontools & runit" according to its blurb, has just come up on Hacker News. I mentioned M. Bercot. * https://news.ycombinator.com/item?id=13994687
Re: nosh version 1.34
Guillermo: > > So, system-control has a new but undocumented 'is-service-manager-client' > subcommand? > Yes. It's part of an incomplete to-do list item; and might change.
Re: nosh git tree
Suggestions are welcome. You can put them on the Supervision mailing list if you like. Please bear in mind the design principles laid out in the Guide http://jdebp.eu./Softwares/nosh/guide.html , though. Bug reports and real world experiences are also welcome, and in fact more interesting. Several recent developments have come from bug reports. The recent changes for the per-user service bundle for gpg-agent came about from a bug report from a person who has a more up-to-date version of gpg-agent than is available on any of my test operating systems, for example. Again, you can use the Supervision mailing list if you like. There is a Roadmap in the WWW blurb http://jdebp.eu./Softwares/nosh/ with some of the items on my own to-do list on it. The to-do list itself has somewhat more on it. Currently, for example, I am further improving the external configuration import, so that /etc/system-control/convert/rc.conf aggregates more stuff on non-BSD systems. Arch users might want to contribute to Archnosh http://repo.or.cz/archnosh.git . Gentoo users might want to contribute to the Gentoo doco https://wiki.gentoo.org/wiki/Nosh and bring it up to the Archnosh level. Debian, Ubuntu, FreeBSD, and TrueOS users note that I supply full package repositories http://jdebp.eu./Softwares/package-repositories.html already. There is no git repository, but the curious can see old source archives via GOPHER or FTP there. The FTP listing is determined by the server's directory order, but they are neatly segregated in the GOPHER menu. It is of course the same repository being served over HTTP, FTP, and GOPHER, this being publicfile. Yes, GOPHER http://www.mail-archive.com/supervision@list.skarnet.org/msg01657.html . (-:
Re: s6-svscanboot, how to exit?
> Type=forking No. > StandardOutput=tty No. > RemainAfterExit=yes No. > SysVStartPriority=99 No. You're a long-time daemontools user, you say. Why on Earth do you think that Type=forking is right? Why on Earth do you think that good daemons fork and exit parent? Why on Earth do you think that svscan -- any svscan, from daemontools-encore svscan to s6-svscan -- forks and exits parent? Or needs a TTY? * http://jdebp.eu./FGA/inittab-is-history.html * http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems.html * http://untroubled.org/daemontools-encore/svscan.8.html * http://www.skarnet.org/software/s6/s6-svscan.html * https://packages.debian.org/sid/runit-systemd * https://packages.debian.org/sid/daemontools-run * https://wiki.archlinux.org/index.php/Runit#Installation * https://wiki.gentoo.org/wiki/Runit#Starting_the_supervision_tree * https://framagit.org/taca/archnosh#nosh-service-manager-only-nosh-run-via-systemd
Re: s6 as a systemd alternative
Casper Ti. Vector: > > (Normally Jonathan would be replying to this point, but I still do not see > him in this thread, so I rashly take this job ;) > Goodonyer. Thank you. I have been distracted. * https://github.com/neovim/neovim/pull/6816 https://github.com/neovim/neovim/pull/6816 I have got back to 1.34 and added some more bits and pieces this week. Looking at the changelog, it is probably time to release that and begin work on 1.35.
Re: s6 as a systemd alternative
Charles Duffy > - Integration with the linux-only cgroups mechanism for managing CPU, > memory, and I/O throughput limits As you say, this is just an exercise in chain loading tools and conversion to scripts that employ them. * https://news.ycombinator.com/item?id=11846083 * http://jdebp.eu./Softwares/nosh/guide/move-to-control-group.html
Runit and perp doco improvement for chpst and runuid
This week's little kerfuffle over systemd and the way that its mechanism to drop superuser privileges works (see https://news.ycombinator.com/item?id=14681377 and the headlined bug report there) has led me to observe that whilst the doco for nosh, s6, daemontools, and daemontools-encore all state that their various setuidgid programs will abend on being given an unknown/invalid account, the doco for chpst in runit and runuid in perp do not come out and explicitly say the same. runuid's doco implies it, albeit indirectly; and I have double-checked (not that I was in much doubt) that chpst actually has this behaviour. So it might be worth improving the doco for chpst and runuid to actually say this outright.
nosh version 1.34
The nosh package is now up to version 1.34 . * http://jdebp.eu./Softwares/nosh/ * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project * http://jdebp.info./Softwares/nosh/ Once again, there are a few more service bundles. The most interesting ones in this version are perhaps the finish-update and finish-install targets, designed to be invoked the first bootstrap after an update or install has been done, and the users target, which is used to auto-start per-user subsystems at bootstrap. Several NFS service bundles are now common across operating systems. And the OpenVPN service bundles are now split into separate client and server services. Several minor bugs have been fixed here and there: a duplicated newline in line-banner that was throwing off publicfile FTP service; a problem with recordio on FreeBSD/TrueOS; and a problem with attempts to use slashes in environment variables in service bundle environment directories. The user-space virtual terminal emulator now implements the Xterm extensions to DECSCUSR, and the framebuffer realizer can display the resultant cursor shapes. This can be made use of by programs such as Neovim. There are now separate service bundles and nosh-run- packages for running eudev and systemd-udev, because the two are now significantly divergent. The various utilities for changing the process environment no longer use the GNU C library/BSD C library functions for doing so, and so no longer suffer from the concomitant memory leaks that their manual pages used to warn about. The convert-systemd-units tool has been slightly enhanced, for the benefit of a fix that has been made to the per-user gpg-agent service. The external configuration imports system has been extended. It now deals with importing the hostname configuration value, taking that responsibility away from and simplifying the set-dynamic-hostname utility. It now imports various Debian and other kernel virtual terminal settings, from /etc/kbd/config, /etc/default/console-setup, and /etc/vconsole.conf . And network configuration import now can set up services for both dhcpcd and dhclient. /etc/system-control/convert/rc.conf now contains more settings on Linux operating systems as a result, including dhclient_program.
Re: A dumb question
Laurent Bercot: > > "It's old software. Its last version is from 2014." > > Old? Let's see... A more amusant counterexample is perhaps the Debian Almquist shell, put into widespread use on Debian and Ubuntu a decade ago in order to speed up bootstrap. M. Almquist first published it in May 1989. Its Debian maintainer is Gerrit Pape. * https://wiki.ubuntu.com/DashAsBinSh * https://wiki.ubuntu.com/DashAsBinSh/Spec * https://lists.debian.org/debian-release/2007/07/msg00027.html * https://wiki.debian.org/DashAsBinSh * https://tracker.debian.org/pkg/dash
Re: A dumb question
Laurent Bercot: > It is also true that it would benefit from closer maintenance. > Gerrit Pape, runit's author, is still around and still reads this list, > but is not as active as he was a few years ago (typically during > runit development). Fortunately, there are not many feature requests. The Debian maintainer is now Dmitry Bogatov. * https://tracker.debian.org/pkg/runit
Re: small proxy
Jorge Almeida: > As said in the first mail, I need to redirect some targets to a ssh tunnel, > and let everything else go its way. Rather than using a proxy, the solution > seems obvious: [...] ... use a Proxy Auto-Configuration script, as that is the function of PAC scripts. * http://jdebp.eu./FGA/web-browser-auto-proxy-configuration.html
Davin McCall's comparison of service and system management
https://github.com/davmac314/dinit/blob/master/doc/COMPARISON#L71 It's a shame that this gets a number of things wrong about several of the systems.
Re: bug in tcpserver?
Laurent Bercot:
s6-tcpserver doesn't have this bug. :P
Neither do tcp-socket-accept, nor
local-{stream,seqpacket}-socket-accept. (-:
Just for the Hell of it, I have patched the djbwares' tcpserver to have
if (0 != s) and if (1 != s) conditions, for version 7 whenever that will be.
Re: cannot set terminal process group
DGSJ: Well, Crtl+C doesn't work, [...] It seems to be the time of year for this question. I just answered another variant of it, complete with daemontools-style chain loading tools. Read the further reading and other questions, too. * https://unix.stackexchange.com/questions/410318/ * https://unix.stackexchange.com/questions/255385/ * https://unix.stackexchange.com/questions/338214/
SpaceFM using nosh service management
For a wider readership: > > I recently had issues where Firefox and the SpaceFM file manager would > hang when started. Apparently they were calling dbus-launch. > If they were calling dbus-launch, that means that they were unable to contact a Desktop Bus broker and tried to run their own broker in the same session as themselves. Run system-control status dbus and system-control --user status dbus to check that the system-wide and your per-user Desktop Bus brokers are running and to see their logs. Also look at /run/dbus/ and whether you have a DBUS_SYSTEM_BUS_ADDRESS variable pointing at a different place. I cannot speak to Firefox but a quick check with SpaceFM reveals that, once it can contact a Desktop Bus broker, it tries to contact and auto-start the org.freedesktop.UDisks2 server over the system-wide Desktop Bus. So you'll need a system-wide org.freedesktop.UDisks2 service that runs that server. I've added such a service bundle for 1.36.
nosh version 1.36
The nosh package is now up to version 1.36 . * http://jdebp.eu./Softwares/nosh/ * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project * http://jdebp.info./Softwares/nosh/ More Java tools This release comes with the |find-default-jvm| and |find-matching-jvm| tools, which will set up the |JAVA_HOME| environment variable to point to a default/matching JVM directory, using the FreeBSD/TrueOS and Debian conventions for locating JVM directories. To match these, |convert-systemd-units| now recognizes |JVMDefault|, |JVMVersion|, |JVMOperatingSystem|, and |JVMManufacturer| extensions to the systemd unit file format. Tool improvements |convert-systemd-units| now recognizes a |MachineEnvironment| extension to the systemd unit file format, which controls the generation of an invocation of |machineenv|. It also now recognizes and translates |RDMAHCAHandlesMax| and |RDMAHCAObjectsMax| settings. The |unshare| command now has flags for specifying process ID and user ID namespaces on Linux. The |setup-machine-id| command now correctly falls back to the old D-Bus files on FreeBSD, which it had not been doing because of a bug. New system management features In support of an initiative by Warner Losh, there is support for power cycling via hardware and a kernel that support it. The system manager treats |SIGRTMIN+6|, unused in the systemd system, as a request to invoke a new |powercycle| service bundle; and |SIGRTMIN+16|, similarly unused, as the underlying actual powercycle request, which it translates to either |RB_POWERCYCLE| if it is present in the C library headers, or |RB_AUTOBOOT| if it is not. There is a new |system-control powercycle| subcommand, which defaults to sending these signals. Note that the binary packages are currently built on a system that lacks |RB_POWERCYCLE| in the C library. The compatibility |shutdown|, |reboot|, |halt|, and |poweroff| commands all now sport a new |-c|/|--powercycle| option. There are new |fastpowercycle| and |powercycle| commands. The |system-control init| subcommand now sports a new |c|/|C| argument, by analogy to |h|/|H|. And this is of course thus reflected automatically in the compatibility |telinit| command and the |initctl-read| server. Service bundles Fixing an oversight in 1.35, the per-user |dbus| services are now renamed to |dbus-daemon| too. There are a few more service bundles, including ones for |jenkins|, |apacheds|, |udisks2|, and |ndppd|. The |linux-utmp| service bundle has been retired, in favour of a unified |utx| service bundle, which was previously FreeBSD-only, that operates across platforms. In support of this, there is a new |login-update-utmpx| command, and a new |freebsd-shims| package that aliases that to the |utx| command on non-FreeBSD platforms.
Re: New reboot flag: -c for 'power cycle'
Warner Losh, FreeBSD and embedded systems developer, has just invented a new shutdowngoal, in addition to the ones that we already know. In addition to the conventional reset, power off, halt, and kexec goals; xe has added a power-off-and-then-on-again goal. Xe has named it power cycle, and its use case is systems with hardware that can trigger a full power cycle under the kernel's controland where an ACPI reset or even an EFI ResetSystem() of type EfiResetColdjust do not cutthe mustard. It involvesanew flag to the FreeBSD reboot() system call, RB_POWERCYCLE; new FreeBSD kernel functionality for enacting this with hardwares that support it; new -c (for "cycle") flags to the various reboot, halt, poweroff, and shutdown commands; and a new signal sent to process #1. * https://marc.info/?l=freebsd-arch=150886821625848=2 * https://reviews.freebsd.org/D12777 This of course impacts those of us who write process #1 programs. Since SIGWINCH to process #1 is already taken by Linux, I have adjusted my softwares to use SIGRTMIN+7 and SIGRTMIN+17 for the signals to process #1 for this, ready for when Linux eventually catches up with this idea. (-: (There is an on-going discussion of whether FreeBSD's old init program should use real-time signals, too. It currently does not.But that is a side issue here. I have also added to my softwares powercycle and fastpowercycle commands and subcommands, a "c" or "C" argument to the compatibility telinit program (by analogy to the existing letters, although I might have to pick another letter), a powercycle target (again by analogy to existing targets), and a --powercycle long option equivalent for -c .) Please do not use SIGRTMIN+7 and SIGRTMIN+17 for an incompatible purpose.
Re: [RFC] prctl: (Please do not!) Deprecate non PR_SET_MM_MAP operations
In case you weren't aware, programs can make use of a facility on Linux to make their ps listings contain appropriate command-line arguments and environment strings. There's also a similar kernel API on FreeBSD. This is used by the built-ins in nosh to good effect, even more so in the forthcoming version 1.38, allowing system adminstrators to debug things more easily from ps listings by showing them the actually effective states of the chained argument and environment vectors rather than what they were at the beginning of a chain of built-in commands. * https://unix.stackexchange.com/a/438007/5132 * https://unix.stackexchange.com/a/432681/5132 The subject of deprecating and removing the API that permits this came up recently on the Linux Kernel Mailing List. I raised my concerns, and the Linux Kernel developers have been laudably responsive on the subject. As I said to them, I would welcome a better API, that allows setting these things atomically, start and length together as a pair. * https://lkml.org/lkml/2018/4/3/739 * https://lkml.org/lkml/2018/5/6/127 * https://lkml.org/lkml/2018/5/7/78 * https://lkml.org/lkml/2018/5/7/79 * https://lkml.org/lkml/2018/5/7/80 * https://lkml.org/lkml/2018/5/7/81
