Re: Readiness notification for systemd
On 16/06/2015 20:40, Avery Payne wrote: Logging generally (but not always) implies calling printf() with a newline at some point. What if we could come up with a simple standard that extends your newline concept into the logging output? A newline itself may be emitted as part of a startup string sent to a log, so I can't be assured that a daemon is really ready. But what if we could ensure that a universally agreed pattern were present in the log? Something like "Ready.\n" when the daemon is up. We literally could watch the stdout/stderr for this pattern and would solve the entire "readiness notification" problem in one go. Well, I really don't like that approach because it mixes a control flow and a data flow; and when you do that, bad things tend to happen. Data flows are big beasts that 1. require efficiency, and 2. will really mess you up if you take just one wrong step. Data analysis is really a job for application programs; for system software, I'd rather not get in the way, and just let logs flow. (If you add a log filter just for readiness notification, you can't remove that filter afterwards. You have to keep filtering the log 100% of the time. It's possible - on the skaware mailing-list, Patrick needed such a setup - but I wouldn't want it to be a standard, it's too kludgy and inefficient.) -- Laurent
Re: Readiness notification for systemd
On 6/13/2015 11:48 AM, Laurent Bercot wrote: It's a wrapper for daemons using the simple "write a newline" readiness notification mechanism advertised by s6, which converts that notification to the sd_notify format. This had me tossing around some ideas yesterday while I was headed home. Most (but not all) daemons provide logging. Logging generally (but not always) implies calling printf() with a newline at some point. What if we could come up with a simple standard that extends your newline concept into the logging output? A newline itself may be emitted as part of a startup string sent to a log, so I can't be assured that a daemon is really ready. But what if we could ensure that a universally agreed pattern were present in the log? Something like "Ready.\n" when the daemon is up. We literally could watch the stdout/stderr for this pattern and would solve the entire "readiness notification" problem in one go. It would be an enormous problem politically because most daemon authors aren't going to add the 3-4 lines of code needed to support this. But I could dream...
Re: Readiness notification for systemd
On 13/06/2015 22:53, Steve Litt wrote: That's easy enough. One problem though: It seems like everything I write to stderr ends up in my readproctitle display in ps. By default, svscan only redirects stdout from your services to your loggers. So stderr falls through; it's actually the svscanboot script that redirects it, into a pipe to readproctitle. IOW, if you're using svscanboot, your services' default stderr will be the pipe to readproctitle. That's why it's often preferrable to copy stdout to stderr at the start of your run scripts: start them with "exec 2>&1". So your services' stderr will be redirected to the dedicated loggers. -- Laurent
Re: Readiness notification for systemd
On Sat, 13 Jun 2015 22:35:13 +0200 Laurent Bercot wrote: > On 13/06/2015 21:46, Steve Litt wrote: > > When I write my own daemons (for use with daemontools, > > daemontools-encore, runit, and soon to be S6), I use stdout to > > write to the log. Wouldn't the writenewline/close interfere with > > that? > > It's more idiomatic to actually use stderr to write to the log. > stderr is traditionally "the place where I write diagnostic > messages for an external observer", and stdout is traditionally > "the place where I write stuff as a part of my workflow". > Logs fall in the former category, not in the latter; and daemons > usually have no use for stdout. You could arguably call it a > misdesign in daemontools (repeated in runit and s6) that the > logging pipe is tied to the daemon's stdout, not its stderr, by > default; fortunately, it's very easy to fix in the run script. > Almost all of my run scripts start with "fdmove -c 2 1", which > redirects stdout to stderr (the shell equivalent would be > "exec 2>&1"). That's the behaviour I would also recommend for > your daemons. > > That said, sdnotify-wrapper, as well as s6-notifywhenup, comes > with a "-d" option that allows you to choose what descriptor > the notification will be read on. So if you're dead set on > logging to stdout, you can write your readiness newline to > descriptor 3, for instance, and document this; the admin wanting > to run your daemon will simply give the "-d 3" option to the > wrapper. > > > > How do I take advantage of sdnotify-wrapper if I'm using S6? > > You don't use sdnotify-wrapper, because you're already using > a kickass supervision suite and have no need for crutches. :) > > > > How do I take advantage of sdnotify-wrapper if I'm using > > daemontools-encore? > > You don't. You only use sdnotify-wrapper if you're an > administrator using systemd to manage a daemon that refuses to > use sd_notify() and uses the "write a newline to a file > descriptor" notification mechanism instead. > > The point of the wrapper is to prevent having to include > systemd code in the daemon's code to make it systemd-compatible. > If you've never been tempted to use sd_notify() in the code you > write, and if you are not using systemd as a supervision > framework, then you can quietly ignore everything about it. :) That's easy enough. One problem though: It seems like everything I write to stderr ends up in my readproctitle display in ps. SteveT Steve Litt June 2015 featured book: The Key to Everyday Excellence http://www.troubleshooters.com/key
Re: Readiness notification for systemd
On 13/06/2015 21:46, Steve Litt wrote: When I write my own daemons (for use with daemontools, daemontools-encore, runit, and soon to be S6), I use stdout to write to the log. Wouldn't the writenewline/close interfere with that? It's more idiomatic to actually use stderr to write to the log. stderr is traditionally "the place where I write diagnostic messages for an external observer", and stdout is traditionally "the place where I write stuff as a part of my workflow". Logs fall in the former category, not in the latter; and daemons usually have no use for stdout. You could arguably call it a misdesign in daemontools (repeated in runit and s6) that the logging pipe is tied to the daemon's stdout, not its stderr, by default; fortunately, it's very easy to fix in the run script. Almost all of my run scripts start with "fdmove -c 2 1", which redirects stdout to stderr (the shell equivalent would be "exec 2>&1"). That's the behaviour I would also recommend for your daemons. That said, sdnotify-wrapper, as well as s6-notifywhenup, comes with a "-d" option that allows you to choose what descriptor the notification will be read on. So if you're dead set on logging to stdout, you can write your readiness newline to descriptor 3, for instance, and document this; the admin wanting to run your daemon will simply give the "-d 3" option to the wrapper. How do I take advantage of sdnotify-wrapper if I'm using S6? You don't use sdnotify-wrapper, because you're already using a kickass supervision suite and have no need for crutches. :) How do I take advantage of sdnotify-wrapper if I'm using daemontools-encore? You don't. You only use sdnotify-wrapper if you're an administrator using systemd to manage a daemon that refuses to use sd_notify() and uses the "write a newline to a file descriptor" notification mechanism instead. The point of the wrapper is to prevent having to include systemd code in the daemon's code to make it systemd-compatible. If you've never been tempted to use sd_notify() in the code you write, and if you are not using systemd as a supervision framework, then you can quietly ignore everything about it. :) -- Laurent
Re: Readiness notification for systemd
On Sat, 13 Jun 2015 20:48:15 +0200 Laurent Bercot wrote: > > Hello, > > A part of systemd's Embrace And Extend strategy is to entice > daemon authors to link against the systemd library (and thus make > every possible daemon systemd-dependent) by pretending it is the > only way for them to notify readiness. > > As a step to obstruct that, I have written "sdnotify-wrapper". It's > a wrapper for daemons using the simple "write a newline" readiness > notification mechanism advertised by s6, which converts that > notification to the sd_notify format. So, it makes it possible for > daemons to *not* use the sd_notify() interface and still be properly > managed under systemd. > > http://skarnet.org/software/sdnotify-wrapper.c And now the questions begin :-) When I write my own daemons (for use with daemontools, daemontools-encore, runit, and soon to be S6), I use stdout to write to the log. Wouldn't the writenewline/close interfere with that? How do I take advantage of sdnotify-wrapper if I'm using S6? How do I take advantage of sdnotify-wrapper if I'm using daemontools-encore? I really like the fact that you've taken the fight to systemd. By all means, let *them* add the extra word in their unit files. SteveT Steve Litt June 2015 featured book: The Key to Everyday Excellence http://www.troubleshooters.com/key
Re: Readiness notification for systemd
In fairness, readiness notification does seem to require application-specific integration in general under current constraints of most Unixes. Good work, nonetheless. sd_notify isn't complicated, to their credit. On 06/13/2015 02:48 PM, Laurent Bercot wrote: Hello, A part of systemd's Embrace And Extend strategy is to entice daemon authors to link against the systemd library (and thus make every possible daemon systemd-dependent) by pretending it is the only way for them to notify readiness. As a step to obstruct that, I have written "sdnotify-wrapper". It's a wrapper for daemons using the simple "write a newline" readiness notification mechanism advertised by s6, which converts that notification to the sd_notify format. So, it makes it possible for daemons to *not* use the sd_notify() interface and still be properly managed under systemd. http://skarnet.org/software/sdnotify-wrapper.c Enjoy, Bug-reports welcome.
Readiness notification for systemd
Hello, A part of systemd's Embrace And Extend strategy is to entice daemon authors to link against the systemd library (and thus make every possible daemon systemd-dependent) by pretending it is the only way for them to notify readiness. As a step to obstruct that, I have written "sdnotify-wrapper". It's a wrapper for daemons using the simple "write a newline" readiness notification mechanism advertised by s6, which converts that notification to the sd_notify format. So, it makes it possible for daemons to *not* use the sd_notify() interface and still be properly managed under systemd. http://skarnet.org/software/sdnotify-wrapper.c Enjoy, Bug-reports welcome. -- Laurent
