Re: thoughts on rudimentary dependency handling
Laurent Bercot: I'm of the opinion that packagers will naturally go towards what gives them the less work, and the reason why supervision frameworks have trouble getting in is that they require different scripting and organization, so supporting them would give packagers a lot of work; whereas sticking to the SysV way allows them not to change what they already have. Especially with systemd around, which they already kinda have to convert services to, I don't see them bothering with another out of the way packaging design. So, to me, the really important work that you are doing is the run script collection and standardization. If you provide packagers with already made run scripts, you are helping them tremendously by reducing the amount of work that supervision support needs, and they'll be more likely to adopt it. nosh 1.12 comes with a collection of some 177 pre-built service bundles. As I said to the FreeBSD people, I have a goal of making the 155 service bundles that should replace most of FreeBSD rc.d . (There are a load of non-FreeBSD bundles in there, including ones for VirtualBox services, OpenStack, RabbitMQ, and so forth. This is why I haven't reached 155 even though I've made 177.) It also comes with a tool for importing system service and socket units into service bundles. And the nosh Guide chapter on creating service bundles has pointers to the run file collections by Gerrit Pape, Wayne Marshall, Kevin J. DeGraaf, and Glenn Strauss. xdg-open /usr/local/share/doc/nosh/creating-bundles.html Incidental note: I just added another service bundle, for nagios, to version 1.13 because of this: http://unix.stackexchange.com/questions/179798#179798 Enjoy this, too: http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/run-scripts-and-service-units-side-by-side.html
Re: thoughts on rudimentary dependency handling
Avery Payne: But from a practical perspective there isn't anything right now that handles dependencies at a global level. Now you know that there is. The nosh design is, as you have seen, one that separates policy and mechanism. The service-manager provides a way of loading and unloading services, and starting and stopping them. The dependency system is layered entirely on top of that. Whilst one can use svscan/service-dt-scanner and svc/service-control and have things run the daemontools way, one can alternatively use the start/stop/enable/disable subcommands of system-control and have service dependency management instead. The trick is that dependency management is calculated by the system-control program. When you tell it systemctl start workstation.target it follows all of the wants/, required-by/, and conflicts/ symbolic links recursively to construct a set of start/stop jobs for the relevant services. Then it follows the after/ and before/ symbolic links to turn that set into a ordered graph of jobs. Finally, it iterates through the graph repeatedly, sending start and stop commands to the service manager for the relevant services and polling their statuses, until all jobs have been enacted. conflicts/ is actually easy, although it took me two tries. If A/wants/B exists, then a start job for A creates a start job for B. If A/conflicts/B exists then a start job for A creates a stop job for B. The important point is that the service manager is entirely ignorant of this. It is just told to start and stop individual services, and it knows nothing at all of dependencies or orderings. (A dependency is not an ordering relationship, of course, but the manual page that I said to read has already explained that. (-:) It's all worked out outwith the service manager. Which means, of course, that it is alterable without changing the service manager. And indeed, nosh demonstrates such alteration in action. The dependency-based system that one gets with system-control is one of two alternative policies that come in the box; the other being an old daemontools-style system with a /service directory and ./log relationships, as implemented by the service-dt-scanner (a.k.a. svscan) daemon. Again, the trick is that the daemontools-style stuff is all worked out in svscan, and the service manager proper has no dealings in it. The service manager provides a flexible plumbing layer. Higher layers decide how they want that plumbing put together.
Re: thoughts on rudimentary dependency handling
John Albietz: I wonder if this will help address a common situation for me where I install a package and realize that at the end of the installation the daemon is started using upstart or sysv. At that point, to 'supervise' the app, I first have to stop the current daemon and then start it up using runit or another process manager. On Debian, for one, they aren't started using upstart or sysv (whatever that is). Maintainer scripts enable them with update-rc.d and start them with invoke-rc.d. You are expected to have update-rc.d and invoke-rc.d tools that are appropriate to your init system, as well as the respective control files of course. openrc comes with update-rc.d and invoke-rc.d that understand openrc scripts. The sysv-rc package comes with update-rc.d and invoke-rc.d that understand systemd units, upstart jobs, and System 5 rc.d scripts. Ironically, the systemd and upstart packages do not come with their own update-rc.d and invoke-rc.d commands, relying instead upon the sysv-rc package to supply them. This is all a bit shakey and rickety, though. One well-known fly in the ointment is that what may be a single init.d script for System 5 rc may be multiple service and socket units for systemd, and stopping a socket-activated service for package upgrade might not do the right thing as the socket activation might activate the service unit mid-upgrade. Last year, I gave Debian Developer Russ Allbery a patch for an improved version of the Debian Policy Manual that sets this out more clearly than the current one. You might want to get it off him. The sections (of the patched document) that you are interested in are 9.3.1.2, 9.3.6, 9.3.6.1, 9.3.6.2, and 9.3.6.3.
Re: thoughts on rudimentary dependency handling
On 1/19/2015 2:31 PM, Jonathan de Boyne Pollard wrote: Avery Payne: * implement a ./wants directory. [...] * implement a ./needs directory. [...] * implement a ./conflicts directory. [...] Well this looks familiar. I ducked out of ./needs and ./conflicts for the time being; if I spend too much time with making those features then the scripts won't move forward. I'm already several weeks behind in my own schedule that I have set for scripts. Before you read further, including to my next message, get yourself a copy of nosh and read the manual pages therein for service-manager(1) and system-control(1), paying particular attention in the latter to the section entitled Service bundles. Then grab the nosh Guide and read the new interfaces chapter. On a Debian system this would be: xdg-open /usr/local/share/doc/nosh/new-interfaces.html Sounds like I have my homework cut out. I will do so as soon as I can, although I warn you that it joins an already-long list of material to read and think about.
Re: thoughts on rudimentary dependency handling
On 08/01/15 17:53, Avery Payne wrote: The use of hidden directories was done for administrative and aesthetic reasons. The rationale was that the various templates and scripts and utilities shouldn't be mixed in while looking at a display of the various definitions. Why shouldn't they be mixed in? Surely better to see everything clearly and plainly, than to hide some parts away where people won't expect to find them. I think this may confuse people, especially if they use tools that ignore hidden directories. At least on my Debian system, there are hardly any other hidden files or directories of note under /etc, so this would be setting a bit of a precedent to have quite so many non-trivial items present. The other rationale was that the entire set of definitions could be moved or copied using a single directory, although it doesn't work that way in practice, because a separate cp is needed to move the dot-directories. Move everything down one level then? sv - templates - run-scripts - other-things-not-yet-thought-of FWIW, I started trying to make a debian package, and dpkg got very upset about all those dot files. The basic directory structure is as follows: sv - .bin - .env - .finish - .log - .run Where: sv is the container of all definitions and utilities. Best-case, the entire structure, including dot directories, could be set in place with mv, although this is something that a package maintainer would be likely to do. People initially switching over will probably want to use cp while the project develops. That way, you can pull new definitions and bugfixes with git or mercurial, and copy them into place. Or you could download it as a tarball off of the website(s) and simply expand-in-place. So there's a few different ways to get this done. .bin is meant to store any supporting programs. At the moment this is a bit of a misnomer because it really only stores the framework shunts and the supporting scripts for switching those shunts. It may have actual binaries in the future, such as usersv, or other independent utilities. When you run use-* to switch frameworks, it changes a set of symlinks to point to what should be the tools of your installed framework; this makes it portable between all frameworks, a key feature. .env is an environmental variable directory meant to be loaded with the envdir tool. It represents system-wide settings, like PATH, and some of the settings that are global to all of the definitions. It is used within the templates. .finish will hold ./finish scripts. Right now, it's pretty much a stub. Eventually it will hold a basic finish script that alerts the administrator to issues with definitions not launching, as well as handling other non-standard terminations. .log will hold ./log scripts. It currently has a single symlink, ./run, that points to whatever logging system is the default. At the moment it's svlogd only because I haven't finished logging for s6 and daemontools. Eventually .log/run will be a symlink to whatever loggin arrangement you need. In this fashion, the entire set of scripts can be switched by simply switching the one symlink. .run will hold the ./run scripts. It has a few different ones in them, but the main one at this time is run-envdir, which loads daemon specific settings from the definition's env directory and uses them to launch the daemon. Others include an optional feature for user-defined services, and basic support for one of three getty. I may or may not make a new one for the optional dependency feature; I'm going to see if it can be standardized within run-envdir first. I can always remove the dots, but then you would have these mixed in with all of the definitions, and I think it will add to the confusion more than having them hidden. As it stands, the only time you need to mess with the dot directories is (a) when setting them up for the first time, or (b) when you are switching your logging around. Otherwise there's really no need to be in them, and when you use ls /etc/sv to see what is available, they stay out of your way. If there is a better arrangement that keeps everything in one base directory for easy management but eliminates the dots, I'll listen. Although I think this arrangement actually makes a bit more sense, and the install instructions are careful to include the dots, so you only need to mess around with them at install time. On Thu, Jan 8, 2015 at 8:20 AM, Luke Diamand l...@diamand.org wrote: Is it possible to avoid using hidden files (.env) as it makes it quite a lot harder for people who don't know what's going on to, um, work out what's going on. Thanks! Luke
Re: thoughts on rudimentary dependency handling
Here's an ugly hack that allows you do that using envdir:
set -a
eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort | uniq -u)
set +a
Ugh, in the morning (almost) light it's even uglier than I thought,
because it won't work for values you *change* either, which could
be important for things like PATH. Or if values contains a _=
or such.
Try this instead:
set -a
eval $(envdir ../.env env | grep -v -e ^_= -e ^SHLVL=)
set +a
It will reexport all the variables you already have, which is
also ugly, and it still won't remove anything, but at least it
will pick up changes to existing variables.
Ah, the shell. Everytime I doubt whether execline was worth
writing, it only takes a few minutes of /bin/sh scripting to
renew my confidence.
It will work fine. I'm attempting to pre-load values that will remain
constant inside the scope of the script, so there isn't a need to change
them at runtime.
My point is, if you are using this construct in run scripts, remember
the scripts will inherit some environment from init - that could contain
system-wide settings such as PATH or TZ. If for some reason you need to
remove or modify those variables in a specific script, it is important
that it works as expected. :)
--
Laurent
RE: thoughts on rudimentary dependency handling
I'll be following this intently as I have a project I'm working on that will
use s6 heavily even discretely.
Sent from my Windows Phone
From: Avery Paynemailto:avery.p.pa...@gmail.com
Sent: 1/7/2015 11:58 PM
To: supervision@list.skarnet.orgmailto:supervision@list.skarnet.org
Subject: Re: thoughts on rudimentary dependency handling
On Wed, Jan 7, 2015 at 6:53 PM, Laurent Bercot ska-supervis...@skarnet.org
wrote:
Unfortunately, the envdir tool, which I use to abstract away the daemons
and settings, only chain-loads; it would be nice if it had a persistence
mechanism, so that I could load once for the scope of the shell script.
Here's an ugly hack that allows you do that using envdir:
set -a
eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort |
uniq -u)
set +a
Thanks! When I can carve out a bit of time this week I'll put it in and
finish up the few bits needed. Most of the dependency loop is already
written, I just didn't have a somewhat clean way of pulling in the
$CMDWHATEVER settings without repeatedly reading ./env over and over.
It only works for variables you add, though, not for variables you remove.
It will work fine. I'm attempting to pre-load values that will remain
constant inside the scope of the script, so there isn't a need to change
them at runtime.
Re: thoughts on rudimentary dependency handling
On Wed, 7 Jan 2015 14:25:28 -0800 Avery Payne avery.p.pa...@gmail.com wrote: On Wed, Jan 7, 2015 at 7:23 AM, Steve Litt sl...@troubleshooters.com wrote: I'm pretty sure this conforms to James' preference (and mine probably) that it be done in the config and not in the init program. To satisfy Laurent's preference, everything but the exec cron -f could be commented out, and if the user wants to use this, he/she can uncomment all the rest. Or your run script writing program could have an option to write the dependencies, or not. I've pretty much settled on a system-wide switch in sv/.env (which in the scripts will show up as ../.env). The switch will, by default, follow Laruent's behavior of naive launching, ie. no dependencies are up, missing dependencies cause failures, I'm having trouble understanding exactly what you're saying. You mean the executable being daemonized fails, by itself, because a service it needs isn't there, right? You *don't* mean that the init itself fails, right? This is all pretty cool, Avery! Currently, my best guess is that eventually my daily driver computer will be initted by runit (with Epoch as a backup, should I bork my runit). It sounds like what you're doing will ease runit config by an order of magnitude. and the admin must check logging for notifications. Enabling the feature would be as simple as echo 1 /etc/sv/.env/NEEDS_ENABLED ...and every new service launch would receive it. You could also force-reload with a restart command. Without the flag, the entire chunk of dependency code is bypassed and the launch continues as normal. I'm not sure what you're saying. Are you saying that the dependency code is in the runscript, but within an IF statement that checks for ../env/NEEDS_ENABLED? The goal is the same but the emphasis has changed. This will be considered a fall-back feature for those systems that do not have such a tool available, or have constraints that force the continued use of a shell launcher. It is the option of last resort, and while I think I can make it work fairly consistently, it will come with some warnings in the wiki. For Laurent, he wouldn't even need to lift a finger - it fully complies with his desires out of the box. ;-) As new tools emerge in the future, I will be able to write a shunt into the script that detects the tool and uses it instead of the built-in scripted support. This will allow Laurent's work to be integrated without messing anything up, so the behavior will be the same, but implemented differently. Now THAT'S Unix at work! Finally, with regard to the up vs actually running issue, I'm not even going to try and address it due to the race conditions involved. The best I will manage is to first issue the up, then do a service check to confirm that it didn't die upon launch, which for a majority (but not all) cases should suffice. Yes, there are still race conditions, but that is fine - I'm falling back to the original model of service fails continually until it succeeds, which means a silently-failed child dependency that was missed by the check command will still cause the parent script to fail, because the daemon itself will fail. It is a crude form of graceful failure. So the supervisor starts the parent again...and again...until the truant dependency is up and running, at which point it will bring the parent up. Like I said, this will be a fall-back feature, and it will have minor annoyances or issues. Yes. If I'm understanding you correctly, you're only going so far in determinint really up, because otherwise writing a one size fits all services thing starts getting way too complicated. I was looking at runit docs yesterday before my Init System presentation, and learned that I'm supposed to put my own Really Up code in a script called ./check. Right now the biggest problem is handling all of the service tool calls. They all have the same grammar, (tool) (command) (service name), so I can script that easily. Getting the tools to show up as the correct command and command option is something else, and I'm working on a way to wedge it into the use-* scripts so that the tools are set up out of the box all at the same time. This will create $SVCTOOL, and a set of $CMDDOWN, $CMDUP, $CMDCHECK, etc. that will be used in the scripts. **Once that is done I can fully test the rest of the dependency concept and get it fleshed out.** If anyone wants to see it, email me directly and I'll pass it along, but there's not much to look at. If I read the preceding correctly, you're making service tool calls for runit, s6, perp and nosh grammatically identical. Are you doing that so that your run scripts can invoke the init-agnostic commands, so you just have one version of your scripts? However you end up doing the preceding, I think it's essential to thoroughly document it, complete with examples. I think that the additional layer of
Re: thoughts on rudimentary dependency handling
The use of hidden directories was done for administrative and aesthetic reasons. The rationale was that the various templates and scripts and utilities shouldn't be mixed in while looking at a display of the various definitions. The other rationale was that the entire set of definitions could be moved or copied using a single directory, although it doesn't work that way in practice, because a separate cp is needed to move the dot-directories. The basic directory structure is as follows: sv - .bin - .env - .finish - .log - .run Where: sv is the container of all definitions and utilities. Best-case, the entire structure, including dot directories, could be set in place with mv, although this is something that a package maintainer would be likely to do. People initially switching over will probably want to use cp while the project develops. That way, you can pull new definitions and bugfixes with git or mercurial, and copy them into place. Or you could download it as a tarball off of the website(s) and simply expand-in-place. So there's a few different ways to get this done. .bin is meant to store any supporting programs. At the moment this is a bit of a misnomer because it really only stores the framework shunts and the supporting scripts for switching those shunts. It may have actual binaries in the future, such as usersv, or other independent utilities. When you run use-* to switch frameworks, it changes a set of symlinks to point to what should be the tools of your installed framework; this makes it portable between all frameworks, a key feature. .env is an environmental variable directory meant to be loaded with the envdir tool. It represents system-wide settings, like PATH, and some of the settings that are global to all of the definitions. It is used within the templates. .finish will hold ./finish scripts. Right now, it's pretty much a stub. Eventually it will hold a basic finish script that alerts the administrator to issues with definitions not launching, as well as handling other non-standard terminations. .log will hold ./log scripts. It currently has a single symlink, ./run, that points to whatever logging system is the default. At the moment it's svlogd only because I haven't finished logging for s6 and daemontools. Eventually .log/run will be a symlink to whatever loggin arrangement you need. In this fashion, the entire set of scripts can be switched by simply switching the one symlink. .run will hold the ./run scripts. It has a few different ones in them, but the main one at this time is run-envdir, which loads daemon specific settings from the definition's env directory and uses them to launch the daemon. Others include an optional feature for user-defined services, and basic support for one of three getty. I may or may not make a new one for the optional dependency feature; I'm going to see if it can be standardized within run-envdir first. I can always remove the dots, but then you would have these mixed in with all of the definitions, and I think it will add to the confusion more than having them hidden. As it stands, the only time you need to mess with the dot directories is (a) when setting them up for the first time, or (b) when you are switching your logging around. Otherwise there's really no need to be in them, and when you use ls /etc/sv to see what is available, they stay out of your way. If there is a better arrangement that keeps everything in one base directory for easy management but eliminates the dots, I'll listen. Although I think this arrangement actually makes a bit more sense, and the install instructions are careful to include the dots, so you only need to mess around with them at install time. On Thu, Jan 8, 2015 at 8:20 AM, Luke Diamand l...@diamand.org wrote: Is it possible to avoid using hidden files (.env) as it makes it quite a lot harder for people who don't know what's going on to, um, work out what's going on. Thanks! Luke
Re: thoughts on rudimentary dependency handling
On Thu, Jan 8, 2015 at 9:23 AM, Steve Litt sl...@troubleshooters.com wrote: I'm having trouble understanding exactly what you're saying. You mean the executable being daemonized fails, by itself, because a service it needs isn't there, right? You *don't* mean that the init itself fails, right? Both correct. I'm not sure what you're saying. Are you saying that the dependency code is in the runscript, but within an IF statement that checks for ../env/NEEDS_ENABLED? Correct. If the switch, which is a data value in a file, is zero, it simply skips all of the dependency stuff with a giant if-then wrapper. At least, that's the plan. I won't know until I can get to it. Like I said, this will be a fall-back feature, and it will have minor annoyances or issues. Yes. If I'm understanding you correctly, you're only going so far in determinint really up, because otherwise writing a one size fits all services thing starts getting way too complicated. Correct. I'm taking an approach that has the minimum needed to make things work correctly. I was looking at runit docs yesterday before my Init System presentation, and learned that I'm supposed to put my own Really Up code in a script called ./check. Also correct, although I'm trying to only do ./check scripts where absolutely needed, such as the ypbind situation. Otherwise, the check usually looks at is the child PID still around. If I read the preceding correctly, you're making service tool calls for runit, s6, perp and nosh grammatically identical. Correct. Are you doing that so that your run scripts can invoke the init-agnostic commands, so you just have one version of your scripts? Exactly correct. This how I am able to turn the bulk of the definitions into templates. ./run files in the definition directories are little more than symlinks back to a script in ../.run, which means...write once, use a whole lot. :) It's also the reason that features are slow in coming - I have to be very, very careful about interactions. However you end up doing the preceding, I think it's essential to thoroughly document it, complete with examples. I think that the additional layer of indirection might be skipped by those not fully aware of the purpose. I just haven't gotten around to this part, sorry. I can help with the documentation. https://bitbucket.org/avery_payne/supervision-scripts or https://github.com/apayne/supervision-scripts Feel free to clone, change, and send a pull request.
Re: thoughts on rudimentary dependency handling
On 07/01/15 07:05, James Powell wrote: The way I see it is this... Either you have high level dependency handling within the service supervision system itself, or you have low level dependency handling within the service execution files. Keeping the system as simplified as possible lowers the probability and possibility of issues. That's one of the basic rules of UNIX programming. I, personally, don't see a need other than low level handling. Systemd/uselessd even does this within the unit files, as does some instances of sysvinit/bsdinit by using numbered symlinks for execution order or using a master control script. systemd also has 'wants' directories which add dependencies to a unit in much the way that's being suggested here. sysvinit obviously has all the LSB headers in the init.d files to try to express dependencies. Avery - your suggestion sounds like it ought to be simple to implement and reasonably easy to understand for a sys-admin. Perhaps the way to go is to suck it and see - it'll perhaps then be more obvious with a real implementation if it's worth pursuing. One small concern would be that it's not enough to simply signal a dependency to be up - it needs to be actually running and working before you can start the dependent service successfully. To take my [least-]favourite example, you can't start autofs until ypbind has actually contacted the NIS server. Sent from my Windows Phone From: Avery Paynemailto:avery.p.pa...@gmail.com Sent: 1/6/2015 10:35 PM To: Steve Littmailto:sl...@troubleshooters.com Cc: supervision@list.skarnet.orgmailto:supervision@list.skarnet.org Subject: Re: thoughts on rudimentary dependency handling On Jan 6, 2015, at 4:56 PM, Steve Litt sl...@troubleshooters.com wrote: On Tue, 6 Jan 2015 13:17:39 -0800 Avery Payne avery.p.pa...@gmail.com wrote: On Tue, Jan 6, 2015 at 10:20 AM, Laurent Bercot ska-supervis...@skarnet.org wrote: I firmly believe that a tool, no matter what it is, should do what the user wants, even if it's wrong or can't possibly work. If you cannot do what the user wants, don't try to be smart; yell at the user, spam the logs if necessary, and fail. But don't do anything the user has not explicitly told you to do. And there's the rub. I'm at a crossroad with regard to this because: 1. The user wants service A to run. 2. Service A needs B (and possibly C) running, or it will fail. Should the service fail because of B and C, even though the user wants A up, or Should the service start B and C because the user requested A be running? I thought the way to do the latter was like this: http://smarden.org/runit/faq.html#depends If every upstream simply declared that his program needs B and C running before his program runs, it's easy to translate that into an sv start command in the run script. Normally, with hand written scripts, that would be the case. You cobble together what is needed and go on your way. But this time things are a little different. The project I'm doing uses templates - pre-written scripts that turn the various launch issues into variables while using the same code. This reduces development time and bugs - write the template once, debug it once, and reuse it over and over. The idea for dependencies is that I could write something that looks at symlinks in a directory and if the template finds anything, it starts the dependency. Otherwise it remains blissfully unaware. The issue that Laurent is arguing for is that by creating such a framework - even one as thin and as carefully planned - it shuts out future possibilities that would handle this correctly at a very high level, without the need to write or maintain this layout. To accommodate this possibility and to maximize compatibility, he is arguing to stay with the same service is unaware of its surroundings that have been a part of the design of all the frameworks - let things fail and make the admin fix it. There is merit to this, not just because of future expansions, but also because it allows the end user (read: SysAdmin) the choice of running things. Or long story short, don't set policy, let the end user decide. And I agree; I'm a bit old school about these things and I picked up Linux over a decade ago because I wanted choices. But from a practical perspective there isn't anything right now that handles dependencies at a global level. The approach of minimum knowledge needed and best effort separation of duty would give a minimal environment for the time being. The design is very decentralized and works at a peer level; no service definition knows anything other than what is linked in its ./needs directory, and side effects are minimized by trying hard to keep separation of duty. Because the scripts are distributed and meant to be installed as a whole, I can kinda get away with this because all of the dependencies are hard coded out of the box
Re: thoughts on rudimentary dependency handling
The problem is more to do with finding a nice way to structure things without it turning into a maze of cobbled-together bits of shell script for doing random hacks to get things to come up in the right order. On 7 January 2015 at 10:59, James Powell james4...@hotmail.com wrote: You probably could have it initial a ping or similar callback to the NIS with an if/else switch as well for the checks. Target a file, directory, IP signal, anything really and if found, start the service, else it exits and gets retried. Sent from my Windows Phone From: Luke Diamandmailto:l...@diamand.org Sent: 1/7/2015 12:17 AM To: supervision@list.skarnet.orgmailto:supervision@list.skarnet.org Subject: Re: thoughts on rudimentary dependency handling On 07/01/15 07:05, James Powell wrote: The way I see it is this... Either you have high level dependency handling within the service supervision system itself, or you have low level dependency handling within the service execution files. Keeping the system as simplified as possible lowers the probability and possibility of issues. That's one of the basic rules of UNIX programming. I, personally, don't see a need other than low level handling. Systemd/uselessd even does this within the unit files, as does some instances of sysvinit/bsdinit by using numbered symlinks for execution order or using a master control script. systemd also has 'wants' directories which add dependencies to a unit in much the way that's being suggested here. sysvinit obviously has all the LSB headers in the init.d files to try to express dependencies. Avery - your suggestion sounds like it ought to be simple to implement and reasonably easy to understand for a sys-admin. Perhaps the way to go is to suck it and see - it'll perhaps then be more obvious with a real implementation if it's worth pursuing. One small concern would be that it's not enough to simply signal a dependency to be up - it needs to be actually running and working before you can start the dependent service successfully. To take my [least-]favourite example, you can't start autofs until ypbind has actually contacted the NIS server. Sent from my Windows Phone From: Avery Paynemailto:avery.p.pa...@gmail.com Sent: 1/6/2015 10:35 PM To: Steve Littmailto:sl...@troubleshooters.com Cc: supervision@list.skarnet.orgmailto:supervision@list.skarnet.org Subject: Re: thoughts on rudimentary dependency handling On Jan 6, 2015, at 4:56 PM, Steve Litt sl...@troubleshooters.com wrote: On Tue, 6 Jan 2015 13:17:39 -0800 Avery Payne avery.p.pa...@gmail.com wrote: On Tue, Jan 6, 2015 at 10:20 AM, Laurent Bercot ska-supervis...@skarnet.org wrote: I firmly believe that a tool, no matter what it is, should do what the user wants, even if it's wrong or can't possibly work. If you cannot do what the user wants, don't try to be smart; yell at the user, spam the logs if necessary, and fail. But don't do anything the user has not explicitly told you to do. And there's the rub. I'm at a crossroad with regard to this because: 1. The user wants service A to run. 2. Service A needs B (and possibly C) running, or it will fail. Should the service fail because of B and C, even though the user wants A up, or Should the service start B and C because the user requested A be running? I thought the way to do the latter was like this: http://smarden.org/runit/faq.html#depends If every upstream simply declared that his program needs B and C running before his program runs, it's easy to translate that into an sv start command in the run script. Normally, with hand written scripts, that would be the case. You cobble together what is needed and go on your way. But this time things are a little different. The project I'm doing uses templates - pre-written scripts that turn the various launch issues into variables while using the same code. This reduces development time and bugs - write the template once, debug it once, and reuse it over and over. The idea for dependencies is that I could write something that looks at symlinks in a directory and if the template finds anything, it starts the dependency. Otherwise it remains blissfully unaware. The issue that Laurent is arguing for is that by creating such a framework - even one as thin and as carefully planned - it shuts out future possibilities that would handle this correctly at a very high level, without the need to write or maintain this layout. To accommodate this possibility and to maximize compatibility, he is arguing to stay with the same service is unaware of its surroundings that have been a part of the design of all the frameworks - let things fail and make the admin fix it. There is merit to this, not just because of future expansions, but also because it allows the end user
Re: thoughts on rudimentary dependency handling
On 07/01/2015 09:16, Luke Diamand wrote: One small concern would be that it's not enough to simply signal a dependency to be up - it needs to be actually running and working before you can start the dependent service successfully. To take my [least-]favourite example, you can't start autofs until ypbind has actually contacted the NIS server. Readiness notification is hard: it needs support from the daemon itself, and most daemons are not written that way. Of course, systemd jumped on the opportunity to encourage daemon authors to use systemd-specific interfaces to provide readiness notification. Embrace and extend. Fortunately, it's not the only way. There's a very simple way for a daemon to notify the outside world of its readiness: write something to a descriptor. See http://skarnet.org/software/s6/notifywhenup.html -- Laurent
Re: thoughts on rudimentary dependency handling
On Wed, Jan 7, 2015 at 6:53 PM, Laurent Bercot ska-supervis...@skarnet.org
wrote:
Unfortunately, the envdir tool, which I use to abstract away the daemons
and settings, only chain-loads; it would be nice if it had a persistence
mechanism, so that I could load once for the scope of the shell script.
Here's an ugly hack that allows you do that using envdir:
set -a
eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort |
uniq -u)
set +a
Thanks! When I can carve out a bit of time this week I'll put it in and
finish up the few bits needed. Most of the dependency loop is already
written, I just didn't have a somewhat clean way of pulling in the
$CMDWHATEVER settings without repeatedly reading ./env over and over.
It only works for variables you add, though, not for variables you remove.
It will work fine. I'm attempting to pre-load values that will remain
constant inside the scope of the script, so there isn't a need to change
them at runtime.
RE: thoughts on rudimentary dependency handling
On Wed, Jan 7, 2015 at 7:23 AM, Steve Litt sl...@troubleshooters.com wrote: I'm pretty sure this conforms to James' preference (and mine probably) that it be done in the config and not in the init program. To satisfy Laurent's preference, everything but the exec cron -f could be commented out, and if the user wants to use this, he/she can uncomment all the rest. Or your run script writing program could have an option to write the dependencies, or not. I've pretty much settled on a system-wide switch in sv/.env (which in the scripts will show up as ../.env). The switch will, by default, follow Laruent's behavior of naive launching, ie. no dependencies are up, missing dependencies cause failures, and the admin must check logging for notifications. Enabling the feature would be as simple as echo 1 /etc/sv/.env/NEEDS_ENABLED ...and every new service launch would receive it. You could also force-reload with a restart command. Without the flag, the entire chunk of dependency code is bypassed and the launch continues as normal. The goal is the same but the emphasis has changed. This will be considered a fall-back feature for those systems that do not have such a tool available, or have constraints that force the continued use of a shell launcher. It is the option of last resort, and while I think I can make it work fairly consistently, it will come with some warnings in the wiki. For Laurent, he wouldn't even need to lift a finger - it fully complies with his desires out of the box. ;-) As new tools emerge in the future, I will be able to write a shunt into the script that detects the tool and uses it instead of the built-in scripted support. This will allow Laurent's work to be integrated without messing anything up, so the behavior will be the same, but implemented differently. Finally, with regard to the up vs actually running issue, I'm not even going to try and address it due to the race conditions involved. The best I will manage is to first issue the up, then do a service check to confirm that it didn't die upon launch, which for a majority (but not all) cases should suffice. Yes, there are still race conditions, but that is fine - I'm falling back to the original model of service fails continually until it succeeds, which means a silently-failed child dependency that was missed by the check command will still cause the parent script to fail, because the daemon itself will fail. It is a crude form of graceful failure. So the supervisor starts the parent again...and again...until the truant dependency is up and running, at which point it will bring the parent up. Like I said, this will be a fall-back feature, and it will have minor annoyances or issues. Right now the biggest problem is handling all of the service tool calls. They all have the same grammar, (tool) (command) (service name), so I can script that easily. Getting the tools to show up as the correct command and command option is something else, and I'm working on a way to wedge it into the use-* scripts so that the tools are set up out of the box all at the same time. This will create $SVCTOOL, and a set of $CMDDOWN, $CMDUP, $CMDCHECK, etc. that will be used in the scripts. **Once that is done I can fully test the rest of the dependency concept and get it fleshed out.** If anyone wants to see it, email me directly and I'll pass it along, but there's not much to look at. Unfortunately, the envdir tool, which I use to abstract away the daemons and settings, only chain-loads; it would be nice if it had a persistence mechanism, so that I could load once for the scope of the shell script. Because of that, there will be some odd scripting in there that pulls the values, i.e. [ -f ../.env/CMDUP ] || echo $(basename $0): fatal error: unable to load CMDUP exit 99 CMDUP=$(cat ../.env/CMDUP) with an entry for each command. In my 5 minute thought process, the last remaining challenge, and it's a big one, is to get the right service names for the dependencies, and that requires a standardized list, because, as far as I know, the daemontools-inspired inits don't have provides. Such a list would be hard enough to develop and have accepted, but tinfoil_hatI expect our friends at Red Hat to start changing the names in order to mess us up/tinfoil_hat. Using a ./provides as a rendezvous or advertisement mechanism I think is nice-in-concept but difficult-in-practice. Give it a bit more thought and you'll see that we're not just talking about the *service* but also any *protocol* to speak with it and one or more *data transport* needed to talk to the service. Example: MySQL using a port number bound to 127.0.0.1, vs MySQL using a file socket. Both provide a MySQL database and MySQL's binary client protocol, but the transport is entirely different. Another example: exim4 vs postfix vs qmail vs (insert favorite SMTP server here). All speak SMTP - but some do LMTP at the same time (in either sockets or ports), so
Re: thoughts on rudimentary dependency handling
On 07/01/2015 23:25, Avery Payne wrote:
The goal is the same but the emphasis has changed. This will be considered
a fall-back feature for those systems that do not have such a tool
available, or have constraints that force the continued use of a shell
launcher. It is the option of last resort, and while I think I can make it
work fairly consistently, it will come with some warnings in the wiki. For
Laurent, he wouldn't even need to lift a finger - it fully complies with
his desires out of the box. ;-)
Eh, you don't have to. It's not for me, it's for the users. I honestly
believe it's better this way, but you may disagree, and as long as you're
the one writing the code, you're the one having the say.
Unfortunately, the envdir tool, which I use to abstract away the daemons
and settings, only chain-loads; it would be nice if it had a persistence
mechanism, so that I could load once for the scope of the shell script.
Here's an ugly hack that allows you do that using envdir:
set -a
eval $({ env; envdir ../.env env; } | grep -vF -e _= -e SHLVL= | sort | uniq -u)
set +a
It only works for variables you add, though, not for variables you remove.
Also, be aware that it considerably lengthens the code path... but not
significantly more than dependency management in shell already does. :P
Sshh! Don't talk too loudly...you'll expose my secret plan! :-) But
thanks for saying that, and you are right. The drive of the project is to
increase accessibility of the frameworks, and in doing so, increase
adoption. Adoption will lead to feedback, which leads to improvements,
which drives development forward, increasing accessibility, etc. in a
continuous feedback loop.
This is a very laudable goal, if you think the way to getting into distros
is this kind of accessibility.
I'm of the opinion that packagers will naturally go towards what gives
them the less work, and the reason why supervision frameworks have trouble
getting in is that they require different scripting and organization, so
supporting them would give packagers a lot of work; whereas sticking to
the SysV way allows them not to change what they already have.
Especially with systemd around, which they already kinda have to convert
services to, I don't see them bothering with another out of the way
packaging design.
So, to me, the really important work that you are doing is the run script
collection and standardization. If you provide packagers with already made
run scripts, you are helping them tremendously by reducing the amount of
work that supervision support needs, and they'll be more likely to adopt it.
It's a thankless job, and I understand that you want to have some fun by
playing with a dependency manager ;)
--
Laurent
Re: thoughts on rudimentary dependency handling
Down by default makes sense to me and will be a great feature. I think that having it will require all services to have a 'down' script that defines how to make sure that a service is actually down. I wonder if this will help address a common situation for me where I install a package and realize that at the end of the installation the daemon is started using upstart or sysv. At that point, to 'supervise' the app, I first have to stop the current daemon and then start it up using runit or another process manager. Otherwise I end up with two copies of the app running, with only one of them being supervised. John Albietz m: 516-592-2372 e: inthecloud...@gmail.com l'in: www.linkedin.com/in/inthecloud247 On Tue, Jan 6, 2015 at 10:20 AM, Laurent Bercot ska-supervis...@skarnet.org wrote: On 06/01/2015 18:46, Avery Payne wrote: 1. A service can ask another service to start. 2. A service can only signal itself to go down. It can never ask another service to go down. 3. A service can only mark itself with a ./down file. It can never mark another service with a ./down file. That's it. Numbers 2 and 3 are the only times I would go against what you are saying. So does number 1. When you ask another service to start, you change the global state. If it is done automatically by any tool or service, this means the global state changes without the admin having requested it. This is trying to be smarter than the user, which is almost always a bad thing. Number 2 is in the same boat. If the admin wants you to be up, then you can't just quit and decide that you will be down. You can't change the global state behind the admin's back. Number 3 is different. It doesn't change the global state - unless there's a serious incident. But it breaks resiliency against that incident: it kills the guarantee the supervision tree offers you. I firmly believe that a tool, no matter what it is, should do what the user wants, even if it's wrong or can't possibly work. If you cannot do what the user wants, don't try to be smart; yell at the user, spam the logs if necessary, and fail. But don't do anything the user has not explicitly told you to do. Maybe a dependency manager needs to be smarter than that. In which case I would call it a global state manager. There would be the current state, which starts at everything down when the machine boots, and the wanted state, which starts at everything up; as long as those states are not matched, the global state manager is running, implementing retry policies and such, and may change the global state at any time, bringing individual services up and down as it sees fit, with the ultimate goal of matching the global current state with the global wanted state. It could do stuff like exponential backoff so failing services would not be wanted up all the time; but it would never, ever change the global wanted state without an order to do so from the admin. If you want a dependency manager with online properties, I think this is the way to do it. -- Laurent
Re: thoughts on rudimentary dependency handling
On Tue, Jan 6, 2015 at 10:20 AM, Laurent Bercot ska-supervis...@skarnet.org wrote: I firmly believe that a tool, no matter what it is, should do what the user wants, even if it's wrong or can't possibly work. If you cannot do what the user wants, don't try to be smart; yell at the user, spam the logs if necessary, and fail. But don't do anything the user has not explicitly told you to do. And there's the rub. I'm at a crossroad with regard to this because: 1. The user wants service A to run. 2. Service A needs B (and possibly C) running, or it will fail. Should the service fail because of B and C, even though the user wants A up, or Should the service start B and C because the user requested A be running? For some, the first choice, which is to immediately fail, is perfectly fine. I can agree to that, and I understand the why of it, and it makes sense. But in other use cases, you'll have users that aren't looking at this chain of details. They asked for A to be up, why do they need to bring up B, oh look there's C too...things suddenly look broken, even though they aren't. I'm caught between making sure the script comes up, and doing the right thing consistently. I can certainly make the scripts naive of each other and not start anything at all...and leave everything up to the administrator to figure out how to get things working. Currently this is how the majority of them are done, and it wouldn't take much to change the rest to match this behavior. It's also occurred to me that instead of making the dependency feature a requirement, I can make it optional. It could be a feature that you choose to activate by setting a file or environment variable. Without the setting, you would get the default behavior you are wanting to see, with no dependency support; this would be the default out of the box experience. With the setting, you get the automatic start-up that I think people will want. So the choice is back with the user, and they can decide. That actually might be the way to handle this, and both parties - the ones that want full control and visibility, and the ones after ease of use - will get what they want. On the one hand I can assure that you will get working scripts, because scripts that have dependencies can be made to work that way. On the other hand, if you want strict behavior, that is assured as well. The only drawback is you can't get both because of the limitations of the environment that I am in.
Re: thoughts on rudimentary dependency handling
On 06/01/2015 18:46, Avery Payne wrote: 1. A service can ask another service to start. 2. A service can only signal itself to go down. It can never ask another service to go down. 3. A service can only mark itself with a ./down file. It can never mark another service with a ./down file. That's it. Numbers 2 and 3 are the only times I would go against what you are saying. So does number 1. When you ask another service to start, you change the global state. If it is done automatically by any tool or service, this means the global state changes without the admin having requested it. This is trying to be smarter than the user, which is almost always a bad thing. Number 2 is in the same boat. If the admin wants you to be up, then you can't just quit and decide that you will be down. You can't change the global state behind the admin's back. Number 3 is different. It doesn't change the global state - unless there's a serious incident. But it breaks resiliency against that incident: it kills the guarantee the supervision tree offers you. I firmly believe that a tool, no matter what it is, should do what the user wants, even if it's wrong or can't possibly work. If you cannot do what the user wants, don't try to be smart; yell at the user, spam the logs if necessary, and fail. But don't do anything the user has not explicitly told you to do. Maybe a dependency manager needs to be smarter than that. In which case I would call it a global state manager. There would be the current state, which starts at everything down when the machine boots, and the wanted state, which starts at everything up; as long as those states are not matched, the global state manager is running, implementing retry policies and such, and may change the global state at any time, bringing individual services up and down as it sees fit, with the ultimate goal of matching the global current state with the global wanted state. It could do stuff like exponential backoff so failing services would not be wanted up all the time; but it would never, ever change the global wanted state without an order to do so from the admin. If you want a dependency manager with online properties, I think this is the way to do it. -- Laurent
Re: thoughts on rudimentary dependency handling
I'm not sure exactly in what context your message needs to be taken - is that about a tool you have written or are writing, or something else ? - but if you're going to work on dependency management, it's important that you get it right. It's complex stuff that needs planning and thought. * implement a ./needs directory. This would have symlinks to any definitions that would be required to run before the main definition can run. For instance, Debian's version of lightdm requires that dbus be running, or it will abort. Should a ./needs not be met, the current definition will receive a ./down file, write out a message indicating what service blocked it from starting, and then will send a down service to itself. For instance, I'm convinced that the approach you're taking here actually takes away from reliability. Down files are dangerous: they break the supervision chain guarantee. If the supervisor dies and is respawned by its parent, it *will not* restart the service if there's a down file. You want down files to be very temporary, for debugging or something, you don't want them to be a part of your normal operation. I firmly believe that in order to keep boot and shutdown procedures fast and simple, and avoid reinventing the kitchen sink, any dependency management on top of a supervision system should work *offline*. Keep the dependency manager out of the supervisor's way in normal operation; just use it to generate state change scripts. If your dependency manager works online, you *will* bring services down when you don't want to. You *will* have more headaches making things work than if you had no dependency manager at all. I guarantee it. I don't know how to design such a beast. I'm not there yet, I haven't given it any thought. But a general principle applies: don't do more, do less. If something is unnecessary, don't do it. What a supervision framework needs is a partial order on how to bring services up or down at boot time and shutdown time, and other global state changes; not instructions on what to do in normal operation. Stay out of the way of the supervisor outside of a global state change. -- Laurent
Re: thoughts on rudimentary dependency handling
On Tue, Jan 6, 2015 at 8:52 AM, Laurent Bercot ska-supervis...@skarnet.org wrote: I'm not sure exactly in what context your message needs to be taken - is that about a tool you have written or are writing, or something else ? - but if you're going to work on dependency management, it's important that you get it right. It's complex stuff that needs planning and thought. This is in the context of service definition A needs service definition B to be up. * implement a ./needs directory. This would have symlinks to any definitions that would be required to run before the main definition can run. For instance, Debian's version of lightdm requires that dbus be running, or it will abort. Should a ./needs not be met, the current definition will receive a ./down file, write out a message indicating what service blocked it from starting, and then will send a down service to itself. For instance, I'm convinced that the approach you're taking here actually takes away from reliability. Down files are dangerous: they break the supervision chain guarantee. If the supervisor dies and is respawned by its parent, it *will not* restart the service if there's a down file. You want down files to be very temporary, for debugging or something, you don't want them to be a part of your normal operation. If your dependency manager works online, you *will* bring services down when you don't want to. You *will* have more headaches making things work than if you had no dependency manager at all. I guarantee it. I should have added some clarifications. There are some basic rules I'm using with regard to starting/stopping services: 1. A service can ask another service to start. 2. A service can only signal itself to go down. It can never ask another service to go down. 3. A service can only mark itself with a ./down file. It can never mark another service with a ./down file. That's it. Numbers 2 and 3 are the only times I would go against what you are saying. And since the only reason I would do that is because of some failure that was unexpected, there would be a good reason to do so. And in all cases, there would be a message output as to why it signaled itself back down, or why it marked itself with a ./down file. The ./down file is, I think, being used correctly - I'm trying to flag to the sysadmin that something is wrong with this service, and it shouldn't restart until you fix it. I'm sorry if the posting was confusing. Hopefully the rules clarify when and how I would be using these features. I believe it should be safe if they are confined within the context of the service definition itself, and not other dependencies. If there is something to the contrary that I'm missing in those three rules, I'm listening.
