Re: staggering runsv startup
Off the top of my head an easy solution in perp that requires no
special or supplemental scripting, flagfile tricks, etc.
For multiple service instances of /usr/bin/myserv -- named myserv00,
myserv01, myserv01, ..., myservNN -- deploy the following set of
service definitions.
First, basic myserv00 runscript (in /etc/perp/myserv00/rc.main):
#!/bin/sh
exec 21
TARGET=$1
SVNAME=$2
start() {
echo starting ${SVNAME}...
exec /usr/bin/myserv
}
reset() {
echo resetting ${SVNAME}...
exit 0
}
eval ${TARGET} $@
### eof
Next myserv01 runscript (in /etc/perp/myserv01/rc.main), showing just
the start() stanza for brevity:
...
start() {
if ! perpok -u3 myserv00 ; then
echo ${SVNAME}: not yet running myserv00
exit 1
fi
echo starting ${SVNAME}...
exec /usr/bin/myserv
}
...
Next myserv02 runscript (in /etc/perp/myserv02/rc.main), again the
start() stanza:
...
start() {
if ! perpok -u3 myserv01 ; then
echo ${SVNAME}: not yet running myserv01
exit 1
fi
echo starting ${SVNAME}...
exec /usr/bin/myserv
}
...
And so forth, each instance of myservXX using perpok(8) to check if the
previous instance is up and running before loading current instance.
Many permutations of this basic theme are possible and can be fine tuned
to match your local installation, eg:
* Subsequent instances of myserv may all use myserv00 as the base
instance, and modify/increment -u secs parameter to perpok(8)
accordingly.
* Multiple instances of myservNN may be grouped that use the same
previous instance with the same -u secs parameter to perpok(8), to
permit asynchronous startup of a few at a time, rather than one at a
time as shown above.
See also runtools such as runargs(8) that may allow you to setup your
runscripts with a basic template, loading the perpok(8) parameters from
an external file.
Wayne
http://b0llix.net/perp/
On Thu, 04 Jun 2015 13:41:12 -0700
Jameson Graef Rollins jroll...@finestructure.net wrote:
Hi, all. I am using runit to supervise a large set of nearly
identical processes. Each process accesses certain IO-bound shared
resources (e.g. NFS mount) at startup. At system initialization,
when runsvdir is launched, it launches all these processes (via
runsv) essentially simultaneously. This causes a big resource
contention at initialization that occasionally causes problems.
What I would like is to somehow stagger the startup of the processes,
to avoid the resource contention. I could do this by putting a random
sleep into the ./run scripts, but this would also cause random startup
delays on subsequent process restarts via sv restart or the like
(which we occasionally need to do).
What I would prefer instead is to add random delays to the startup of
the *runsv* processes, since this would only apply at system
initialization. Unfortunately I can't see any way to do that right
now (other than somehow wrapping the runsv binary itself).
Does anyone know any way to accomplish what I'm looking for? I don't
believe runsvdir supports any options that would apply here. Is it
possible to somehow point runsvdir to a alternate runsv executable to
which I could add the random delays?
Any suggestions would be much appreciated. Thanks.
jamie.
RE: staggering runsv startup
If runit had the ability to order processes like OpenRC where you have: before= after= setups, you could order the entire tree structure. The problem with sv check is the command often can only check the status of the service. Sent from my Windows Phone From: Steve Littmailto:sl...@troubleshooters.com Sent: 6/4/2015 3:33 PM To: supervision@list.skarnet.orgmailto:supervision@list.skarnet.org Subject: Re: staggering runsv startup On Fri, 05 Jun 2015 00:10:05 +0200 Laurent Bercot ska-supervis...@skarnet.org wrote: What you really want is a real service manager that works on top of a process supervision system and that would managed a complete, ordered initialization sequence for you. Steve is saying that process supervisors are lacking real service management capabilities, and he's right. Process supervision does not offer service management; service management is more complex and one layer above. I'm not familiar with the definitions of service managers and process supervisors, but the simple ability to declare the order of initial, bootup startup, would go a long way. I'm not saying such ordering would free me from needing to check for required other services being ready in this service's run script. I'm just saying that ordering would eliminate the tendency of such checks to result in one service coming up per cycle around /service, and would produce reasonable boot times, because in most cases teh required services *would* be up and functioning by the time the service that needs them is started. There are some tools to accomplish service management on top of process supervision. One that I like is anopa: http://jjacky.com/anopa/ but it's designed to work with s6, not runit. I'm also working on a service management system for s6, that should hit beta soon. This is very good news. Ordering is sorely needed in the daemontools world, and I think your new S6 service management system would be the first to do that without kludges like I enumerated in other emails. Thanks, SteveT Steve Litt June 2015 featured book: The Key to Everyday Excellence http://www.troubleshooters.com/key
Re: staggering runsv startup
On Fri, 05 Jun 2015 00:10:05 +0200 Laurent Bercot ska-supervis...@skarnet.org wrote: What you really want is a real service manager that works on top of a process supervision system and that would managed a complete, ordered initialization sequence for you. Steve is saying that process supervisors are lacking real service management capabilities, and he's right. Process supervision does not offer service management; service management is more complex and one layer above. I'm not familiar with the definitions of service managers and process supervisors, but the simple ability to declare the order of initial, bootup startup, would go a long way. I'm not saying such ordering would free me from needing to check for required other services being ready in this service's run script. I'm just saying that ordering would eliminate the tendency of such checks to result in one service coming up per cycle around /service, and would produce reasonable boot times, because in most cases teh required services *would* be up and functioning by the time the service that needs them is started. There are some tools to accomplish service management on top of process supervision. One that I like is anopa: http://jjacky.com/anopa/ but it's designed to work with s6, not runit. I'm also working on a service management system for s6, that should hit beta soon. This is very good news. Ordering is sorely needed in the daemontools world, and I think your new S6 service management system would be the first to do that without kludges like I enumerated in other emails. Thanks, SteveT Steve Litt June 2015 featured book: The Key to Everyday Excellence http://www.troubleshooters.com/key
Re: staggering runsv startup
On 04/06/2015 22:41, Jameson Graef Rollins wrote: What I would like is to somehow stagger the startup of the processes, to avoid the resource contention. I could do this by putting a random sleep into the ./run scripts, but this would also cause random startup delays on subsequent process restarts via sv restart or the like (which we occasionally need to do). You could also have an initial ./run that would sleep a random amount of time, then atomically replace itself with the normal run script before executing itself. Of course, it requires read-write service directories. This is one of the reasons why I generally advocate keeping the working copy of service directories in a tmpfs. -- Laurent
Re: staggering runsv startup
On Thu, 04 Jun 2015 13:41:12 -0700 Jameson Graef Rollins jroll...@finestructure.net wrote: Hi, all. I am using runit to supervise a large set of nearly identical processes. Each process accesses certain IO-bound shared resources (e.g. NFS mount) at startup. At system initialization, when runsvdir is launched, it launches all these processes (via runsv) essentially simultaneously. This causes a big resource contention at initialization that occasionally causes problems. What I would like is to somehow stagger the startup of the processes, to avoid the resource contention. I could do this by putting a random sleep into the ./run scripts, but this would also cause random startup delays on subsequent process restarts via sv restart or the like (which we occasionally need to do). What I would prefer instead is to add random delays to the startup of the *runsv* processes, since this would only apply at system initialization. Unfortunately I can't see any way to do that right now (other than somehow wrapping the runsv binary itself). Does anyone know any way to accomplish what I'm looking for? I don't believe runsvdir supports any options that would apply here. Is it possible to somehow point runsvdir to a alternate runsv executable to which I could add the random delays? Any suggestions would be much appreciated. Thanks. jamie. In the 1 script, put down files in every directory under /service except one called order. Then, the order process is called, and it erases the down file from one at a time, sleeping 1 second after each. When all of them have been undowned, have order put a down file in its own directory and stop itself. The one thing I dislike about daemontools type process managers is you can't order the processes. SteveT Steve Litt June 2015 featured book: The Key to Everyday Excellence http://www.troubleshooters.com/key
Re: staggering runsv startup
On Thu, Jun 04 2015, Steve Litt sl...@troubleshooters.com wrote: In the 1 script, put down files in every directory under /service except one called order. Then, the order process is called, and it erases the down file from one at a time, sleeping 1 second after each. When all of them have been undowned, have order put a down file in its own directory and stop itself. Hi, Steve. This is a pretty elegant way to implement Lucy's suggestion. This order service could potentially be used to manage the startup list as well. Thanks to both you and Lucy for the suggestion. jamie. signature.asc Description: PGP signature
Re: staggering runsv startup
On Thu, 04 Jun 2015 14:44:38 -0700 Jameson Graef Rollins jroll...@finestructure.net wrote: On Thu, Jun 04 2015, Lucy Pseudonym ne.tet...@gmail.com wrote: You can create `down` files in the service dirs as described in [1] and enable the services from a script at boot time. Hi, Lucy. That's an interesting suggestion. It would require building out more/other infrastructure, though, which is something I was trying to avoid. I'm also already using the `down` files to control which services are started at boot. If I did what you suggest it would require managing the startup list in some other way. Thanks for the suggestion, though. It is an idea I hadn't considered. jamie. put reallydown files in services you truly intend to be down, and have the script that deletes them not delete them if there's a reallydown file. Obviously this is kludge city, but I think what you really need is ordering of the managed services, and as far as I know, no daemontools inspired program does that in a straightforward manner. SteveT Steve Litt June 2015 featured book: The Key to Everyday Excellence http://www.troubleshooters.com/key
Re: staggering runsv startup
On Thu, Jun 04 2015, Lucy Pseudonym ne.tet...@gmail.com wrote: You can create `down` files in the service dirs as described in [1] and enable the services from a script at boot time. Hi, Lucy. That's an interesting suggestion. It would require building out more/other infrastructure, though, which is something I was trying to avoid. I'm also already using the `down` files to control which services are started at boot. If I did what you suggest it would require managing the startup list in some other way. Thanks for the suggestion, though. It is an idea I hadn't considered. jamie. signature.asc Description: PGP signature
Re: staggering runsv startup
What you really want is a real service manager that works on top of a process supervision system and that would managed a complete, ordered initialization sequence for you. Steve is saying that process supervisors are lacking real service management capabilities, and he's right. Process supervision does not offer service management; service management is more complex and one layer above. There are some tools to accomplish service management on top of process supervision. One that I like is anopa: http://jjacky.com/anopa/ but it's designed to work with s6, not runit. I'm also working on a service management system for s6, that should hit beta soon. -- Laurent
