Re: s6-rc design ; comparison with anopa
On Sat, Apr 25, 2015 at 12:36:24PM +0200, Laurent Bercot wrote:
> On 25/04/2015 09:35, Colin Booth wrote:
> >I've been having a hard time thinking about bundles the right way. At
> >first they seemed like first-class services along with longruns and
> >oneshots, but it sounds more like they are more of a shorthand to
> >reference a collection of atomic services than a service in their own
> >right. Especially since bundles can't depend on anything it clarifies
> >things greatly to think of them as a shorthand for a collection of
> >atomic services instead of a service itself.
>
> Yes, that's exactly what a bundle is: a shorthand for a collection of
> atomic services. Sorry if I didn't make it clear enough.
>
Sounds good. I was having problems reconciling a bundle as a full
service and the ability to bring down bundle A and have it change bundle
B if there was overlap in member services. In more concrete terms, if we
have bundle web {httpd, php, sql} and bundle java {tomcat, sql} and we
ask to shut down java, if bundles were first-class services than I'd
expect to have s6-rc bring down tomcat and keep sql running since it's a
component of another bundle that still needs to be up.
This actually brings up another question, is there any provision for
automatic bundling? If sshd requires sshd-log and a oneshot to create a
chroot directory does s6-compile also create a bundle to represent that
relationship or do we need to define those namings ourselves. This is
the inverse of my question about loggers being implicit dependencies of
services.
>
> >As long as A depends on B depends on C, if you ask s6-rc (or whatever)
> >to shutdown A, the dependency manager should be able to walk the graph,
> >find C as a terminal point, and then unwind C to B then finally A. While
> >packagers will screw up their dependency graph, they'll screw it up (and
> >fix it) in the instantiation direction.
>
> If A depends on B depends on C, and you ask s6-rc to shutdown A, then it
> will *only* shutdown A. Hoever, if you ask it to shutdown C, then it will
> shutdown A first, then B, then C. For shutdowns, s6-rc uses the graph of
> reverse dependencies (which is computed at compile time).
>
That's actually what I meant wrt the order. Mostly it was a comment
about how packagers will screw up dependencies but that it'll get
screwed up in the startup direction.
>
>
> >If you have a wireless router running hostapd (to handle monitor mode on
> >your radios) and dnsmasq (for dhcp and dns) you're going to want an
> >ordering where dnsmasq starts before hostapd is allowed to run. There's
> >isn't anything in hostapd what explicitly requires dnsmasq to be running
> >(so no dependency in the classic sense) but you do need those started in
> >the right order to avoid a race between a wireless connection and the
> >ability to get an ip address.
>
> Hmmm.
> If hostapd dies and is restarted while dnsmasq is down, the race condition
> will also occur, right ?
> Since hostapd, like any longrun process, may die at any time, I would
> argue that there's a real dependency from hostapd to dnsmasq. If dnsmasq
> is down, then hostapd is at risk of being nonfunctional. I still doubt
> "ordering" without "dependency" is a thing.
>
It depends on the persons setup. In my case it's a user-supplied
dependency since there's nothing intrisic to dnsmasq or hostapd that
reqires them to run together which I currently get around by polling for
dnsmasq's status from within the hostapd run script. All in all it's a
semantic difference between dependencies that are needed to start
(dependencies), and depenencies that are needed for correct functioning
but are not needed to run (orderings). The nice part is that while there
is a slim difference between the two, all the mechanisms for handling
dependencies can also handle orderings as long as the dependency tracker
handles user-supplied dependencies. Handling user supplied dependencies
also simplifies the system conceptually since people won't have to track
multiple types of requirements.
One last thing and I'm not sure if this has been mentioned earlier, but
how easy is it to introspect the compiled form without using the
supplied tools? A binary dumper is probably good enough, but I'd hate to
be in a situation where the only way to debug a dependency ordering
issue that the compiler introduced is from within the confines of the
s6-rc-* ecosystem.
Cheers!
-Colin
Re: s6-rc design ; comparison with anopa
On 25/04/2015 09:35, Colin Booth wrote: I've been having a hard time thinking about bundles the right way. At first they seemed like first-class services along with longruns and oneshots, but it sounds more like they are more of a shorthand to reference a collection of atomic services than a service in their own right. Especially since bundles can't depend on anything it clarifies things greatly to think of them as a shorthand for a collection of atomic services instead of a service itself. Yes, that's exactly what a bundle is: a shorthand for a collection of atomic services. Sorry if I didn't make it clear enough. As long as A depends on B depends on C, if you ask s6-rc (or whatever) to shutdown A, the dependency manager should be able to walk the graph, find C as a terminal point, and then unwind C to B then finally A. While packagers will screw up their dependency graph, they'll screw it up (and fix it) in the instantiation direction. If A depends on B depends on C, and you ask s6-rc to shutdown A, then it will *only* shutdown A. Hoever, if you ask it to shutdown C, then it will shutdown A first, then B, then C. For shutdowns, s6-rc uses the graph of reverse dependencies (which is computed at compile time). Will s6-rc make loggers implicit dependencies of services, or will need to define that? In other words, if we have a bundle 'ssh-daemon' that contains the longruns sshd and sshd/log, will the dependency compiler correctly link those so that when you ask the bundle to start it brings up sshd/log first, and when you ask it to start it brings down the logger last. Unless there's a compelling reason not to, if there's a "sshd" service and a "sshd-log" service and there's an annotation somewhere in the definition of sshd or sshd-log that sshd-log is the logger for sshd, then s6-rc-compile will automatically create a dependency from sshd to sshd-log, and ensure that $live/servicedirs/sshd is the servie directory for sshd and $live/servicedirs/sshd/log is the service directory for sshd-log. So yes, when you start the sshd-daemon bundle, s6-rc will start the logger first and then the daemon, or stop the daemon first and then the logger. Are oneshots assumed (requored) to be idempotent? Or does $live/state track if a oneshot has already been fired and no-op if that is the case? $live/state tracks oneshots, of course. :) Not to be too pedantic, but how often do daemons ask for user input? I don't know about you, but being required to pass user input on boot is a big nono in my book. Tell that to Olivier. :) ISTR some encrypted filesystems require a passphrase to be given at mount time, so this is a real use case. I don't intend to add terminal support to s6-rc; if the problem comes up, i.e. if several terminal- using services are started in parallel and conflicts arise, I'll think of a specific solution in time. So components of a bundle can fail and the bundle is still considered to be functional? This make sense only if bundles are really tag sets or some other loose grouping. Yes, that's what they are. The atomic services that can be started are still started; however, the "s6-rc -u bundle" invocation will exit nonzero, since some atomic services failed. It is then possible to use some ui to list running atomic services and see what has succeeded and what has failed. If you have a wireless router running hostapd (to handle monitor mode on your radios) and dnsmasq (for dhcp and dns) you're going to want an ordering where dnsmasq starts before hostapd is allowed to run. There's isn't anything in hostapd what explicitly requires dnsmasq to be running (so no dependency in the classic sense) but you do need those started in the right order to avoid a race between a wireless connection and the ability to get an ip address. Hmmm. If hostapd dies and is restarted while dnsmasq is down, the race condition will also occur, right ? Since hostapd, like any longrun process, may die at any time, I would argue that there's a real dependency from hostapd to dnsmasq. If dnsmasq is down, then hostapd is at risk of being nonfunctional. I still doubt "ordering" without "dependency" is a thing. Shouldn't this be handled prior to Stage-2 handoff? At the tail-end of Stage-1 you Try to start a getty along with the catch-all logger. If that fails we bring up our debug shell. Assuming it doesn't fail, Stage-2 starts, s6-rc does its work, and brings up the `gettys' bundle which no-ops on getty-1 (already started) and bring up 2-N. As long as s6-rc is constrained to Stage-2 and multiple start attempts on a service remain idempotent, we should be able to use the same shell escape-hatch mechanisms in the early stages without having to add extra logic handling to the application. Yes, for the initial getty or debug shell, this can be directly handled in stage 1. For other conditional executions, I think conditionally running different s6-rc invocations will be enough. In any case, it can
Re: s6-rc design ; comparison with anopa
* Laurent Bercot [20150425 03:10]: > Bundles are great. You can define a bundle for every possible combination > of services if you're so inclined (it's still 2^n, so don't overdo it, but > yeah). This means that a service can be part of more than one bundle right? > >What I like with aa-enable is that servicedirs are created from a source > >servicedir and some config data. So one can have a single openvpn@ > >source servicedir, and use it to create different servicedirs with > >different configs. There's also the bits to easily use the same logger, > >but also service-specific log script... > > s6-rc-compile actually generates the servicedirs from the source data, > which isn't exactly a servicedir - it contains run and finish scripts, > but also additional information. I haven't settled on a format yet, if > for instance instantiability is important I'll think about supporting it. FWIW, I do believe instantiability is an important feature: it enables the common use case administratively scaling the ability to serve independent but similar services (a.k.a. "virtual" hosting, or "multi-tenant"). > >The wants dependencies are just a way to pull a service without failing > >if that service fails to start, some sort of "optional dependency" I guess. > >E.g. one could say that php only wants sqld, so even if the later can't > >be started the website can work, and everything that doesn't need db > >will work, the rest failing with db connection errors, instead of having > >no website at all. > > I see. With s6-rc, you would just make a "web" bundle, containing "httpd" > and "php", and a "web+sqld" bundle, containing "web" and "sqld". You would > launch web+sqld, and even if sqld fails, httpd and php are still brought up. Remember to put this example in the documentation ;) > >Separating order & dependency also allows to set up some order in > >services in case, but without the need for them to be actual > >dependencies, which may also be handy for source servicedirs to plan for > >different uses/configs. > > Do you have a real-life example ? I can imagine how that can be useful, > but a dependency manager is complex enough as is, and I don't want to > add more complexity if there's no real, actual use for it that cannot > be achieved another way. I believe the motivation here is easy of use: the ability to re-use service definitions for different setups. It's similar to the "wants" dependencies: it's easier to setup a configuration with wider purposes if "wants" is supported. As I see it it's a "nice-to-have": it need not be in an initial release, but it should be in the roadmap since it is not required but useful feature. qvb -- pica
Re: s6-rc design ; comparison with anopa
* Laurent Bercot [20150424 00:24]: > On 23/04/2015 23:26, Joan Picanyol i Puig wrote: > >I'd really expect a ui that can "diff" "compiled" & "live" vs. "source" > >(and > >obviously, to inspect "compile" & "live"). > > There will definitely be a ui to inspect compiled + live. > > As for diffing the current state vs. source, I think it will be too > complex, because it would amount more or less to performing the work > of the compiler again. What can be done is something that compares > two "compiled" databases, so to diff against the source, you would > compile the source into a temp database then compare the temp to the > current compiled. That would certainly be enough. > >I find ./down so convinient that would like having support for it in the > >"source" format. > > The thing is, s6-rc already makes use of ./down internally. When you > run "s6-rc -d this-longrun-service", it first brings down everything > that depends on this-longrun-service, then creates ./down in > this-longrun-service's service directory in live, then calls > s6-svc -d on this-longrun-service. It says that the service is supposed > to be down and remain that way, because that is the state you want to > see enforced. I probably did not explain myself. What I'd like is the ability to have some services "ready-to-run", but not up by default. Some of them might be there for contingency purposes (so that an operator can start a failover), some of them might have to go up (and down) at certain times only. If a longrun-service can't be source-annotated with ./down, this use case requires configuring a oneshot-service just to 's6-rc -d' it which looks akward to say the least. > > > I am thinking about a utility, "s6-rc-update", that would take the > > > live, the old compiled and the new compiled as inputs, and that > > > would update the live as smartly as possible, with carefully > > > designed heuristics; users could also tell s6-rc-update exactly > > > what to do via annotations in the source, that s6-rc-compile would > > > translate into the new compiled. > > > > >Any heuristics will face unsolvable situations. I'd aim at getting the > >"patch" (dual of "diff" above) action right all the time first. > > > That can be done, but with or without heuristics, there will still need > to be a tool to actually update the live state. "diff" is easier than > "patch"; the details of "patch" are what I'm interested in. I lack knowledge & experience to attempt to provide details, so I'll just handwave poiting that the first concept to have clear is that of "identity": is this servicedir a "new" service definition or a modification of an existing one? It then should be feasible to compute the modifications needed to the live DAG (inserting/removing nodes as well as restarting them). qvb -- pica
