Re: s6-rc shutdown timing issue

[Laurent Bercot]

On 13/09/2015 09:08, Colin Booth wrote:

I've been digging into managing a system completely under s6 and I
can't seem to find the right time to run `s6-rc -da change'. Run it
before sending s6-svscan the shutdown/reboot/halt commands and can end
up with a situation where your read/write drive has been set read-only
before your services have come down.


 This is the right way to proceed:
 * First s6-rc -da change
 * Then s6-svscanctl -t /run/service

 I don't understand the issue you're having: why would your rw filesystem
be set read-only before the services come down ?
 - Your non-root filesystems should be unmounted via the corresponding
"down" scripts of the oneshot services you're using to mount them, so
they will be unmounted in order.
 - Your root filesystem will be remounted read-only in stage 3, i.e.
when everything is down.

 If your dependencies are correct, there should be no problem. What
sequence of events is happening to you ?



One other question that doesn't really belong here but doesn't need
its own thread. If I have a oneshot that only does any work on
shutdown, can I get away with having the required ./up script be
empty


 Yes, an empty ./up script will work. (Oneshot scripts are interpreted
at compile time by the execlineb parser, which treats an empty script
as "exit 0".)

--
 Laurent

Re: s6-rc shutdown timing issue

[Laurent Bercot]

On 13/09/2015 20:25, Colin Booth wrote:

My current issue is that I'm initially remounting my root filesystem
as r/w as one of the first steps for s6-rc, which means that if I'm
doing everything correctly, s6-rc attempts to remount root as
read-only as part of its shutdown.


 Yeah, indeed, that won't work in all cases, because the situation is not
symmetrical. Remounting your rootfs rw at boot time will always succeed,
but remounting it ro before killing everything may fail.

 You have absolutely no way of ensuring that nothing will attempt to write
to your rootfs before you nuke everything. That's the fundamental difference
between startup and shutdown: during shutdown, and until your nuke, other
stuff may still be running that you have no control on.

 If it's only about your rootfs, I'd simply keep the "remount it rw/ro" part
out of s6-rc.
 If it's about a more complex fs infrastructure and you may still have
processes with open handles to the mounted filesystems, it's more annoying,
and I don't have a perfect solution. The simplest thing is to do all the
unmounts outside of s6-rc, but it's asymmetrical and doesn't feel right.
Another possibility is to have a nuke in a ./down script of a oneshot that
depends on all your filesystem-mounting service, so you're sure to have
already killed everything when you get to unmounting; but that's not great
either, especially if you switch databases and s6-rc-update computes that
it has to remount one of your filesystems - oops, it just killed everything
on your machine and can't complete its work. That shouldn't happen if your
"mount stuff" services are very low-level and always up, but if you give
users a set of teeth, they will manage to bite themselves.

 I'm afraid there's no real solution to the stragglers problem, and the
only safe approach is to keep everything mounted all the time and only
perform the unmounts in stage 3 after everything else has been done and
the processes have been killed.



Cool, thanks. That's what I thought but I wasn't sure to what degree
execlineb cared about script validity and I don't have a terribly
great test methodology for oneshots figured out yet.


 If you run s6-rc -v3, it should show you exactly what commands it's
running.

 In other news, I'm now in the process of testing s6-rc-update. I've
finished brooming the obvious bugs, now is the most annoying part: so
many test cases, so many things that can go wrong, and I have to try
them one by one with specifically crafted evil databases. Ugh. I said
I'd release that thing in September, but that won't happen if I die of
boredom first.
 If you're totally crazy, you can try running it, but unless you're
doing something trivial such as switching to the exact same database,
chances are that something will blow up in your face - in which case
please let me analyze the smoke and ashes.

--
 Laurent

Re: s6-rc shutdown timing issue

[Colin Booth]

On Sun, Sep 13, 2015 at 2:34 PM, Laurent Bercot  wrote:
>
>  I'm afraid there's no real solution to the stragglers problem, and the
> only safe approach is to keep everything mounted all the time and only
> perform the unmounts in stage 3 after everything else has been done and
> the processes have been killed.
>
Yeah, the asymmetry is annoying, but all the other options are worse.
Sounds like current best practices should be to keep rw->ro remounts
out of s6-rc (at least / and /var), call `s6-rc -da change' before
signalling s6-svscan, and then explicitly do those remounts in stage
3. Like I said, not great but better than the alternatives, and a lot
less magic than it could be.
>
>  In other news, I'm now in the process of testing s6-rc-update. I've
> finished brooming the obvious bugs, now is the most annoying part: so
> many test cases, so many things that can go wrong, and I have to try
> them one by one with specifically crafted evil databases. Ugh. I said
> I'd release that thing in September, but that won't happen if I die of
> boredom first.
>  If you're totally crazy, you can try running it, but unless you're
> doing something trivial such as switching to the exact same database,
> chances are that something will blow up in your face - in which case
> please let me analyze the smoke and ashes.
>
When I've got some time I'll spin up a vm and let you know what the
disaster looks like. What do you want in terms of debuging? I'm
assuming old and new service databases at the minimum. Odds are they
won't be nearly as terrible as your custom-built databases, but who
knows.

Cheers!


-- 
"If the doors of perception were cleansed every thing would appear to
man as it is, infinite. For man has closed himself up, till he sees
all things thru' narrow chinks of his cavern."
  --  William Blake