Hi Casper
Am 20.10.21 um 20:01 schrieb Casper Ti. Vector:
On Wed, Oct 20, 2021 at 09:53:58AM +0200, Ben Franksen wrote:
Interesting, I didn't know about recordio, will take a look.
Hello from a fellow sufferer from EPICS. (If you see a paper on some
synchrotron-related journal in a few months that mentions "automation
of automation", it will be from me, albeit not using a pseudonym.
Another shameless plug: <https://github.com/CasperVector/ADXspress3>.)
Interesting, I didn't know you are from the accelerator community!
As has been said by Laurent, in the presence of a supervision system
with reliable logging and proper rotation, what `procServ' mainly does
can be done better by something like `socat' which wraps something like
`recordio', which in turn wraps the actual service process (EPICS IOC).
Yeah, that's what I was thinking, too.
The devil is in the details: most importantly, when the service is to
be stopped, the ideal situation is that the actual service process gets
killed, leading to the graceful exit of `recordio' and then `socat'.
So the two wrapping programs need to propagate the killing signal, and
then exit after waiting for the subprocess; since `procServ' defaults
to kill the subprocess using SIGKILL, `recordio' also needs to translate
the signal if this is to be emulated. `socat' does this correctly when
the `sighup'/`sigint'/`sigquit' options are given for `exec' addresses,
but its manual page does not state about SIGTERM. `recordio' does not
seem to propagate (let alone translate) the signal; additionally, its
output format (which is after all mainly used for debugging) feels too
low-level to me, and perhaps needs to be adjusted.
I agree. BTW, another detail is the special handling of certain control
characters by procServ: ^X to restart the child, ^T to toggle
auto-restart, and the possibility to disable some others like ^C and
especially ^D; which is not only convenient but also avoids accidental
restarts (people are used to ^D meaning "exit the shell").
At the facility where I am from, we use CentOS 7 and unsupervised
procServ (triple shame for a systemd opponent, s6 enthusiast and
minimalist :(), because we have not yet been bitten by log rotation
problems. It also takes quite an amount of code to implement the
dynamic management of user supervision trees for IOCs, in addition
to the adjustments needed for `recordio'. To make the situation even
worse, we are also using procServControl; anyway, I still hope we can
get rid of procServ entirely someday.
Our approach uses a somewhat hybrid mixture of several components. Since
the OS is Debian we use systemd service units, one for each IOC. They
are executing `/usr/bin/unshare -u sethostname %i runuser -u ioc --
softIOC-run %i` which fakes the host name to trick EPICS' Channel Access
"Security" into the proper behavior, and then drops privileges.
softIOC-run is the script of which I posted a simplified version, with
the pipeline between procServ and multilog. Despite the disadvantages
explained by Laurent, so far this works pretty well (I have never yet
observed multilog to crash or otherwise misbehave). Finally, the
configuration for all IOCs (name, which host do they run on, path to the
startup script) all reside in a small database and there are scripts to
automatically install everything, including automatic enabling and
disabling of the service units.
When I started developing this scheme I thought that systemd was a great
leap forward from /etc/init.d scripts. I still think so, but I quickly
became frustrated with its monolithic approach. Despite 1000s of
configuration options, it always seemed like the one I needed was
missing. I spend days and days debugging service units that should have
worked according to the docs but did not, for reasons I wasn't always
able to figure out. Nowadays my standing assumption about systemd is
that nothing you didn't thoroughly test should be expected to work,
regardless of what the docs claim.
In contrast, I found that small specialized tools that use the
chain-loading technique to modify a particular aspect of a program much
more reliably produce exactly the desired effect and nothing more. The
fine-grained control this gives you over the order of these effects
(like, first fake the host name, then drop privileges) is something that
a monolith with an unstructured flat configuration language cannot give
you. The syntactic simplicity of systemd's configuration language is
certainly appealing, especially for non-programmers, but this easily
lets you forget the extreme complexity of its semantics. I cannot help
but see the machine executing it as an idiosyncratic monster with lots
of poorly handled corner cases.
I would like to experiment with alternatives like s6/s6-rc but that
means using one of the small distros that support it and I am sure such
a proposal would not be well received.
Cheers
Ben
--
I would rather have questions that cannot be answered, than answers that
cannot be questioned. -- Richard Feynman
