Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 21.04.2017 21:31, k...@aspodata.se wrote:

> a, If you have that impression, then why daemonize() which sounds as
>the same thing ?
> 
> b, No it isn't. That's why the -f, which is nice to have when debugging.

If you dont want the service to daemonize itself, a debug flag doesn't
seem the correct approach, just a workaround. So they should at least
implemented some --foreground option that just prevents daemonizing,
but doesn't enable extra logging.

And here we are - lots of code duplication in all the individual
daemons. And operators have to look up how to actually call it in the
individual case. We could consolidate that into exactly one place.
(actually, I'd prefer services never daemonize themselves and always
leave that to a separate program of operators choice)

>> That '-f' switch would be in
>> the lib in a different form, eg. via env variables.
> 
> Yes, it's messier to pick just the -f from argc/argv in a lib than
> calling getenv().

You're assuming that everybody implements an '-f' switch, and it's
always exactly '-f' - that flag could mean something completely
different (perhaps f for file). Again and again, operators have to
look up the manpage and be careful not to mix things up.
(or even patch up services that don't even have that flag yet)

With that library function, we have all of that consolidated in
one place and don't even have to change any command line syntax
anymore.

>> My goal here is not voting for any particular way of daemonizing,
>> service monitoring, etc, but just to provide an abstraction layer, so
>> all that stuff can be moved out of the individual application.
> ...
> 
> Well, that's fine, but replaceing -f and daemon() with your entry 
> doesn't seem worth it. Maybe it's better to write up some
> "best practices" ?

Sure, best practices would be nice, but then we can run into problems
like -f already used for something else. And again the operator needs
to handle it individually in his startup scripts / service rules / etc.

The best practice could also state: "please use that library if can,
and leave the rest to the operator / distro". Or perhaps "don't ever
let your programs daemonize themselves".

> Then perhaps it's better to make it easier to use "standard"
> command line options, and put that in a lib. E.g. a few funcs.
> that
> parses argc/argv according to some data, extracts values, puts in
> default values, prints error messages, usage text or version
> strings without all the usual mess with option handling.

That would be nice, too, but I would put it into a different library,
and certain point the program would still carry the code looking at the
flag and deciding whether to daemonize. Yes, not much code (OTOH:
rewriting to do the command line parsing by the lib would be more
invasive, and you'll have trouble if -f is already used otherwise),
but we'd have it consolidated in one place and operators can easily
replace that whenever they feel necessary.

> I think it is more fruitful in that case to write a lib that unifies the
> command line, then environment and the config file handling, and make
> it easy for the daemon writer to use it. 

Same as above.

> Also of value could be a lib that unifies logging, or rather removes the
> distinction between logging to syslog(), to a file or to stdout/stderr;
> possible with default callback for SIGHUP to reopen/rotate log file.

Actually, applications should either use syslog (except for debugging)
or just use stderr and leave it to the daemonizing/monitor program.

> Yes, and as a intermittent daemon writer, I try to understand why I
> should use it. If I cannot find any reason then maybe we should think
> of something better.

Well, lots of developers see the need for having such an reporting.
I, personally, never really had a real usecase for that, but these
folks want that, and therefore use the only API they have - systemd.

> There are two things,
> 1, the monitor needs
> 2, the (possible monitored) process/daemon needs
> 
> To a daemon writer, point 1 is simply uninteresting. To get point 1
> satisfied, you have to provide something that first and formost
> fulfill point 2, and by providing for point 2, you in your lib, can
> introduce things that makes it easier for the monitor.
>
> So, instead of your proposal, I propose a lib that streamlines the
> dull tasks of option/env/config_file, logging, and possible signal
> handling. With it you can introduce things that is helpful for a 
> monitor.

That would be a much wide scope. I was just trying to design a sane
API for what many applications link in systemd now.


--mtx
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Enrico Weigelt:
> On 18.04.2017 14:46, k...@aspodata.se wrote:
> 
> > Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in
> > either camp, end case.
> 
> I've had the impression that daemon() wasnt't good idea for everybody,

a, If you have that impression, then why daemonize() which sounds as
   the same thing ?

b, No it isn't. That's why the -f, which is nice to have when debugging.

> and there might be some need for anyone doing something these.

The any other thing I can think of is running in the foreground and
"logging" to stdout/stderr, and possible being able to receive
commands via stdin.

> So I
> proposed to move that out to some library, which the operator easily
> replace, w/o touching the application itself. (patching up libc to their
> needs isn't quite what operators do ...).

 if (do_daemon) { daemon(0,0); }

isn't much code to justify a lib entry, so what else is there in it to 
lure a daemon writer to use it ?

> That '-f' switch would be in
> the lib in a different form, eg. via env variables.

Yes, it's messier to pick just the -f from argc/argv in a lib than
calling getenv().

> My goal here is not voting for any particular way of daemonizing,
> service monitoring, etc, but just to provide an abstraction layer, so
> all that stuff can be moved out of the individual application.
...

Well, that's fine, but replaceing -f and daemon() with your entry 
doesn't seem worth it. Maybe it's better to write up some
"best practices" ?

> Certainly, everybody can implement the '-f' switch on his own (and it
> shouldn't be so difficult), but all of that repeated work (and the
> operator's work of reading manpages to find out which exact parameters
> he has to pass to an individual program to switch it on/off), could
> be saved if there was a single point for that. If you (as a service
> author) don't wanna use that, it's still your decision.

Then perhaps it's better to make it easier to use "standard"
command line options, and put that in a lib. E.g. a few funcs. that
parses argc/argv according to some data, extracts values, puts in
default values, prints error messages, usage text or version
strings without all the usual mess with option handling.

> > Why srvmgt_droppriv(), either accept that the process set's it itself
> > or force it to some uid:gid, what is it more to it ?
> 
> It could even detect (eg. via env variables) whether it's already suid'd
> and then not even try and fail, it could fetch the target uid/gid from
> env, instead of having it's own (application specific) config or cmdline
> options. Again, consolidating repeated code into one place. Nothing
> more, nothing less. Similiar to srvmgt_daemonize(), it's only meant for
> cases where the service can't be started with the final uid/gid in the
> first place.

I think it is more fruitful in that case to write a lib that unifies the
command line, then environment and the config file handling, and make
it easy for the daemon writer to use it. In that kind of lib, you could
include default options for things like to daemon or not, uid/gid,
and where to log things.

I, as a daemon writer can see the value in having that handling defaulted
to some lib, and by it you can introduce default handling for -f and 
--uid/--gid, so a possible monitor can set that what it to whatever it
wants.

Also of value could be a lib that unifies logging, or rather removes the
distinction between logging to syslog(), to a file or to stdout/stderr;
possible with default callback for SIGHUP to reopen/rotate log file.

You have to introduce what you want the other way around.

> > Why srvmgt_report_state(), just do a normal query to the process and
> > you'll know if it is ready to serve you or not, 
> 
> There might be cases when the process being alive doesn't necessarily
> mean it's ready to serve (usually it takes *some* time). By the call,
> the application can tell it to the outside world (however the actual
> signalling might happen, and who might listen).
> 
> NOTE: I'm just proposing an *API*, *not* an particular implementation.

Yes, and as a intermittent daemon writer, I try to understand why I
should use it. If I cannot find any reason then maybe we should think
of something better.

> All I wanna get is that individual applications have an API (and ABI)
> to do these things w/o ever having to care whatever init system,
> service monitor, etc, there might or might not be. And the actual
> implementation shall be replacable w/o ever recompiling applications.

There are two things,
1, the monitor needs
2, the (possible monitored) process/daemon needs

To a daemon writer, point 1 is simply uninteresting. To get point 1
satisfied, you have to provide something that first and formost
fulfill point 2, and by providing for point 2, you in your lib, can
introduce things that makes it easier for the monitor.

So, instead of your proposal, I propose a lib that streamlines the
dull tasks of option/env/config_file, logging, and possible 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 18.04.2017 12:21, Alessandro Selli wrote:

>   Again, as far as I understood what Enrico Weigelt wrote, the monitor
> does not want to know the user:group the daemons runs under, it needs to
> *set* them to the appropriate values when it launches the daemon to have
> them reflect config file settings or local policies.

Right. In that case, srvmgt_droppriv() would essentially do nothing
(execpt perhaps some logging note).

OTOH, we might have situations where the daemon needs to do some root
operations first (eg. opening privileged sockets) before dropping privs.
In that case the monitor/init can pass the target uid:gid via env, so
it doesn't need to be configured within the application.

>   Again, the relevant info is supposed to be set in config files.  The
> user:group the daemon must run under is not up to the daemon itself,
> they are set in config files, either the deamon's or the monitor's.

Exactly. My approach also gives the choice of passing that from the
init/monitor, even if the application needs to to the setuid() stuff
itself.

At that point we can consolidate all these individual assignments from
dozens of separate configfiles to one place, somewhere within the init
system / service monitor (which ever the operator might have chosen).
In the end, the application doesn't even care about that anymore.

> Again: if you do not want to take advantage of the monitor's policing
> capabilities, don't use them.

Exactly. And ease deployment and configuration for both cases, I'm
just proposing to move these things out of individual applications and
consolidate them in one point (the library). If one doesn't want any
monitor, then he'll just install a minimalistic version, which wouldn't
be much more than dummies or thin wrappers around things like daemon().
Anybody can choose his preferred implementation easily, w/o ever
recompiling or even patching.


--mtx
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 18.04.2017 14:46, k...@aspodata.se wrote:

> Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in
> either camp, end case.

I've had the impression that daemon() wasnt't good idea for everybody,
and there might be some need for anyone doing something these. So I
proposed to move that out to some library, which the operator easily
replace, w/o touching the application itself. (patching up libc to their
needs isn't quite what operators do ...). That '-f' switch would be in
the lib in a different form, eg. via env variables.

My goal here is not voting for any particular way of daemonizing,
service monitoring, etc, but just to provide an abstraction layer, so
all that stuff can be moved out of the individual application.

All the srvmgt_daemonize() call tells is that after that point the
process will be daemonized - whatever that *actually* means in the
exact setup (which is up to the operator's choice). This is *only* for
the usecase where an service wants to have a straight line at all
(eg. before that it might print out error messages to stdout).
Those who don't need that at all, probably just run in foreground
and delegate daemonizing to the caller (monitor, start-stop-daemon,
etc, etc, etc)

Certainly, everybody can implement the '-f' switch on his own (and it
shouldn't be so difficult), but all of that repeated work (and the
operator's work of reading manpages to find out which exact parameters
he has to pass to an individual program to switch it on/off), could
be saved if there was a single point for that. If you (as a service
author) don't wanna use that, it's still your decision.

> Why srvmgt_droppriv(), either accept that the process set's it itself
> or force it to some uid:gid, what is it more to it ?

It could even detect (eg. via env variables) whether it's already suid'd
and then not even try and fail, it could fetch the target uid/gid from
env, instead of having it's own (application specific) config or cmdline
options. Again, consolidating repeated code into one place. Nothing
more, nothing less. Similiar to srvmgt_daemonize(), it's only meant for
cases where the service can't be started with the final uid/gid in the
first place.

> Why srvmgt_report_state(), just do a normal query to the process and
> you'll know if it is ready to serve you or not, 

There might be cases when the process being alive doesn't necessarily
mean it's ready to serve (usually it takes *some* time). By the call,
the application can tell it to the outside world (however the actual
signalling might happen, and who might listen).

NOTE: I'm just proposing an *API*, *not* an particular implementation.

All I wanna get is that individual applications have an API (and ABI)
to do these things w/o ever having to care whatever init system,
service monitor, etc, there might or might not be. And the actual
implementation shall be replacable w/o ever recompiling applications.



--mtx
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 18.04.2017 15:15, Arnt Gulbrandsen wrote:

> There's hypothetical stuff, what if service x needs service y. Well,
> what if it does. Should it demand that y be running at every moment and
> on the same host? DOES it demand that?

nfs daemons (plural) need portmap.

many years ago, I've patched portmap to be fully stateless (directly
maintains the table in some state files), so it can even be started
on per-request basis. no idea whether my patches ever went into
mainline ...

> This isn't a good thing to spend effort on. The trend today is towards
> services that require only an OS absolutely. 

In general a good idea (unless they don't duplicate so dozens of things
over and over again). But there're still services out there that have
those constraints - and somehow we gotta cope w/ that.

Of course, often it's just bad sw architecture - just like most of the
stuff that distros have to cope with. We could start w/ dist patches ...


--mtx
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 18.04.2017 16:08, Alessandro Selli wrote:

Hi folks,

seems that things got a bit overheated here ...

>   Where did Enrico Weigelt write the monitor should get it's child PID
> from the process itself?  This subthread started from these lines:
> 
>>> * srvmgt_droppriv(...)
>>>   --> drop root privileges (if we are still root)
>>>   --> several versions, eg. with fetching the target uid/gid from env
>> Given the pid, it isn't that hard for a monitor to find the uid/gid of 
>> the process, why has this have to go through the monitor ?

Perhaps I should clarify this a bit:

This library functions just shall drop privileges if the process is
still running as root - and therefore might fetch the target uid/gid
from environment (defined by start script, monitor, etc), instead of
fetching from it's own config file. It's just implementing common
repeated stuff in once central place and giving it a uniform interface.
As that library would be provided by $whatever_initsystem, any further
customizations would be done *there* instead of individual applications.

The whole thing only applies to services which *need* to be started as
root, instead of directly under the final uid/gid. (whether such things
are good design at all is a whole different discussion).

> «All daemons detach from the control terminal. The monitor and the
> daemon must coordinate each other about who's doing the detach to avoid
> launch time failures.»

ACK. That's why I would put the code that does all (on the daemon side)
into a separate library, which can be customized to whatever init/
monitor system the operator might choose, and the daemon itself doesn't
need to be touched (not even recompiled) anymore.

In that scenario, each init system (distro) package would provide an
implementation of that library - all of them share the same API/ABI,
so they can be easily replaced, w/o ever having to rebuild any daemon.

> Daemons should behave the same no matter under what monitor they are
> running under, they are not to care about this.

ACK. But there are some points where they potentially need to care
about, eg. who does the actual dataching (unfortunately, there isn't
any ultimate standard) - and I'd like to get rid of individual
per-daemon configuration (in both daemon and monitor) and delegate that
part to a library. The operator (or the init system package maintainer)
is free to put in the appropriate implementation.


--mtx

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[KatolaZ]

On Tue, Apr 18, 2017 at 11:10:24PM +0200, Alessandro Selli wrote:
> On 18/04/2017 at 19:01, KatolaZ wrote:
> > On Tue, Apr 18, 2017 at 06:48:31PM +0200, Alessandro Selli wrote:
> >
> > [cut]
> >
> >>> There is really no other reasonable alternative, IMVHO.
> >>   Yes, there is.  It's called a good compromise.  Then, outside of it,
> >> you find the purists and the extremists.
> >>
> > We have seen what an irresistible avalanche of compromises has created
> > in the systemd camp.
> 
>   Today I read someone (perhaps Enrico Weigelt) stating that one of
> systemd's
> pitfalls is wanting to be the perfect init.  I agree.  systemd does not
> compromise, it wants thing to be done the way it wants, it forces all
> userland to comply to it's dictates.  systemd is the opposite of compromise,
> it's dictatorial.
> 
> > It turns out that I am a purist. And an extremist.
> 
>   This is one of the best recipies for failure.
> 

It would be very good if you would like to actually contribute to
Devuan :) There are lots of thigs to do, besides deciding who will
fail and who will succeed. 

HND

KatolaZ

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[ "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[   @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[ @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]


signature.asc
Description: Digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 19:01, KatolaZ wrote:
> On Tue, Apr 18, 2017 at 06:48:31PM +0200, Alessandro Selli wrote:
>
> [cut]
>
>>> There is really no other reasonable alternative, IMVHO.
>>   Yes, there is.  It's called a good compromise.  Then, outside of it,
>> you find the purists and the extremists.
>>
> We have seen what an irresistible avalanche of compromises has created
> in the systemd camp.

  Today I read someone (perhaps Enrico Weigelt) stating that one of
systemd's
pitfalls is wanting to be the perfect init.  I agree.  systemd does not
compromise, it wants thing to be done the way it wants, it forces all
userland to comply to it's dictates.  systemd is the opposite of compromise,
it's dictatorial.

> It turns out that I am a purist. And an extremist.

  This is one of the best recipies for failure.

>  And I am not at all
> interested in blurry concepts like "market shares",

  If this feeling is shared by over 25% of Devuan's managers and
developers, it's going to be the 1000th distribution so nice, so good, so
"perfect" it's going to starve out of it's own failure to gain any
significant foothold into the marketplace.  You might dislike the market as
much as you can, but that's what feeds projects and keeps distributions
alive.  Voluntary donations might get you started, but will not prop you up
indefinitely into a viable OS capable of sustaining itself.

>  since the quest to
> let Linux conquest "new market shares" has only brought problems and
> created monsters. Get your conclusions.

  The quest to get Linux into as many lucrative markets as possible has made
it what it is, that is a success, unlike worthy, perfectionist projects that
failed nonetheless, like Plan9 or many of the successors to OpenSolaris.
Who mentioned GNU?


-- 
Alessandro Selli 
VOIP SIP: dhatarat...@ekiga.net
Chiave PGP/GPG key: B7FD89FD

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 16:16, KatolaZ wrote:
> On Tue, Apr 18, 2017 at 02:48:54PM +0100, Arnt Gulbrandsen wrote:
>> kato...@freaknet.org writes:
>>> Unfortunately we are already paying the consequences of badly-written
>>> software implementing oddly-designed solutions to non-existing
>>> problems...
>> Indeed. But what's your point?
>>
> Oh, you are right, I didn't make it explicit: my humble opinion is
> that the whole thread is not particularly interesting or useful in
> this context, since all this fuss about monitoring/supervision boils
> down to two alternative options:
>
>   a) you *really* need *strict* monitoring/supervsion. In this case,
>   you *don't* use a unix-like operating system, since you are always
>   left at least with the problem of monitoring the monitor (or
>   supervising the supervisor). But this is not a real problem, since
>   in those environments where *strict* supervision/monitoring is
>   *really* necessary, unix-like systems are rarely used, if ever, and
>   the scheduler also acts as a monitor/supervisor. So if you need
>   *real* *strict* monitoring, the whole topic is off-topic here.

  I do not agree, as Linux systems are employed both in real-time
scenarios (though I admit this is a really niche use case) and in High
Availability infrastructures.

>   b) you need *loose* monitoring/supervision. In this case, you pick
>   and choose among what is available, or write your own. In any case,
>   you get a monitor/supervisor running as a user process on top of any
>   general-purpose operating system, possibly a unix-like one, and your
>   applications must still be able to deal with faults,
>   inconsistencies, delays, and other issues. And sometimes you will
>   still need manual intervention nonetheless. The "everything will be
>   all right" theme is just dust in your eyes, because things are never
>   going to be all right all the time.

  This is an argument that sounds as an excuse to me: "Since you'll
never get perfection, just do nothing."  Everyone knows that even top500
supercomputers do need some manual intervention from time to time.  This
is not a good reason to put the sysadmin in charge of manually managing
all the processes and events that they produce.  Some OS could go this
way, of course, free to.  But you know very well that such an OS, though
philosophically and academically fine and technically elegantly
constructed, is not going to win any significant market share, as today
everywhere the utmost of automation is a prime requirement.

> There is really no other reasonable alternative, IMVHO.

  Yes, there is.  It's called a good compromise.  Then, outside of it,
you find the purists and the extremists.


Alessandro

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 14:46, k...@aspodata.se wrote:

[...]

> Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in
> either camp, end case.

  This is well *if* the daemon has -f and does daemon().  I know the
only programs that do not do it are those that were not intended to be
run as daemons, still from time to time people do that, with netcat for
instance.

> Why srvmgt_droppriv(), either accept that the process set's it itself
> or force it to some uid:gid, what is it more to it ?

  "What if you do not trust the program?" (quote)

  Then you answered yourself already: "force it to some uid:gid": this
is the monitor's job.  One of them.

> Why srvmgt_report_state(), just do a normal query to the process and
> you'll know if it is ready to serve you or not,

  This way you only know that the process is up, not that "it is ready
to serve you or not".  I'm sure you've had experiences of processes that
are running but are not delivering the services you expected from them.

>  if you want to know
> if it is up to speed, do a benchmark, if you want to test the 
> integrity, do a regression test; what is it more to it ?

  Knowing if it is ready to serve you or not", for instance.  Learning
if it got stuck on something, if that is temporary or definitive, if
it's best to kill it or maybe raise it's allotted max number of open
files.  Or just write a line in the logs and let the sysadmin pick it up
and see for himself what to do.

  If you want an init that only launches programs and daemons and as
soon forgets all about them, fine.  I'm not in principle against such a
minimalistic init.  Just don't call it a monitor thought, 'cause it ain't.


Alessandro

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 16:42, k...@aspodata.se wrote:
> Alessandro:
>> On 18/04/2017 at 15:11, k...@aspodata.se wrote:
>>> Alessandro Selli:
>>> ...
 And
 what I don't like of Karl Aspo's idea is that it takes any instrument of
 policy checking and enforcing out of the monitor, which ends up not
 being able to monitor anything, it becomes just a shell: fire and forget.
>>> I honestly do not understand what I have written which gives you that idé.
>> Your many lines that the monitor is not to do any monitoring.  Like:
>> "Why do any monitor program need to know if the program has detached or
>> not, the only thing it needs to know is the pid and the state,".
> You cut out the thing after the "," which makes that a question for 
> someone else to fill in the blanks, the things that's not obvious
> to me.

  I got you had a question, I did understand you had no idea why would
"any monitor program need to know if the program has detached or not". 
Again I quote myself:

«All daemons detach from the control terminal. The monitor and the
daemon must coordinate each other about who's doing the detach to avoid
launch time failures.»

  It's been obvious to all Unix sysadmins for thirty sound years, if
it's still not clear to you I think I must give up.

> If a process knows the pid of another process and have sufficient 
> rights, can it not manage that process or is there anything else it 
> needs to know, what am I missing ?

  It manages them based on what?  Local config files, events
transmitted, signals?  You're missing the whole "monitoring" and
policing the monitor does besides launching the daemon.


Alessandro

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Alessandro:
> On 18/04/2017 at 15:11, k...@aspodata.se wrote:
> > Alessandro Selli:
> > ...
> >> And
> >> what I don't like of Karl Aspo's idea is that it takes any instrument of
> >> policy checking and enforcing out of the monitor, which ends up not
> >> being able to monitor anything, it becomes just a shell: fire and forget.
> > I honestly do not understand what I have written which gives you that idé.
> Your many lines that the monitor is not to do any monitoring.  Like:
> "Why do any monitor program need to know if the program has detached or
> not, the only thing it needs to know is the pid and the state,".

You cut out the thing after the "," which makes that a question for 
someone else to fill in the blanks, the things that's not obvious
to me.

If a process knows the pid of another process and have sufficient 
rights, can it not manage that process or is there anything else it 
needs to know, what am I missing ?

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Katola2:
> On Tue, Apr 18, 2017 at 03:11:51PM +0200, k...@aspodata.se wrote:
...
> > If I can get a value from the kernel instead of from the process,
> > I'd take the kernel value.
> > 
> > Why do a process have to query the kernel to get a value and then 
> > sending it to a monitor over a communication link; why don't the
> > monitor query the kernel itself, saving one step.

> I genuinely don't understand why the kernel should know about the
> internals of running processes, or get notified if a process is
...

What is the problem with you guys, I did write "if". That "if" applies 
to thing like PID, uid, gid and such of processes, not wheter they are 
in some sense ready or not. Please, don't continously misinterpret me.

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[KatolaZ]

On Tue, Apr 18, 2017 at 02:48:54PM +0100, Arnt Gulbrandsen wrote:
> kato...@freaknet.org writes:
> >Unfortunately we are already paying the consequences of badly-written
> >software implementing oddly-designed solutions to non-existing
> >problems...
> 
> Indeed. But what's your point?
> 

Oh, you are right, I didn't make it explicit: my humble opinion is
that the whole thread is not particularly interesting or useful in
this context, since all this fuss about monitoring/supervision boils
down to two alternative options:

  a) you *really* need *strict* monitoring/supervsion. In this case,
  you *don't* use a unix-like operating system, since you are always
  left at least with the problem of monitoring the monitor (or
  supervising the supervisor). But this is not a real problem, since
  in those environments where *strict* supervision/monitoring is
  *really* necessary, unix-like systems are rarely used, if ever, and
  the scheduler also acts as a monitor/supervisor. So if you need
  *real* *strict* monitoring, the whole topic is off-topic here.

  b) you need *loose* monitoring/supervision. In this case, you pick
  and choose among what is available, or write your own. In any case,
  you get a monitor/supervisor running as a user process on top of any
  general-purpose operating system, possibly a unix-like one, and your
  applications must still be able to deal with faults,
  inconsistencies, delays, and other issues. And sometimes you will
  still need manual intervention nonetheless. The "everything will be
  all right" theme is just dust in your eyes, because things are never
  going to be all right all the time.

There is really no other reasonable alternative, IMVHO. But I am not
an expert in the field (though I am quite happy about not being an
expert, having seen what the "experts in the field" have been able to
come up with in the last couple of years...)

My2Cents

KatolaZ

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[ "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[   @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[ @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]


signature.asc
Description: Digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 15:11, k...@aspodata.se wrote:
> Alessandro Selli:
> ...
>> And
>> what I don't like of Karl Aspo's idea is that it takes any instrument of
>> policy checking and enforcing out of the monitor, which ends up not
>> being able to monitor anything, it becomes just a shell: fire and forget.
> I honestly do not understand what I have written which gives you that idé.

  Your many lines that the monitor is not to do any monitoring.  Like:
"Why do any monitor program need to know if the program has detached or
not, the only thing it needs to know is the pid and the state,".

>> I was also been ironic on Aspo, as many times he can only counter
>> another person's ideas asking "What if  cannot be trusted?",
>> as if this constitutes a valid argument against being able to set
>> policies for the monitor to enforce on the daemons it runs.
> If I can get a value from the kernel instead of from the process,
> I'd take the kernel value.

  What value?  Of course there are values that are set and managed by
the kernel, not by the process, and are thus most dependable when
extracted from the kernel itself.  However other values are specific to
the daemon and the kernel doesn't care/know anything of, like what
virtual domains the webserver is handling, of how many domains the DNS
server is master.  Anyway, this has very little (if anything) to do with
the monitor's need to be able to... monitor the processes it launches,
set them to the wanted user:group and the daemons knowing whether they
are to detach from the controlling terminal or not.

> Why do a process have to query the kernel to get a value and then 
> sending it to a monitor over a communication link; why don't the
> monitor query the kernel itself, saving one step.

  Where did Enrico Weigelt write the monitor should get it's child PID
from the process itself?  This subthread started from these lines:

>> * srvmgt_droppriv(...)
>>   --> drop root privileges (if we are still root)
>>   --> several versions, eg. with fetching the target uid/gid from env
> Given the pid, it isn't that hard for a monitor to find the uid/gid of 
> the process, why has this have to go through the monitor ?


>>  In this
>> case, asking "why should a program/daemon care if it has a monitor or
>> not ?" is moot, because that the daemon is aware or not that it was
>> launched by a monitor and if so by what monitor, does not change
>> anything about the monitor's functions and duties.  The daemon doesn't
>> care if it's run by a monitor?  The monitor is still there, and it does
>> care about the program.
> It would help my understanding if you simply answered my question.

  An answer to a moot question is Mu!
(https://en.wikipedia.org/wiki/Mu_%28negative%29)
  The only point of any technical interest was already answered:

«All daemons detach from the control terminal. The monitor and the
daemon must coordinate each other about who's doing the detach to avoid
launch time failures.»

  And again, so long as it's configured the right way, that the daemon
is aware that it is run under a monitor or not is irrelevant: it is the
monitor that need to be aware of it's children processes, their status
and operational parameters, because it is responsible of what they do,
starting from setting the user:group they run under.  Daemons should
behave the same no matter under what monitor they are running under,
they are not to care about this.  This is one of the reasons systemd is
bad.  Still, the monitor does the monitoring.


>> A driver does not care if the traffic light is
>> red or green?  The street police does.
> This is not as a suitable metaphor as you think it is.

  It is, no matter what you think about it.


Alessandro

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Arnt Gulbrandsen]

kato...@freaknet.org writes:

Unfortunately we are already paying the consequences of badly-written
software implementing oddly-designed solutions to non-existing
problems...


Indeed. But what's your point?

Arnt
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Arnt Gulbrandsen]

kato...@freaknet.org writes:

I genuinely don't understand why the kernel should know about the
internals of running processes, or get notified if a process is
"ready" to do whatever it is supposed to do, or get queried by other
processes which would like to access this kind of information, or
maintain such information in the first instance...


Indeed. Particularly in light of the modern fashion of running things on 
other hosts reachable across the network, perhaps via load balancers that 
complicate the availability state machine.


Arnt
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[KatolaZ]

On Tue, Apr 18, 2017 at 02:15:38PM +0100, Arnt Gulbrandsen wrote:
> I've read this thread fairly thoroughly, but fail to see much of a use
> case...
> 
> There's hypothetical stuff, what if service x needs service y. Well, what if
> it does. Should it demand that y be running at every moment and on the same
> host? DOES it demand that?
> 

Unfortunately we are already paying the consequences of badly-written
software implementing oddly-designed solutions to non-existing
problems...

HND

KatolaZ

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[ "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[   @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[ @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]


signature.asc
Description: Digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Alessandro Selli:
...
> And
> what I don't like of Karl Aspo's idea is that it takes any instrument of
> policy checking and enforcing out of the monitor, which ends up not
> being able to monitor anything, it becomes just a shell: fire and forget.

I honestly do not understand what I have written which gives you that idé.

> I was also been ironic on Aspo, as many times he can only counter
> another person's ideas asking "What if  cannot be trusted?",
> as if this constitutes a valid argument against being able to set
> policies for the monitor to enforce on the daemons it runs.

If I can get a value from the kernel instead of from the process,
I'd take the kernel value.

Why do a process have to query the kernel to get a value and then 
sending it to a monitor over a communication link; why don't the
monitor query the kernel itself, saving one step.

>  In this
> case, asking "why should a program/daemon care if it has a monitor or
> not ?" is moot, because that the daemon is aware or not that it was
> launched by a monitor and if so by what monitor, does not change
> anything about the monitor's functions and duties.  The daemon doesn't
> care if it's run by a monitor?  The monitor is still there, and it does
> care about the program.

It would help my understanding if you simply answered my question.

> A driver does not care if the traffic light is
> red or green?  The street police does.

This is not as a suitable metaphor as you think it is.

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Allesandro, I'm not interested in a debate (people talking past each 
other), I'm interested in understanding why the proposed library and
it function calls are such a good idé, since I don't think it is.

Wheter to run a monitor or not is a different question, which I 
consider is up to the local admin.

Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in
either camp, end case.

Why srvmgt_droppriv(), either accept that the process set's it itself
or force it to some uid:gid, what is it more to it ?

Why srvmgt_report_state(), just do a normal query to the process and
you'll know if it is ready to serve you or not, if you want to know
if it is up to speed, do a benchmark, if you want to test the 
integrity, do a regression test; what is it more to it ?
(And, yes, a monitoring program can do the above for you.)

I don't get the point of all this, can you explain ?

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Adam Borowski]

On Tue, Apr 18, 2017 at 11:32:00AM +0100, KatolaZ wrote:
> On Tue, Apr 18, 2017 at 12:21:05PM +0200, Alessandro Selli wrote:
> >   Why should a car driver care if the traffic light has the red or the
> > green light turned on?
> >   Why should a sysadmin care if the OS he uses runs systemd or not?
> 
> This is a totally wrong assumption. Being a sysadmin is all about
> setting and implementing *policies*. And the choice between systemd or
> anything else is exactly a matter of policy. So yes, a sysadmin *must*
> care if the OS he/she uses runs systemd or anything else.

I don't care whether the OS runs systemd, sysv-rc, openrc or an enslaved
young boy sitting at the console and starting daemons by hand.  What I do
care about is whether it:
* implements standard interfaces
* is correct while doing so

My beef with systemd is that it has egregious bugs (some due to mistakes,
some due to intentional brain damage), and that working around bugs or
places where it doesn't meet your particular use case is a matter of a
single line of shell on sysv-rc but on systemd it's a hopeless case of
digging through a giant blob of spaghetti code.


A sample openrc (init script?) bug:

[ ok ] Asking all remaining processes to terminate...done.
Currently running processes (pstree):
init-+-2*[nbd-client]
 `-rc---openrc---openrc-run---sendsigs---pstree
[FAIL] Killing all remaining processes...failed.
Stopping NBD client process: 
disconnect, [17892.932316] block nbd0: NBD_DISCONNECT
sock, [17892.937274] block nbd0: shutting down sockets
done
disconnect, [17892.951774] block nbd1: NBD_DISCONNECT
sock, [17892.956734] block nbd1: shutting down sockets
done
rmmod: ERROR: Module nbd is in use
nbd-client.
[warn] not deconfiguring network interfaces: network devices still mounted. ... 
(warning).
[info] Saving the system clock.
[info] Hardware Clock updated to Tue Apr 18 13:18:23 CEST 2017.
[ ok ] Deactivating swap...done.
[] Unmounting local filesystems...[17896.568144] block nbd0: Attempted send 
on invalid socket
[17896.573501] blk_update_request: I/O error, dev nbd0, sector 4257856
[17896.579804] block nbd0: Attempted send on invalid socket
[17896.585125] blk_update_request: I/O error, dev nbd0, sector 4258112
[17896.591402] BTRFS error (device nbd0): bdev /dev/nbd0 errs: wr 1, rd 0, 
flush 0, corrupt 0, gen 0
[17896.600422] block nbd0: Attempted send on invalid socket
[17896.605741] blk_update_request: I/O error, dev nbd0, sector 4262496
[17896.612018] block nbd0: Attempted send on invalid socket

A sysadmin who doesn't know the proper way to fix this can still trivially
deal with the issue.  Don't know how to make nbd-client stop _after_ network
filesystems are unmounted?  Just plop in a manual unmount somewhere.  Here,
your system is fixed, you can then learn when you got a bit of time.


A sample systemd bug:

A common way to backup/search/etc a filesystem that has something mounted in
random subdirectories is to "mount --bind / /mnt/ref/rootfs; mount --bind
/home /mnt/ref/home" and so on, so you don't risk a naive script getting
into a loop.  Except, systemd sometimes decides to convert that --bind into
a --rbind (specifically what you wanted to avoid!); it might happen a
fraction of second after the mount, several minutes or apparently never.  So
you can't even rely on that bug happening or not.  It will also sometimes
decide to _unmount_ what you just mounted (most often on a degraded raid).
Now try to work around this...

-- 
⢀⣴⠾⠻⢶⣦⠀ Meow!
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Collisions shmolisions, let's see them find a collision or second
⠈⠳⣄ preimage for double rot13!
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 12:32, KatolaZ wrote:
> On Tue, Apr 18, 2017 at 12:21:05PM +0200, Alessandro Selli wrote:
>
> [cut]
>
>>>  And why should a program/daemon care if
>>> it has a monitor or not ?
>>   Why should a car driver care if the traffic light has the red or the
>> green light turned on?
>>   Why should a sysadmin care if the OS he uses runs systemd or not?
> This is a totally wrong assumption. Being a sysadmin is all about
> setting and implementing *policies*. And the choice between systemd or
> anything else is exactly a matter of policy. So yes, a sysadmin *must*
> care if the OS he/she uses runs systemd or anything else.

  Which is exactly my point.  I am not advocating in favor of systemd
any more than I'm advocating letting drivers ignore traffic lights.  And
what I don't like of Karl Aspo's idea is that it takes any instrument of
policy checking and enforcing out of the monitor, which ends up not
being able to monitor anything, it becomes just a shell: fire and forget.
I was also been ironic on Aspo, as many times he can only counter
another person's ideas asking "What if  cannot be trusted?",
as if this constitutes a valid argument against being able to set
policies for the monitor to enforce on the daemons it runs.  In this
case, asking "why should a program/daemon care if it has a monitor or
not ?" is moot, because that the daemon is aware or not that it was
launched by a monitor and if so by what monitor, does not change
anything about the monitor's functions and duties.  The daemon doesn't
care if it's run by a monitor?  The monitor is still there, and it does
care about the program.  A driver does not care if the traffic light is
red or green?  The street police does.


Alessandro

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Ron]

On Tue, 18 Apr 2017 11:32:00 +0100
KatolaZ  wrote:

> >   Why should a car driver care if the traffic light has the red or the
> > green light turned on?
> >   Why should a sysadmin care if the OS he uses runs systemd or not?
> >   
> 
> This is a totally wrong assumption. Being a sysadmin is all about
> setting and implementing *policies*. And the choice between systemd or
> anything else is exactly a matter of policy. So yes, a sysadmin *must*
> care if the OS he/she uses runs systemd or anything else.

He should care, as he has to know whether it is a unix/linux system or a 
systemd system, the two being rather different in their management...
 
Cheers,
 
Ron.
-- 
   Eat drink and be merry,
  for tomorrow we may diet.

   -- http://www.olgiati-in-paraguay.org --
 
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[KatolaZ]

On Tue, Apr 18, 2017 at 12:21:05PM +0200, Alessandro Selli wrote:

[cut]

> 
> >  And why should a program/daemon care if
> > it has a monitor or not ?
> 
>   Why should a car driver care if the traffic light has the red or the
> green light turned on?
>   Why should a sysadmin care if the OS he uses runs systemd or not?
> 

This is a totally wrong assumption. Being a sysadmin is all about
setting and implementing *policies*. And the choice between systemd or
anything else is exactly a matter of policy. So yes, a sysadmin *must*
care if the OS he/she uses runs systemd or anything else.

HND

KatolaZ

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[ "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[   @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[ @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]


signature.asc
Description: Digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 18/04/2017 at 10:35, k...@aspodata.se wrote:
> Alessandro Selli:
>> On Fri, 14 Apr 2017 at 12:06:44 +0200 (CEST)
>> k...@aspodata.se wrote:
>>> Enrico Weigelt:
>>> ...
 Let's just take some example: libsrvmgt with funcs like that:

 * srvmgt_daemonize()  
   --> detach from controlling terminal, etc  
>>> Why do any monitor program need to know if the program has detached
>>> or not, the only thing it needs to know is the pid and the state,
>>> which would/could be provided by the *_state() functions,
>>> or am I wrong ?
>>   All daemons detach from the control terminal.
> A daemon from a daemon ?

  Isn't init a daemon?  Isn't inetd/xinetd a daemon?

>>  The monitor and the
>> daemon must coordinate each other about who's doing the detach to avoid
>> launch time failures.
> The program to become the daemon, how does it knows if it has a monitor 
> or not ?

  In the case of init being the monitor, it can check if it's parent has
PID 1.  The classical approach is to configure the daemon to do the
right thing, i.e. to do the detach himself when run with the --daemon
switch or when the config file has a similar item configured, to let the
monitor do it otherwise or if explicitly told not to daemonize (some
programs have a --no-daemon or --foreground or --debug switch to
override the config file settings).  This is old-school Unix, classical
Unix daemon have had these items (switches and config files) for many
ten years...  the modern approach in Linux entails IPC, dbus, sockets to
have messages passed from the monitor to daemons and back.

>  And if it doesn't know, how can one demand that it should
> coordinate with something unknown ?

  Config files, command line switches.

>  And why should a program/daemon care if
> it has a monitor or not ?

  Why should a car driver care if the traffic light has the red or the
green light turned on?
  Why should a sysadmin care if the OS he uses runs systemd or not?

 * srvmgt_droppriv(...)  
   --> drop root privileges (if we are still root)
   --> several versions, eg. with fetching the target uid/gid from env  
>>> Given the pid, it isn't that hard for a monitor to find the uid/gid of 
>>> the process, why has this have to go through the monitor ?
>>   I don't think here the monitor is used to extract process info (such
>> as user/group it's running under), rather the environment is used to
>> determine what user:group the daemon must be set to by the monitor.
> I don't follow you. If the monitor shouldn't know much about the child
> as stated by someone somewhere else in this thread, why should the 
> monitor need to know user:group ?

  Again, as far as I understood what Enrico Weigelt wrote, the monitor
does not want to know the user:group the daemons runs under, it needs to
*set* them to the appropriate values when it launches the daemon to have
them reflect config file settings or local policies.

>  And if it should know it, why trust 
> the monitored program when the kernel can give you the info ?

  Again, the relevant info is supposed to be set in config files.  The
user:group the daemon must run under is not up to the daemon itself,
they are set in config files, either the deamon's or the monitor's.

>>> Also, given that you can do the above without this lib opens gives the
>>> program the option too cheat, say, if the monitor wishes to have a
>>> tight control of the process.
>>   It depends by where is that environment set.  It might be a config
>> file.  You're supposed to trust your config files.
> Or, rather, the user is responsible for the config file, not the program
> reading it, so yes the program generally trusts the config file "since I
> demand it". And what is your point ?

  That the relevant info is set in the config files, they are not made
up by the program itself and trusted by the monitor.  The trust is
placed into the config files.

 * srvmgt_report_state(...)
>>>   --> report the service state to the supervisor  
>>> ...
>>>
>>> There could be something like *_a_would_like_to_be_monitored() and
>>> *_I_wish_to_regain_my_free_will().
>>   The monitor is supposed to implement local security/service
>> continuity/resource allocation policies the daemon must not be able to
>> bail itself out of.
> It seems you have the view that a monitor has to police a program.

  Many monitor also do this, some are setup mainly for this reason. 
Xinetd has several config items related to security or resource
policing: some "flags" (like IDONLY and INTERCEPT), the "user" and
"group" settings, "libwrap", "only_from", "no_access", "access_times",
"redirect", "bind", "per_source", "cps" (connections per second),
"max_load", "umask", "rlimit_(as|cpu|data|rss|stack)", "passenv",
"deny_time".  It's good they are available, that you use them or don't.

> I don't think one should have such a strict view.

  If you don't want them, don't use them.  Let other people decide
otherwise, though.  There is not a single use case that 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Alessandro Selli:
> On Fri, 14 Apr 2017 at 12:06:44 +0200 (CEST)
> k...@aspodata.se wrote:
> > Enrico Weigelt:
> > ...
> > > Let's just take some example: libsrvmgt with funcs like that:
> > > 
> > > * srvmgt_daemonize()  
> > >   --> detach from controlling terminal, etc  
> >
> > Why do any monitor program need to know if the program has detached
> > or not, the only thing it needs to know is the pid and the state,
> > which would/could be provided by the *_state() functions,
> > or am I wrong ?
>   All daemons detach from the control terminal.

A daemon from a daemon ?

>  The monitor and the
> daemon must coordinate each other about who's doing the detach to avoid
> launch time failures.

The program to become the daemon, how does it knows if it has a monitor 
or not ? And if it doesn't know, how can one demand that it should
coordinate with something unknown ? And why should a program/daemon care if
it has a monitor or not ?

> > > * srvmgt_droppriv(...)  
> > >   --> drop root privileges (if we are still root)
> > >   --> several versions, eg. with fetching the target uid/gid from env  
> >
> > Given the pid, it isn't that hard for a monitor to find the uid/gid of 
> > the process, why has this have to go through the monitor ?
> 
>   I don't think here the monitor is used to extract process info (such
> as user/group it's running under), rather the environment is used to
> determine what user:group the daemon must be set to by the monitor.

I don't follow you. If the monitor shouldn't know much about the child
as stated by someone somewhere else in this thread, why should the 
monitor need to know user:group ? And if it should know it, why trust 
the monitored program when the kernel can give you the info ?

> > Also, given that you can do the above without this lib opens gives the
> > program the option too cheat, say, if the monitor wishes to have a
> > tight control of the process.
> 
>   It depends by where is that environment set.  It might be a config
> file.  You're supposed to trust your config files.

Or, rather, the user is responsible for the config file, not the program
reading it, so yes the program generally trusts the config file "since I
demand it". And what is your point ?

> > > * srvmgt_report_state(...)
> >   --> report the service state to the supervisor  
> > ...
> >
> > There could be something like *_a_would_like_to_be_monitored() and
> > *_I_wish_to_regain_my_free_will().
> 
>   The monitor is supposed to implement local security/service
> continuity/resource allocation policies the daemon must not be able to
> bail itself out of.

It seems you have the view that a monitor has to police a program.
I don't think one should have such a strict view.

> > >  --> states could be eg.  
> > >   * SRVMGT_STATE_STARTUP -- still within the startup phase
> > >   * SRVMGT_STATE_READY_LOCAL -- ready for local clients only
> > >   * SRVMGT_STATE_READY_ALL   -- ready for all clients
> > >   * SRVMGT_STATE_BUSY-- too busy to process new requests
> > >   * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
> > >  queued requests
> > >   * SRVMGT_STATE_DEFERRED-- temporarily can't accept new
> > >  requests (eg. overload)
> > >   * SRVMGT_STATE_WAITING -- wait for resource (eg. printer
> > >  needs paper or ink)
> > >   * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
> > >  fatal error)  
> >
> > Given the choises given, it seems that the target of the monitor is
> > network servers. Couldn't the monitor find out the ready_local,
> > ready_all and shutdown by itself by monitoring which ports are open ?
> 
>   The monitor is not supposed to always grant any daemon any port it
> wishes.
...

Why, the monitor is not a firewall. It seems you and I have different
views about what a monitor is and should/shouldn't do.

> > And, what if the monitor cannot trust the program it monitors ?
> > Given the lib suggestion, it seems that we are trusting the program.
> 
>   Well... what if you do not trust the monitor?

Then I wouldn't run it, would you ?

> What do you mean by what you wrote?  How do you think you can teach a
> monitor if some program can be trusted?
...

The monitor doesn't have to trust a specific program if it can get the
the wanted info from the kernel. If I can get the info I want from the 
kernel, why should I ask the program ? If the kernel say something 
and the program something else, which one do you trust more ?

> > But what if if we don't know if we can trust the walues provided by
> > the program, the we have to check what the program is really doing,
> > and then the lib have no value, isn't that so ?
> 
>   Could you please provide us with an example of untrusted
> behaviour/program provided data that the monitor is supposed to detect
> and handle accordingly?

First off, we seems to have 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[KatolaZ]

On Tue, Apr 18, 2017 at 03:36:09AM -0400, Steve Litt wrote:
> On Mon, 17 Apr 2017 10:09:02 +0200
> exha...@posteo.net wrote:
> 
> > in the same way that there's not a perfect graphical desktop for all
> > the people,
> 
> Quoted for truth!
> 
> You can't *download* or *install* the perfect GUI desktop, you must
> construct the GUI desktop that's best for you. And that requires
> interchangeable parts.
> 
> I sense a theme here.
>  


The theme is quite easy to spot, and has been around for the best part
of the last 50 years: small, reusable, interchangeable components,
each providing a single, atomic, simple mechanism, avoiding to mess up
with policies (which are for the perusal of humans, not
machines...). This is what KISS is all about.

We have always needed and will always need more software. But never
"more standard policies" or "unique and perfect solutions" for each
problem. Those are quests for fools.


HND

KatolaZ

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[ "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[   @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[ @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]


signature.asc
Description: Digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Mon, 17 Apr 2017 10:09:02 +0200
exha...@posteo.net wrote:

> in the same way that there's not a perfect graphical desktop for all
> the people,

Quoted for truth!

You can't *download* or *install* the perfect GUI desktop, you must
construct the GUI desktop that's best for you. And that requires
interchangeable parts.

I sense a theme here.
 
SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On 17/04/2017 at 03:57, Steve Litt wrote:

[...]

> There's no need to search for the perfect init or supervisor. Long ago
> we got a bunch of them that are all good enough, and can be combined to
> fill almost any need. This continuing search for a perfect init is just
> wheel spinning, or perhaps an excuse for profit-driven complexification.

  I fully agree.  Than you for having put down such a good manifesto.



-- 
Alessandro Selli 
Tel. 3701355486
VOIP SIP: dhatarat...@ekiga.net
Chiave PGP/GPG key: B7FD89FD

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Alessandro Selli]

On Fri, 14 Apr 2017 at 12:06:44 +0200 (CEST)
k...@aspodata.se wrote:

> Enrico Weigelt:
> ...
> > Let's just take some example: libsrvmgt with funcs like that:
> > 
> > * srvmgt_daemonize()  
> >   --> detach from controlling terminal, etc  
>
> Why do any monitor program need to know if the program has detached
> or not, the only thing it needs to know is the pid and the state,
> which would/could be provided by the *_state() functions,
> or am I wrong ?

  All daemons detach from the control terminal.  The monitor and the
daemon must coordinate each other about who's doing the detach to avoid
launch time failures.

> > * srvmgt_droppriv(...)  
> >   --> drop root privileges (if we are still root)
> >   --> several versions, eg. with fetching the target uid/gid from env  
>
> Given the pid, it isn't that hard for a monitor to find the uid/gid of 
> the process, why has this have to go through the monitor ?

  I don't think here the monitor is used to extract process info (such
as user/group it's running under), rather the environment is used to
determine what user:group the daemon must be set to by the monitor.

> Also, given that you can do the above without this lib opens gives the
> program the option too cheat, say, if the monitor wishes to have a
> tight control of the process.

  It depends by where is that environment set.  It might be a config
file.  You're supposed to trust your config files.

[...]

> > * srvmgt_report_state(...)
>   --> report the service state to the supervisor  
> ...
>
> There could be something like *_a_would_like_to_be_monitored() and
> *_I_wish_to_regain_my_free_will().

  The monitor is supposed to implement local security/service
continuity/resource allocation policies the daemon must not be able to
bail itself out of.

> >  --> states could be eg.  
> > * SRVMGT_STATE_STARTUP -- still within the startup phase
> > * SRVMGT_STATE_READY_LOCAL -- ready for local clients only
> > * SRVMGT_STATE_READY_ALL   -- ready for all clients
> > * SRVMGT_STATE_BUSY-- too busy to process new requests
> > * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
> >  queued requests
> > * SRVMGT_STATE_DEFERRED-- temporarily can't accept new
> >  requests (eg. overload)
> > * SRVMGT_STATE_WAITING -- wait for resource (eg. printer
> >  needs paper or ink)
> > * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
> >  fatal error)  
>
> Given the choises given, it seems that the target of the monitor is
> network servers. Couldn't the monitor find out the ready_local,
> ready_all and shutdown by itself by monitoring which ports are open ?

  The monitor is not supposed to always grant any daemon any port it
wishes.  Monitors are there also to prevent daemons to do just as they
will.  One thing is to trust the monitor, another thing is to trust the
whole collection of deamons it manages.

[...]

> ///
>
> And, what if the monitor cannot trust the program it monitors ?
> Given the lib suggestion, it seems that we are trusting the program.

  Well... what if you do not trust the monitor?
What do you mean by what you wrote?  How do you think you can teach a
monitor if some program can be trusted?  I don't think I will love a
monitor that is not going to start, say, the sshd daemon because it
thinks it cannot be trusted. I must know what it means by this (bad to
system security? to the available resourses?), I want to know the
specific reason it believes sshd is bad and I want this behaviour to be
configurable.

> But what if if we don't know if we can trust the walues provided by
> the program, the we have to check what the program is really doing,
> and then the lib have no value, isn't that so ?

  Could you please provide us with an example of untrusted
behaviour/program provided data that the monitor is supposed to detect
and handle accordingly?


Alessandro
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Simon Hobson]

exha...@posteo.net wrote:

> In my opinion, this is the best that has been written about init or 
> supervisor,

+1

> (I'm absolutely agree with this)
> 
> "There's no need to search for the perfect init or supervisor. Long ago
> we got a bunch of them that are all good enough, and can be combined to
> fill almost any need. This continuing search for a perfect init is just
> wheel spinning, or perhaps an excuse for profit-driven complexification"

Absolutely.
And I confess it's something I fall prey to from time to time - seeking to "do 
it right" which sometimes results in not doing it at all. Lets just say there 
was "some disagreement" between myself and SWMBO as to what "decorate the spare 
bedroom" means - though at the end of it (a year per room, two rooms so far !) 
she's been happy with the results :-)

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Hendrik Boom]

On Mon, Apr 17, 2017 at 12:33:06PM +0200, marc wrote:
> the kernel thesedays
> allows one to delegate the "default-child-collector" function of 
> init to other processes - that is what the container infrastructure uses.

Which seems to be an effective rebuttal against the argument for systemd 
being a universal service supervisor as pid 1.

-- hendrik
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Hendrik Boom]

On Sun, Apr 16, 2017 at 09:57:38PM -0400, Steve Litt wrote:
> On Sun, 16 Apr 2017 19:22:36 -0400
> Hendrik Boom  wrote:
> 
> > On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote:
> > > On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT
> > > consult wrote:  
> 
> > > > By the way: maybe we should write an RFC draft for the whole
> > > > issue ...  
> > > 
> > > Looked for a relevant RFC.  But found only this: 
> > > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html
> > > [RFC] [PATCH] notify init daemon when children are reparented to it
> > > 
> > > But this doesn't seem to be quite what we want, and I can't say I
> > > have enough context to understand it.  
> > 
> > That so-calledRFC seems to be very much in the style of 
> > requiring the init process to be intimately involved in process 
> > supervision actions, which is a strong constraint on what init
> > systems can be chosen.  And, I suspect, an init system that is so
> > involved is a potential tool to gradually take over the entire OS, as
> > systemd has done.
> 
> And, in fact, being written in 2008, that so called rfc might have been
> either the excuse or the inspiration for systemd.
> 
> The "rfc" suffers from a perfectionism related to bikeshedding. Heaven
> forbid there's ever a zombie or a process with a 'T' in the ps status
> field! We simply MUST make a perfect init or supervisor system: Nothing
> less will do. And while we're indulging our perfection, it's important
> to remember that simplicity not be a priority at all. We will remain
> oblivious to the fact that complexity and lack of encapsulation
> creates flaws far worse than the flaws we're moving heaven and earth
> to get rid of. And for gosh sakes, let's forget the facts:
> 
> THE FACTS
> 
> * sysvinit is perfectly workable for the vast majority of users, or at
>   least it was until the systemd people influenced the "upstreams" to
>   build in code to fail with sysvinit.
> 
> * sysvinit plus daemontools is perfectly workable for almost all users
>   who are capable of writing a short run file and creating a symlink.
>   Daemontools suffers from none of the problems hand-wrung by the
>   "rfc", and indeed in a daemontools world (or at least as a runit
>   world which I assume is the same), the only process whose parent is
>   PID1 is runsvdir (equivalent of daemontools svscanboot). On my
>   computer, the executables reparented to PID1 are all stuff spawned by
>   dmenu or UMENU, as well as the gvim executable, which doubleforks
>   itself automatically. More on this later...
> 
> * sysvinit plus OpenRC is perfectly workable for most users who don't
>   want daemon respawning.
> 
> * sysvinit plus OpenRC plus daemontools is perfectly workable for users
>   who want some daemons respawnable.
> 
> * A PID1 consisting of an rc file that somehow respawns the virtual
>   terminals, background-runs any daemons necessary, and then ends in a
>   loop that listens for and handles signals to PID1, is perfectly
>   workable for a person operating a small, special purpose computer. If
>   I knew how to respawn virtual terminals I'd do just that as a
>   demonstration, but respawning gettys is incredibly difficult: It
>   often kills the process that tried to respawn it.
> 
> * The 83 line Suckless Init spawning an rc file is perfectly workable
>   for someone wanting simplicity and the ultimately simple PID1.
> 
> Bottom line, all this bikeshedding perfectionism is unnecessary unless
> you're a big commercial company trying to turn Free Software into a
> cash cow by complexifying it. All the problems were solved long ago, or
> are easily solvable by simple, user space addons needing no
> modification to the likes of sysvinit, daemontools, runit, s6, Epoch,
> OpenRC and the like.
> 
> For instance, I have a real problem with zombies and T status processes
> created by my use of dmenu and UMENU. It wouldn't be particularly
> difficult for me to build a userspace daemontools analog, where one
> process parented to PID1 (my analog of svscanboot) would spin around
> listening for commands and/or scripts to run in such a way that IT was
> the parent, or perhaps via analogs of svscan that don't respawn. I
> could then modify dmenu and UMENU to message my daemon in order to run
> commands. Notice the idea in this paragraph requires not one
> modification to whatever "init system" you're using. And it's simple
> enough that even I could implement it, given the time.
> 
> There's no need to search for the perfect init or supervisor. Long ago
> we got a bunch of them that are all good enough, and can be combined to
> fill almost any need. This continuing search for a perfect init is just
> wheel spinning, or perhaps an excuse for profit-driven complexification.

Thank you for this eloquent and extensive statement of principle.

-- hendrik
___
Dng mailing list
Dng@lists.dyne.org

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Didier Kryn]

Le 14/04/2017 à 10:57, Enrico Weigelt, metux IT consult a écrit :

* srvmgt_report_state(...)
   --> report the service state to the supervisor
   --> states could be eg.
* SRVMGT_STATE_STARTUP -- still within the startup phase
* SRVMGT_STATE_READY_LOCAL -- ready for local clients only
* SRVMGT_STATE_READY_ALL   -- ready for all clients
* SRVMGT_STATE_BUSY-- too busy to process new requests
* SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
   queued requests
* SRVMGT_STATE_DEFERRED-- temporarily can't accept new
   requests (eg. overload)
* SRVMGT_STATE_WAITING -- wait for resource (eg. printer
   needs paper or ink)
* SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
   fatal error)



Reporting state is usefull, but this could be achieved by writing 
the state in plain text to stdout. It is up to the supervisr to 
establish a pipe between itself and the daemon's stdout. If there isn't 
a supervisor, then writing to /dev/null doesn't harm. Or why not write 
to a file in /run? Anyway, I don't think it deserves to link to a 
library to perform such a simple thing.


Didier


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[marc]

> > I wonder why that general task of daemonizing cant just be done in a
> > generic separate program and left out of the individual daemons.
> > So, everybody can decide on this own how to actually start/manage
> > the daemons - some use a supervisor, some just call via a daemonizer
> > program from init scripts, etc, etc.

Only the program itself[*] knows when the transition between it being
started and it being ready has happened. If the program daemonises
itself, it can communicate this by having the parent exit at the
correct point. 

[*]For a generic program to understand this, there would have
to be some sort of convention (close stderr seems to be sanest
I could think of). But no convention has been established yet
and we have had some major boneheaded ideas (child processes 
stops itself when ready (WTF!?), write a magic sequence to
stdout (fugly?)). Hiding this behind some library call makes it 
worse, as it ties things to a particular implementation.

The daemontools advocates take the view that this notification
when ready isn't strictly needed, as the dependent tasks should
be smart enough to retry/persist. They have a point, but it can 
mask the actual source of failures and isn't workable for hard
dependencies (fs not mounted).

> > By the way: maybe we should write an RFC draft for the whole issue ...
> 
> Looked for a relevant RFC.  But found only this: 
> https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html
> [RFC] [PATCH] notify init daemon when children are reparented to it
> 
> But this doesn't seem to be quite what we want, and I can't say I have 
> enough context to understand it.

It isn't what we want. Sadly some of the statements are outdated
or something between misleading or incorrect. For example:

*> *but* due to a quirk of POSIX, if it were to be made to open() a tty
*> device, it would end up owning it!  FAIL.

POSIX anticipates this problem and provides a O_NOCTTY to the open
call to prevent this problem happening. So NO FAIL.

*> If there's two apache2 daemons running (in different chroots, or for
*> different IPs or ports?), it [init] doesn't know which of the two died 
*> because the PID that died is unknown to it.

This isn't the case. With a sigaction hander init can retrieve the pid
and uid of the exited process. Also outdated because the kernel thesedays
allows one to delegate the "default-child-collector" function of 
init to other processes - that is what the container infrastructure uses.

> It seems to follow the practice of creating two processes for a 
> daemon, a parent and a child, and then orphaning the child.  
> This is not what we seem to be discussing here, and leads to 
> init having nontrivial ongoing work.

Hmm... it is a bit of a tradeoff. This approach makes it possible
to indicate readiness and thus order the system startup properly. 
It is also needed when random users need to daemonise their
tasks (eg screen). 

On the other hand it makes automated respawning tricky. However it 
is an open question if well written daemons need to ever respawn. 
Respawning a crashed daemon favours availability over all other
considerations. And uncontrolled respawn is a fork bomb, an oracle
for an attacker, a log file filler, a database corruptor, etc, etc.

> And although it says [RFC] in the title, it doesn't seem to be an 
> official RFC.

Anybody can make a Request For Comments :-) (The RFCs of the IETF 
are different, they don't really relate to OS internals)

Anyway here is my request for comments: If you want to signal that
your daemon is ready close stderr, even if you don't do a fork().

Then the proposed generic wrapper can notice that.

regards

marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[marc]

> OK, I completely understand. It makes sense. It's pretty slick, too. I
> never would have thought of it.
> 
> Being a daemontools enthusiast, I still don't see fork_parent() as an
> asset in late boot. But I have a feeling it, or something similar,
> could be an asset in different contexts. As a matter of fact, in my
> UMENU program, I might change my backgrounding code to incorporate
> that, and perhaps in doing so eliminate some sleep kludges.

Yay. I am glad that the explanation was helpful. 

regards

marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[exhaust]

In my opinion, this is the best that has been written about init or 
supervisor,


(I'm absolutely agree with this)

"There's no need to search for the perfect init or supervisor. Long ago
we got a bunch of them that are all good enough, and can be combined to
fill almost any need. This continuing search for a perfect init is just
wheel spinning, or perhaps an excuse for profit-driven complexification"


the search for the perfect dictator is not the great idea, at least 
that's what I think


in the same way that there's not a perfect graphical desktop for all the 
people, is sure that there's not a perfect init system for all the 
situations in which you can use a computer


the init system must be something that the people can change in an easy 
way, and so, choose between different options for different situations


sometimes I think that the people that is focused in develop software 
quickly, is the people that search for a perfect init system, so they 
can develop fast software


but fast software is like fast food,

I think that freedom is better that fast software


José María Ávila

On 17.04.2017 03:57, Steve Litt wrote:

On Sun, 16 Apr 2017 19:22:36 -0400
Hendrik Boom  wrote:


On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote:
> On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT
> consult wrote:



> > By the way: maybe we should write an RFC draft for the whole
> > issue ...
>
> Looked for a relevant RFC.  But found only this:
> https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html
> [RFC] [PATCH] notify init daemon when children are reparented to it
>
> But this doesn't seem to be quite what we want, and I can't say I
> have enough context to understand it.

That so-calledRFC seems to be very much in the style of
requiring the init process to be intimately involved in process
supervision actions, which is a strong constraint on what init
systems can be chosen.  And, I suspect, an init system that is so
involved is a potential tool to gradually take over the entire OS, as
systemd has done.


And, in fact, being written in 2008, that so called rfc might have been
either the excuse or the inspiration for systemd.

The "rfc" suffers from a perfectionism related to bikeshedding. Heaven
forbid there's ever a zombie or a process with a 'T' in the ps status
field! We simply MUST make a perfect init or supervisor system: Nothing
less will do. And while we're indulging our perfection, it's important
to remember that simplicity not be a priority at all. We will remain
oblivious to the fact that complexity and lack of encapsulation
creates flaws far worse than the flaws we're moving heaven and earth
to get rid of. And for gosh sakes, let's forget the facts:

THE FACTS

* sysvinit is perfectly workable for the vast majority of users, or at
  least it was until the systemd people influenced the "upstreams" to
  build in code to fail with sysvinit.

* sysvinit plus daemontools is perfectly workable for almost all users
  who are capable of writing a short run file and creating a symlink.
  Daemontools suffers from none of the problems hand-wrung by the
  "rfc", and indeed in a daemontools world (or at least as a runit
  world which I assume is the same), the only process whose parent is
  PID1 is runsvdir (equivalent of daemontools svscanboot). On my
  computer, the executables reparented to PID1 are all stuff spawned by
  dmenu or UMENU, as well as the gvim executable, which doubleforks
  itself automatically. More on this later...

* sysvinit plus OpenRC is perfectly workable for most users who don't
  want daemon respawning.

* sysvinit plus OpenRC plus daemontools is perfectly workable for users
  who want some daemons respawnable.

* A PID1 consisting of an rc file that somehow respawns the virtual
  terminals, background-runs any daemons necessary, and then ends in a
  loop that listens for and handles signals to PID1, is perfectly
  workable for a person operating a small, special purpose computer. If
  I knew how to respawn virtual terminals I'd do just that as a
  demonstration, but respawning gettys is incredibly difficult: It
  often kills the process that tried to respawn it.

* The 83 line Suckless Init spawning an rc file is perfectly workable
  for someone wanting simplicity and the ultimately simple PID1.

Bottom line, all this bikeshedding perfectionism is unnecessary unless
you're a big commercial company trying to turn Free Software into a
cash cow by complexifying it. All the problems were solved long ago, or
are easily solvable by simple, user space addons needing no
modification to the likes of sysvinit, daemontools, runit, s6, Epoch,
OpenRC and the like.

For instance, I have a real problem with zombies and T status processes
created by my use of dmenu and UMENU. It wouldn't be particularly
difficult for me to build a userspace daemontools analog, where one
process parented to PID1 (my analog of svscanboot) would 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sun, 16 Apr 2017 19:22:36 -0400
Hendrik Boom  wrote:

> On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote:
> > On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT
> > consult wrote:  

> > > By the way: maybe we should write an RFC draft for the whole
> > > issue ...  
> > 
> > Looked for a relevant RFC.  But found only this: 
> > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html
> > [RFC] [PATCH] notify init daemon when children are reparented to it
> > 
> > But this doesn't seem to be quite what we want, and I can't say I
> > have enough context to understand it.  
> 
> That so-calledRFC seems to be very much in the style of 
> requiring the init process to be intimately involved in process 
> supervision actions, which is a strong constraint on what init
> systems can be chosen.  And, I suspect, an init system that is so
> involved is a potential tool to gradually take over the entire OS, as
> systemd has done.

And, in fact, being written in 2008, that so called rfc might have been
either the excuse or the inspiration for systemd.

The "rfc" suffers from a perfectionism related to bikeshedding. Heaven
forbid there's ever a zombie or a process with a 'T' in the ps status
field! We simply MUST make a perfect init or supervisor system: Nothing
less will do. And while we're indulging our perfection, it's important
to remember that simplicity not be a priority at all. We will remain
oblivious to the fact that complexity and lack of encapsulation
creates flaws far worse than the flaws we're moving heaven and earth
to get rid of. And for gosh sakes, let's forget the facts:

THE FACTS

* sysvinit is perfectly workable for the vast majority of users, or at
  least it was until the systemd people influenced the "upstreams" to
  build in code to fail with sysvinit.

* sysvinit plus daemontools is perfectly workable for almost all users
  who are capable of writing a short run file and creating a symlink.
  Daemontools suffers from none of the problems hand-wrung by the
  "rfc", and indeed in a daemontools world (or at least as a runit
  world which I assume is the same), the only process whose parent is
  PID1 is runsvdir (equivalent of daemontools svscanboot). On my
  computer, the executables reparented to PID1 are all stuff spawned by
  dmenu or UMENU, as well as the gvim executable, which doubleforks
  itself automatically. More on this later...

* sysvinit plus OpenRC is perfectly workable for most users who don't
  want daemon respawning.

* sysvinit plus OpenRC plus daemontools is perfectly workable for users
  who want some daemons respawnable.

* A PID1 consisting of an rc file that somehow respawns the virtual
  terminals, background-runs any daemons necessary, and then ends in a
  loop that listens for and handles signals to PID1, is perfectly
  workable for a person operating a small, special purpose computer. If
  I knew how to respawn virtual terminals I'd do just that as a
  demonstration, but respawning gettys is incredibly difficult: It
  often kills the process that tried to respawn it.

* The 83 line Suckless Init spawning an rc file is perfectly workable
  for someone wanting simplicity and the ultimately simple PID1.

Bottom line, all this bikeshedding perfectionism is unnecessary unless
you're a big commercial company trying to turn Free Software into a
cash cow by complexifying it. All the problems were solved long ago, or
are easily solvable by simple, user space addons needing no
modification to the likes of sysvinit, daemontools, runit, s6, Epoch,
OpenRC and the like.

For instance, I have a real problem with zombies and T status processes
created by my use of dmenu and UMENU. It wouldn't be particularly
difficult for me to build a userspace daemontools analog, where one
process parented to PID1 (my analog of svscanboot) would spin around
listening for commands and/or scripts to run in such a way that IT was
the parent, or perhaps via analogs of svscan that don't respawn. I
could then modify dmenu and UMENU to message my daemon in order to run
commands. Notice the idea in this paragraph requires not one
modification to whatever "init system" you're using. And it's simple
enough that even I could implement it, given the time.

There's no need to search for the perfect init or supervisor. Long ago
we got a bunch of them that are all good enough, and can be combined to
fill almost any need. This continuing search for a perfect init is just
wheel spinning, or perhaps an excuse for profit-driven complexification.
 
SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Hendrik Boom]

On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote:
> On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT consult 
> wrote:
> > On 15.04.2017 19:50, Steve Litt wrote:
> > 
> > > About my characterizations: "Baroque" is a relative thing. What I wrote
> > > was based on "why would you not simply use a process supervisor like
> > > systemd?" If a person has a reason not to use such a supervisor, and in
> > > fact the whole OpenRC init system seems to be based on such objections
> > > to supervisors, then the six system call solution you outline
> > > transitions from "a whole bunch of needless stuff" to "hey, that's a
> > > pretty darn kool solution." So your solution is baroque only so far as
> > > one enjoys using daemontools or similar.
> > 
> > If one doesn't want a supervisor, why not just using something like
> > start-stop-daemon ? IIRC, it should handle that quite well.
> > 
> > I wonder why that general task of daemonizing cant just be done in a
> > generic separate program and left out of the individual daemons.
> > So, everybody can decide on this own how to actually start/manage
> > the daemons - some use a supervisor, some just call via a daemonizer
> > program from init scripts, etc, etc.
> > 
> > By the way: maybe we should write an RFC draft for the whole issue ...
> 
> Looked for a relevant RFC.  But found only this: 
> https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html
> [RFC] [PATCH] notify init daemon when children are reparented to it
> 
> But this doesn't seem to be quite what we want, and I can't say I have 
> enough context to understand it.

That so-calledRFC seems to be very much in the style of 
requiring the init process to be intimately involved in process 
supervision actions, which is a strong constraint on what init systems 
can be chosen.  And, I suspect, an init system that is so involved is 
a potential tool to gradually take over the entire OS, as systemd has 
done.

Of course, even with such an init system, it's possible to use the 
other mechanisms we have been discussing here to enable specific 
daemons to ignore the init system.

-- hendrik

> 
> It seemms to follow the practice of creating two processes for a 
> daemon, a parent and a child, and then orphaning the child.  
> This is not what we seem to be discusison here, and leads to 
> init having nontrivial ongoing work.
> 
> And although it says [RFC] in the title, it doesn't seem to be an 
> official RFC.
> 
> -- hendrik
> 
> > 
> > 
> > --mtx
> > ___
> > Dng mailing list
> > Dng@lists.dyne.org
> > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
> ___
> Dng mailing list
> Dng@lists.dyne.org
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Hendrik Boom]

On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT consult 
wrote:
> On 15.04.2017 19:50, Steve Litt wrote:
> 
> > About my characterizations: "Baroque" is a relative thing. What I wrote
> > was based on "why would you not simply use a process supervisor like
> > systemd?" If a person has a reason not to use such a supervisor, and in
> > fact the whole OpenRC init system seems to be based on such objections
> > to supervisors, then the six system call solution you outline
> > transitions from "a whole bunch of needless stuff" to "hey, that's a
> > pretty darn kool solution." So your solution is baroque only so far as
> > one enjoys using daemontools or similar.
> 
> If one doesn't want a supervisor, why not just using something like
> start-stop-daemon ? IIRC, it should handle that quite well.
> 
> I wonder why that general task of daemonizing cant just be done in a
> generic separate program and left out of the individual daemons.
> So, everybody can decide on this own how to actually start/manage
> the daemons - some use a supervisor, some just call via a daemonizer
> program from init scripts, etc, etc.
> 
> By the way: maybe we should write an RFC draft for the whole issue ...

Looked for a relevant RFC.  But found only this: 
https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html
[RFC] [PATCH] notify init daemon when children are reparented to it

But this doesn't seem to be quite what we want, and I can't say I have 
enough context to understand it.

It seemms to follow the practice of creating two processes for a 
daemon, a parent and a child, and then orphaning the child.  
This is not what we seem to be discusison here, and leads to 
init having nontrivial ongoing work.

And although it says [RFC] in the title, it doesn't seem to be an 
official RFC.

-- hendrik

> 
> 
> --mtx
> ___
> Dng mailing list
> Dng@lists.dyne.org
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Enrico Weigelt:
...
> If one doesn't want a supervisor, why not just using something like
> start-stop-daemon ? IIRC, it should handle that quite well.

Why not just start the program and kill it when not needed anymore ?
You know, you don't have to have a supervisor.

> I wonder why that general task of daemonizing cant just be done in a
> generic separate program and left out of the individual daemons.
> So, everybody can decide on this own how to actually start/manage
> the daemons - some use a supervisor, some just call via a daemonizer
> program from init scripts, etc, etc.
...

http://software.clapper.org/daemonize/daemonize.html

BTW, -f and using daemon() is darn easy to implement.

We did have a kind of supervisor in inetd (mostly to save memory
and reduce load), but it's use has fallen out of favour. So why do
we have to invent new ways of starting and killing programs ?

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 15.04.2017 19:50, Steve Litt wrote:

> About my characterizations: "Baroque" is a relative thing. What I wrote
> was based on "why would you not simply use a process supervisor like
> systemd?" If a person has a reason not to use such a supervisor, and in
> fact the whole OpenRC init system seems to be based on such objections
> to supervisors, then the six system call solution you outline
> transitions from "a whole bunch of needless stuff" to "hey, that's a
> pretty darn kool solution." So your solution is baroque only so far as
> one enjoys using daemontools or similar.

If one doesn't want a supervisor, why not just using something like
start-stop-daemon ? IIRC, it should handle that quite well.

I wonder why that general task of daemonizing cant just be done in a
generic separate program and left out of the individual daemons.
So, everybody can decide on this own how to actually start/manage
the daemons - some use a supervisor, some just call via a daemonizer
program from init scripts, etc, etc.

By the way: maybe we should write an RFC draft for the whole issue ...


--mtx
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sun, 16 Apr 2017 10:01:02 +0200
marc  wrote in response to Steve Litt:


> > And therefore I was wondering
> > why fork_parent() didn't take a function address as an argument, and
> > call that callback function's address where you have the elipses.  
> 
> No function pointers needed. After a fork() system call you have two
> almost identical copies of the same process. The one which sees a
> return code greater than 0 is the parent. The parent instance however
> never makes it out of the fork_parent() library call.
> 
> > Yes. It makes sense. My question was where you put the code
> > represented by your elipses above.   
> 
> The code in the elipses runs outside the fork_parent() call. It could
> run in the main() function or elsewhere.

OK, I completely understand. It makes sense. It's pretty slick, too. I
never would have thought of it.

Being a daemontools enthusiast, I still don't see fork_parent() as an
asset in late boot. But I have a feeling it, or something similar,
could be an asset in different contexts. As a matter of fact, in my
UMENU program, I might change my backgrounding code to incorportate
that, and perhaps in doing so eliminate some sleep kludges.

Thanks,
 
SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sat, 15 Apr 2017 22:50:20 +0200
marc  wrote:

> > Concerning the fork-parent.c and fork-parent.h revealed at
> > http://welz.org.za/notes/fork-parent.tar.gz , I can't envision how
> > you'd bind the fork-parent() function to the stuff your daemon
> > actually does, in order to have your daemon properly fork and then
> > terminate the parent when ready for business.
> > 
> > I look at the code in fork-parent.c, and it looks to me like all the
> > actual tasks of the daemon should be placed immediately above the
> > child's closing p[1], so p[1] is closed when the daemon is ready.
> > But in fact, fork-parent() has no interface to the real work of the
> > daemon, as far as I can see.  
> 
> So fork_parent() can be seen as a minor extension of the
> daemon(3) library call: In both cases after you have called it
> your function is now running in the child process. The parent
> process stays "stuck" in the fork_parent() or daemon() 
> functions. 
> 
> Note that there are "extra" close() calls in the fork_parent()
> code, as after the fork() both ends of the pipe are visible
> in both processes. Without closing the "other" end in each
> process a read would not return EOF and the parent would stay
> up even after the child has closed its end.
> 
> Here pseudocode, commented differently:
> 
>   pipe()  /* done in fork_parent */
>   if(fork() > 0){ /*  in fork_parent */
> close()   /*  in fork_parent, not the stderr of the child
> */ read()/*  in fork_parent */
> exit()/*  in fork_parent */
> /* parent no longer exists beyond this point */
> /* this point in code, not in time */
>   }
> 
>   /* here we are in the child, the parent is still running */
>   /* here do all the complicated setup routes, eg */
>   /* check databases, bind ports */
>   ...

Wait. Are you saying that the way fork_parent() is used is to modify
the source code of fork_parent() itself? If that's the case, I
understand. I was under the impression that fork-parent() was a library
call that you made to properly background your source code, which
exists outside of function fork_parent(). And therefore I was wondering
why fork_parent() didn't take a function address as an argument, and
call that callback function's address where you have the elipses.


>   /* when everything is ready, you do: */
> 
>   close(STDERR_FILENO);

Yes. It makes sense. My question was where you put the code represented
by your elipses above. Is fork_parent() a template (in the general
sense, not the C++ sense) that you modify, or is it a tool you use, as
is, to background code you put in a separate function?

 
SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[marc]

> Concerning the fork-parent.c and fork-parent.h revealed at
> http://welz.org.za/notes/fork-parent.tar.gz , I can't envision how
> you'd bind the fork-parent() function to the stuff your daemon actually
> does, in order to have your daemon properly fork and then terminate the
> parent when ready for business.
> 
> I look at the code in fork-parent.c, and it looks to me like all the
> actual tasks of the daemon should be placed immediately above the
> child's closing p[1], so p[1] is closed when the daemon is ready. But
> in fact, fork-parent() has no interface to the real work of the daemon,
> as far as I can see.

So fork_parent() can be seen as a minor extension of the
daemon(3) library call: In both cases after you have called it
your function is now running in the child process. The parent
process stays "stuck" in the fork_parent() or daemon() 
functions. 

Note that there are "extra" close() calls in the fork_parent()
code, as after the fork() both ends of the pipe are visible
in both processes. Without closing the "other" end in each
process a read would not return EOF and the parent would stay
up even after the child has closed its end.

Here pseudocode, commented differently:

  pipe()  /* done in fork_parent */
  if(fork() > 0){ /*  in fork_parent */
close()   /*  in fork_parent, not the stderr of the child */
read()/*  in fork_parent */
exit()/*  in fork_parent */
/* parent no longer exists beyond this point */
/* this point in code, not in time */
  }

  /* here we are in the child, the parent is still running */
  /* here do all the complicated setup routes, eg */
  /* check databases, bind ports */
  ...
  /* when everything is ready, you do: */

  close(STDERR_FILENO);

  /* at this point the parent exits (see above) */
  /* and the rcS scripts know to start the dependencies */

It turns out that the same basic effect can be achieved with a 
pause() and kill(getppid()), but that makes it tricky to relay 
errors like "-x requires a parameter" to stderr, where usage 
errors should be reported.

regards

marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sat, 15 Apr 2017 08:14:48 +0200
marc  wrote:

> I have written up the details at 
> http://welz.org.za/notes/on-starting-daemons.html

Concerning the fork-parent.c and fork-parent.h revealed at
http://welz.org.za/notes/fork-parent.tar.gz , I can't envision how
you'd bind the fork-parent() function to the stuff your daemon actually
does, in order to have your daemon properly fork and then terminate the
parent when ready for business.

I look at the code in fork-parent.c, and it looks to me like all the
actual tasks of the daemon should be placed immediately above the
child's closing p[1], so p[1] is closed when the daemon is ready. But
in fact, fork-parent() has no interface to the real work of the daemon,
as far as I can see.

So how does one bolt fork-parent() on to his or her daemon code to
enable it with proper backgrounding and timing?

Thanks,

SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sat, 15 Apr 2017 13:50:22 -0400
Steve Litt  wrote:

Oops. Concerning my statement:

> What I
> wrote was based on "why would you not simply use a process supervisor
> like systemd?"

s/systemd/daemontools/

SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sat, 15 Apr 2017 19:12:31 +0200
marc  wrote:

> I appreciate that Steve likes the daemontools approach where the
> server process stays in the foreground, and I have no quarrel with 
> the advice of including a -f foreground option... The daemontools 
> approach makes a completely different set of tradeoffs - some 
> people really like them, others consider them to be the lunatic
> fringe. To each their own.
> 
> But for a classic sysvinit startup path, the above logic
> will do a decent job of signalling that a process is ready to 
> serve requests. And has been an appropriate mechanism for
> decades. No need for yet another API.

I'll admit this: For the person not wanting to use either a
daemontools-inspired process supervisor, nor systemd (which I
understand has the same capability as daemontools to background a
foreground process), the fork-early, parent blocks, kid does the work,
kid messages (and in this case the message is just closing the pipe),
parent unblocks and exits is an excellent dependency tool, assuming all
daemons do it, do it right, and do it accurately.

About my characterizations: "Baroque" is a relative thing. What I wrote
was based on "why would you not simply use a process supervisor like
systemd?" If a person has a reason not to use such a supervisor, and in
fact the whole OpenRC init system seems to be based on such objections
to supervisors, then the six system call solution you outline
transitions from "a whole bunch of needless stuff" to "hey, that's a
pretty darn kool solution." So your solution is baroque only so far as
one enjoys using daemontools or similar.

LOL, as far as subtle, I'm just not seeing subtle in your solution. The
implementation is elegant, but is the solution subtle? Not in my
opinion. Then again, I don't think any software I ever wrote was subtle.

I have some technical questions about your software, but that's for
another email.
 
SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[marc]

Hello again Steve

> > The TLDR is that "A good daemon should fork soon, but have the
> > parent process exit only once the child is ready to service requests"
> > 
> > Sadly it seems too subtle for many people and the concept goes
> > un-noticed ... 
> 
> It's as subtle as a 10 megaton nuclear device, and goes un-noticed
> because most of us who know of it hope it will be forgotten, so we
> don't speak of it.
> 
> Read http://welz.org.za/notes/on-starting-daemons.html. According to
> this document, quite apart from the daemon's real function, the daemon
> must fork itself early, form a pipeline between parent and child, and
> when the child believes it's ready to function completely, it sends a
> message to the parent, who upon receipt of the message exits. That's
> not subtle: It's quite baroque. Your idea of canning it into a neat
> little tool is quite elegant, but it's an elegant implementation of
> something baroque.

Baroque, 10 megaton nuclear device, eh ? It is (error checking/reporting 
elided) an effort of 6 system calls:

  pipe()
  if(fork() > 0){ /* if we are in parent */
close()   /* close the child end of the pipe */
read()/* wait for child */
exit()/* exit the parent process */
  }
  ... /* do whatever is need to bring up the daemon */
  close()/dup2()  /* no message required, Steve misunderstood that */

If these system calls make you hide in a nuclear fallout shelter 
with your powdered white wig, one worries what will happen if somebody 
tells you what gethostbyname() does.

Or even worse have you straced python recently ? It does 800+ system
calls before it even gets going ... and it only goes downhill from 
there. 

I appreciate that Steve likes the daemontools approach where the
server process stays in the foreground, and I have no quarrel with 
the advice of including a -f foreground option... The daemontools 
approach makes a completely different set of tradeoffs - some 
people really like them, others consider them to be the lunatic
fringe. To each their own.

But for a classic sysvinit startup path, the above logic
will do a decent job of signalling that a process is ready to 
serve requests. And has been an appropriate mechanism for
decades. No need for yet another API.

regards

marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 15.04.2017 00:04, Daniel Abrecht wrote:

> I still think this isn't a problem the service manager should attempt to
> solve. This is a situation where the database is temporary unavailable,
> for which there are many possible reasons. The services which relay on
> the database should be able to handle such a situation gracefully, or
> they are just not yet stable enough.

In theory, you're right. But the actual practise tends to be very
different. Probably, databases aren't the prime usecase for that.

There could be other scenarios, where a service has to do some
preparations for others to work (maybe certain filesystems mounted,
etc). I remember, eg., problems w/ nfs, when one of the dozens servers
isn't up yet, so they needed to be started in specific order. Years
went by since I last touched that, no idea that's still the case.


--mtx

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 15.04.2017 03:17, Steve Litt wrote:

> Sounds like a good idea, but it's not, for the following reasons:
> 
> 1) There will be endless arguments about HOW each and every daemon will
>let its init system know "I'm now ready for business". The number of
>different ways will make the number of incompatible Unixes of 1985
>seem trivial to reconcile.

Exactly the reason why I'd put it an separate library - so it can be
easily replaced (even on binary level), w/o ever having to touch the
application itself. It will be the operator's choice, which one shall
be used.

The whole problem arised w/ upstreams wanting such an mechanism,
and nobody there who gagged up Lennart quick enough ...

So, what do we make out of that now ?
a) patch behind all the upstreams all that thoughless upstreams
   for all eternity
b) maintain an dummy drop-in replacement (and keep up w/ everyting that
   the Lennartists come up) for all eternity
c) provide a simple and stable API for exactly the root problem,
   (and convince upstreams to use that instead of Lennartware) and
   everybody can go its own way in piece.

> 1.5) The resource daemon's opinion of being "ready for business" might
>  be erroneous, or not reflect how the user daemon will use the
>  resource daemon.

Such things must be defined (on per service level) anyways - we'll
never get around that. All we can do is try to standardize certain
scenarios. For start, I'd define "READY" as fully operational.
If some daemon provides several services, which be up/down independently
(which I'd consider bad design), we could introduce sub-services
(eg. apache might serve http as well as ftp) - would just add another
const char* parameter.

> 2) The connected and powerful will unilaterally declare one method of
>"phoning home" is the one true way, thereby driving compatability
>wedges between various software, and facilitating the defacto
>banning of some daemons from a distro.

That something we do not want. Actually the (de)potterization is one
aspect of that.

> 3) If the "one true way" is complex, for instance, if it has to
>intimately mesh with dbus (sy what?), small daemon projects
>will be ostricized out of the community.

There doesn't need to be "one true way". It's only for a specific range
of scenario: let the daemon just shout out it's overall status (what's
required for startup ordering, and similar stuff, eg. some *trivial*
monitoring) to anybody who's interested in it - the exact definition
of that "anybody" would be confined to the specific implementation of
the library - the application has no idea about that, and shouldn't
even care whether somebody's listening.

So, again, in the trivial case that would just be an dummy.
If the library also maintains pidfiles (which I'd recommend, so we now
have that stuff confined in one place), the lib of course has to
implement that. But whether anybody actually reacts on the service
notifies should be completely optional.

As an application developer, I can just call that library, and dont have
to care what happens then (okay, I'll have to consider that it might a
few cycles in there).

> 4) The problem this backward communication purports to solve was solved
>around the time the Western Roman Empire was conquored by the
>Barbarians.

You mean, back then when the fascists just killed everybody who didn't
submit to their rule and destroyed freedom (we still suffer from the
roman law) ? ...

>Regardless of your init system, run your daemons from
>daemontools, and have your daemontools run scripts refuse to start
>unless a test of the needed resource indicates it's working.
>Whatever init system you're using, it's easy to get Daemontools, or
>preferably Daemontools-Encore, running.[1]

How exactly does it get that notification from the application ?
Or does it go the other way round, having - per application - monitoring
programs, that try to figure out whether the service is up and signals
that to the rest ?

Again: application developers want an interface for sending such
notifies to the init system / service monitor. Whether you, or me, or
anybody else, is actually using that, isn't in their scope - they just
want that interface, therefore they're using the only thing they know,
which happens to be Lenntart's proprietary one. So, let's just give them
what they want, in a way that doesn't cause us any pain.

> A word about the source of this thread: A guy, whom I /dev/nulled over a
> year ago for making very disturbing political posts in response to
> technical discussions, magically woke up a few days ago and started
> resurrecting all sorts of dead threads. 

You're not talking about me, don't you ?
(you wouldn't reply my mails then, right ?)

> My /dev/null log tells me he's
> sent 20 or 30 emails. I don't begin to know what causes this person to
> act this way, but he's not a friend of our community, and we might want
> to think twice about 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 15.04.2017 11:42, Steve Litt wrote:

> Once upon a time, daemon self-backgrounding was necessary, and this

nohup (or something similar) was not sufficient ?

I don't know when start-stop-daemon was incarnated (related to
daemontools ?), but IMHO it seems to handle the daemonizing quite well.

IMHO, a daemon should not daemonize itself, but always leave that to
the some supervisor or startup tool. And if it can handle one connection
per process, it should even leave socket listening to something
inted-alike. Even, for multi-connection clients, opening the listener
socket should be left to it. (people can still optionally provide
wrappers that glue everything together to an completely-standalone
server machinery, for convenience).


--mtx
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Sat, 15 Apr 2017 08:14:48 +0200
marc  wrote:

> > > Go on down your path, but I suspect not many people would cheer
> > > at you in this camp...  
> > 
> > I can see the merit - on two points, technical and political.
> >   
> > > On the technical side, I can see the usefulness of a system  
> > wide standardised service status reporting - making it easy
> > for one process to see if a service it depends on is actually
> > running and ready (as opposed to, "has started"). I have a
> > customer system I've inherited where it regularly fails to
> > startup properly because Asterisk starts before MySQL has
> > finished starting up.  
> 
> So it turns out that there exists an subtle yet elegant mechanism
> for a process to report that it is ready. It has been in use 
> for decades - daemons as old as sysklog from the previous century 
> use it. 
> 
> I have written up the details at 
> http://welz.org.za/notes/on-starting-daemons.html

Danger Will Robinson!

If you write a daemon that backgrounds itself, whether early or late,
whether parent and child communicate via signals or pipes or, well,
what the heck, network sockets, be ABSOLUTELY sure to include a command
line option to tell your daemon NOT to background itself, but to stay
in the foreground all the time. If your daemon cannot stay in the
foreground, your daemon is incompatible with runit, s6, perp and
daemontools. And this foreground option should be separate from any
"debugging" options, because nobody wants production logs to have
debugging output.

> 
> The TLDR is that "A good daemon should fork soon, but have the
> parent process exit only once the child is ready to service requests"
> 
> Sadly it seems too subtle for many people and the concept goes
> un-noticed ... 

It's as subtle as a 10 megaton nuclear device, and goes un-noticed
because most of us who know of it hope it will be forgotten, so we
don't speak of it.

Read http://welz.org.za/notes/on-starting-daemons.html. According to
this document, quite apart from the daemon's real function, the daemon
must fork itself early, form a pipeline between parent and child, and
when the child believes it's ready to function completely, it sends a
message to the parent, who upon receipt of the message exits. That's
not subtle: It's quite baroque. Your idea of canning it into a neat
little tool is quite elegant, but it's an elegant implementation of
something baroque.

It's also not a good solution. It won't work unless:

1) ALL daemons act this way
2) All daemons accurately time parent destruction to full readiness
3) Nobody uses daemontools, runit, s6, perp, etc.

Once upon a time, daemon self-backgrounding was necessary, and this
fork-early, parent terminate on readiness was state of the art and the
Cadillac of the industry. But then, in 2001, daemontools was invented,
making self-backgrounding unnecessary. From 2001 on, a daemon author
could write a daemon as a normal foreground program, confident that
daemontools, or one of its descendents (runit, s6, perp,
daemontools-encore) would handle all aspects of getting it into the
background while still supervising it, without the need of a PID file.

Because daemontools, daemontools-encore, runit, s6 (and I would guess)
perp can run as mere process supervisors on any arbitrary init system,
the daemontools method of supervising foreground daemons is available
to anyone willing to install daemontools, and installing daemontools is
trivial.

Anyway, by far the most important thing I said in this email is this:
If you insist on making your daemon self-background, for gosh sakes
give it a command line option to stay in the foreground, and make that
command line option quite apart from any option that increases
debugging output.

SteveT

Steve Litt 
April 2017 featured book: Troubleshooting Techniques
 of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[marc]

> > Go on down your path, but I suspect not many people would cheer at you
> > in this camp...
> 
> I can see the merit - on two points, technical and political.
> 
> > On the technical side, I can see the usefulness of a system
> wide standardised service status reporting - making it easy
> for one process to see if a service it depends on is actually
> running and ready (as opposed to, "has started"). I have a
> customer system I've inherited where it regularly fails to
> startup properly because Asterisk starts before MySQL has
> finished starting up.

So it turns out that there exists an subtle yet elegant mechanism
for a process to report that it is ready. It has been in use 
for decades - daemons as old as sysklog from the previous century 
use it. 

I have written up the details at 
http://welz.org.za/notes/on-starting-daemons.html

The TLDR is that "A good daemon should fork soon, but have the
parent process exit only once the child is ready to service requests"

Sadly it seems too subtle for many people and the concept goes
un-noticed ... 

regards

marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Steve Litt]

On Fri, 14 Apr 2017 21:16:18 +0100
Simon Hobson  wrote:

> "Enrico Weigelt, metux IT consult"  wrote:
> 
> >> For those of us who put consistency above boot speed, simply
> >> changing the init script so MySQL doesn't flag as "started" until
> >> the daemon is up and ready to accept requests would fix it;  
> > 
> > But then you'll have kind of daemon who watches mysql until it's
> > really ready and then signals back to the service monitor, so it
> > can proceed with the other services. In the long run, sounds like a
> > maintenance hell to me.  
> 
> Which is why I said I can see some value in a simple
> communication/notification system. Ideally you'd have MySQL itself
> make status calls along the lines of "I'm starting but not ready" and
> hopefully end with "I'm now ready for business". 

Sounds like a good idea, but it's not, for the following reasons:

1) There will be endless arguments about HOW each and every daemon will
   let its init system know "I'm now ready for business". The number of
   different ways will make the number of incompatible Unixes of 1985
   seem trivial to reconcile.

1.5) The resource daemon's opinion of being "ready for business" might
 be erroneous, or not reflect how the user daemon will use the
 resource daemon.

2) The connected and powerful will unilaterally declare one method of
   "phoning home" is the one true way, thereby driving compatability
   wedges between various software, and facilitating the defacto
   banning of some daemons from a distro.

3) If the "one true way" is complex, for instance, if it has to
   intimately mesh with dbus (sy what?), small daemon projects
   will be ostricized out of the community.

4) The problem this backward communication purports to solve was solved
   around the time the Western Roman Empire was conquored by the
   Barbarians. Regardless of your init system, run your daemons from
   daemontools, and have your daemontools run scripts refuse to start
   unless a test of the needed resource indicates it's working.
   Whatever init system you're using, it's easy to get Daemontools, or
   preferably Daemontools-Encore, running.[1]

A word about the source of this thread: A guy, whom I /dev/nulled over a
year ago for making very disturbing political posts in response to
technical discussions, magically woke up a few days ago and started
resurrecting all sorts of dead threads. My /dev/null log tells me he's
sent 20 or 30 emails. I don't begin to know what causes this person to
act this way, but he's not a friend of our community, and we might want
to think twice about responding to him, even in the few cases where he
has a good idea.

[1] To run daemontools-encore, download the lastest tarball, untar it,
modify its conf-* files to reflect correct directories and compile
commands (usually unnecessary if you're using gcc on Linux), make,
become root, make install, make the repository for daemontools
daemons (I used /etc/daemontools/service), make the symlink
directory (I used /var/daemontools/service), then edit the
svscanboot file to reflect the actual directories that are being
used, run svscanboot, and then make run files under the repository
and symlink them to the symlink directory to install daemons. This
way, no matter what init system you're using, you have a complete
supervisor for daemons you believe need to be supervised. And pay
no attention to the guy on Debian-User who tells you not to use the
svscanboot file to instantiate your daemontools system.[2]

[2] Recently a guy posted on Debian-User that you shouldn't use the
svscanboot shellscript to start up daemontools. He states a few
edge-case situations where svscanboot has bugs, most of which have
nothing to do with Linux or normal Intel/AMD hardware. He links to
a few people who have run daemontools by doing various init-foo
instead of just spawning svscanboot. He just loves to hear himself
hold forth on all things technical, but what he forgets to tell
his audience is:

1) svscanboot is by far the simplest way to run daemontools.

2) djb created svscanboot, and although this guy holding forth is
   very smart (IMHO in a Poettering sort of way), if you're going
   to criticize djb on technical grounds, your name had better be
   Einstein or Hawking.

3) Differently named copies of svscanboot, each with its own
   directories, can implement some degree of startup order in
   daemontools, even though daemontools is widely regarded as
   having indeterminate start order.

4) If one were to run into one of those edge cases where a
   daemontools system has bugs if started with svscanboot, they
   might be fixable by changing the svscanboot shellscript, and
   that would be infinitely preferable than all sorts of init-foo.

 
SteveT

Steve Litt 
April 2017 featured book: 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Daniel Abrecht]

On 2017-04-14 20:16, Simon Hobson wrote:
> "Enrico Weigelt, metux IT consult"  wrote:
>
>>> For those of us who put consistency above boot speed, simply changing
>>> the init script so MySQL doesn't flag as "started" until the daemon
>>> is up and ready to accept requests would fix it;
>> But then you'll have kind of daemon who watches mysql until it's really
>> ready and then signals back to the service monitor, so it can proceed
>> with the other services. In the long run, sounds like a maintenance
>> hell to me.
> Which is why I said I can see some value in a simple 
> communication/notification system. Ideally you'd have MySQL itself make 
> status calls along the lines of "I'm starting but not ready" and hopefully 
> end with "I'm now ready for business". The latter would then allow other 
> processes that depend on a working database to continue.
> At present, IIRC the script starts the MySQL process, then loops round 
> waiting for  to appear - at which point it assumes MySQL is ready 
> for action and prints OK on the console.
>
>
> That way, there's a standard way for processes to communicate their status, 
> and a simple and standard way for dependent processes to determine if 
> something they need is ready yet.

I still think this isn't a problem the service manager should attempt to
solve. This is a situation where the database is temporary unavailable,
for which there are many possible reasons. The services which relay on
the database should be able to handle such a situation gracefully, or
they are just not yet stable enough.
I would expect a service which has to handle a task for which it needs
the database to return an error in this situation, but to continue
working. Once the database is available, and a new task arrives which
requires the database, the service should work as expected. If such a
service needs to do some initialization using the database, it can just
re-queue that until the first successful database connection. This way,
there isn't even a need to start the database first, the service would
be rock solid, and there wouldn't be any need for the service manager to
handling such edge cases. Just keep it simple.




signature.asc
Description: OpenPGP digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Simon Hobson]

"Enrico Weigelt, metux IT consult"  wrote:

>> For those of us who put consistency above boot speed, simply changing
>> the init script so MySQL doesn't flag as "started" until the daemon
>> is up and ready to accept requests would fix it;
> 
> But then you'll have kind of daemon who watches mysql until it's really
> ready and then signals back to the service monitor, so it can proceed
> with the other services. In the long run, sounds like a maintenance
> hell to me.

Which is why I said I can see some value in a simple communication/notification 
system. Ideally you'd have MySQL itself make status calls along the lines of 
"I'm starting but not ready" and hopefully end with "I'm now ready for 
business". The latter would then allow other processes that depend on a working 
database to continue.
At present, IIRC the script starts the MySQL process, then loops round waiting 
for  to appear - at which point it assumes MySQL is ready for action 
and prints OK on the console.


That way, there's a standard way for processes to communicate their status, and 
a simple and standard way for dependent processes to determine if something 
they need is ready yet.

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Enrico Weigelt:
> On 14.04.2017 12:06, k...@aspodata.se wrote:
> > Enrico Weigelt:
> > ...
> >> Let's just take some example: libsrvmgt with funcs like that:
> >>
> >> * srvmgt_daemonize()
> >>   --> detach from controlling terminal, etc
> > 
> > Why do any monitor program need to know if the program has detached
> > or not, the only thing it needs to know is the pid and the state,
> > which would/could be provided by the *_state() functions,
> > or am I wrong ?
> 
> That function should also do the daemonizing, so it doesn't need to do
> be implemented individually anymore.

What you are requesting is that developers should learn a new api for 
the same end as the old, I won't support that. What is wrong with 
daemon(3) ?

> BTW: I'd add pidfile handling here, so it's just done once and for all.
> Reduces repeated code between individual packages and gives dists a
> central hook to do whatever tweaks they might wanna do. Might also come
> handy when somebody wants to have per-user services.

I'd say that the individual program has no need nor any interest of any 
pid file, it will be running just as fine without it.

A pid file is only in the interest of a monitoring program, so why not 
let the monitor create it for it's own use, why clutter all idividual 
programs with it.

And, you didn't answer the question, is there anything besides the pid and 
the state that a monitoring program needs to know ?

> >> * srvmgt_droppriv(...)
> >>   --> drop root privileges (if we are still root)
> >>   --> several versions, eg. with fetching the target uid/gid from env
> > 
> > Given the pid, it isn't that hard for a monitor to find the uid/gid of 
> > the process, why has this have to go through the monitor ?
> 
> As above. It's not just notifying, it's doing that. If somebody wants
> an explicit notification for whatever reason (maybe just as simple as
> a syslog message), he can do it there, instead of within the
> individal application.

If you "do it there", you are actually doing it within the individual 
program, and if you are so fond of replacing standard libc calls, why 
don't you go the fakeroot route:

 fakeroot  works by replacing the file manipulation library functions
 (chmod(2), stat(2) etc.) by ones that simulate the effect the real library
 functions would have had, had the user really been root. These wrapper
 functions are in a shared library /usr/lib/libfakeroot.so* which is loaded
 through the LD_PRELOAD mechanism of the dynamic loader. (See ld.so(8))

Then you won't need to change any program, your monitor just provides a
libc wrapper containing the wrappers for whatever functions it fancy
monitoring. Why do I have to modify "my" source code just to please a
monitor ?

...
> > Given the choises given, it seems that the target of the monitor is
> > network servers. Couldn't the monitor find out the ready_local,
> > ready_all and shutdown by itself by monitoring which ports are open ?
> 
> That's not easy - the monitor would need a lot of application specific
> handling. For example sendmail doesn't close the inbound socket if it
> just cant accept new mails temporarily - instead it just tells that
> within the smtp handshake.

Well there you have your service notification, it is already 
implemented. Why don't you use it ?
And if you really care about the mail server, you look in its logs.

And, in parenting a child, you always need to know something about 
the child.

> My goal here is to have an simple notification for simple scenarios,
> where you don't have a full-blown network monitoring (eg. nagios).
> The whole thing could also be useful for per-user services that might
> not even be accessible from the outside.

So what are thoose scenarios, if you really want monitoring you could 
use simple setups of mon or nagios. At the other end is: just start the 
program, if it dies, you investigate and complain to upstream.

> > What's the difference between busy and deferred, seems to be the same 
> > thing.
> 
> A little bit: busy means waiting for some external resource (eg.
> printer ran out of paper), while deferred is due some load limit
> (too high load, disk full, ...). Maybe we'd have to find better names.
> 
> > And, what if the monitor cannot trust the program it monitors ?
> 
> It's meant to be trusted (regarding security, etc) - it's just for
> easier notification, so that user/operator has at least an idea what's
> going on, and allows the monitor to defer startup of other depending
> services, etc.

So, this is only about staggered startup at root ?
Why don't other programs just sleep() and retry till the problem is 
gone. You have the same problem if your db is on one box, you dns 
server is on another, etc. Or do you propose that the monitor should 
run multihost ?

> > But what if if we don't know if we can trust the walues provided by
> > the program, the we have to check what the program is really doing,
> > and then the lib have no value, isn't that so ?
> 
> That would be 

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Daniel Abrecht]

Hi

From my point of view, systemd always tries to keep services running, no
matter how hard they fail, and to mask possible problems when starting a
service, so the service maintainers don't have to fix their service,
which is really unfortunate.

In case of those service state notifications with sd_notify, I think
they are usually used to signal when a service is starting, but not
ready yet. This may seam reasonable at the beginning, but I think it
fixes the problem at the wrong place; When a service needs another
service, but it's temporary unavailable, it should cause an error or
warning to be returned and logged, but it should never be a fatal error
which causes the service to stop. When a service needs to handle this
situation on it's own, it will become more stable over time. If this
task is taken away from it, it will become fragile, and relay on the
service manager. I think this would be really dangerous, because it
could lead to cascades of failing services once one service fails. At
some point, a service manager couldn't possibly handle that anymore and
there would just be waves of restarting and failing processes.

Daniel Abrecht




signature.asc
Description: OpenPGP digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 14.04.2017 12:17, Simon Hobson wrote:

> For those of us who put consistency above boot speed, simply changing
> the init script so MySQL doesn't flag as "started" until the daemon
> is up and ready to accept requests would fix it;

But then you'll have kind of daemon who watches mysql until it's really
ready and then signals back to the service monitor, so it can proceed
with the other services. In the long run, sounds like a maintenance
hell to me.

Having an simple status reporting, an application like mysql could at
least tell "hey, I'm not ready yet - give a coffee break". Whether to
actually use it, should be the decision of the operator (and distro
should give a proper default)


--mtx

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 14.04.2017 12:29, Simon Hobson wrote:

> But I also recognise that others aren't happy and if they want to 
> use something else then that's OK by me - as long as what they
> propose isn't something that "infects" stuff I want to run under
> SysVInit.

Exactly my goal. sysvinit users would probably just have the minimal
version installed, that doesn't do more than a tiny bit syslog()'ing,
others may install more complex versions fitting the service monitor
of their choice.

--mtx

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Enrico Weigelt, metux IT consult]

On 14.04.2017 12:06, k...@aspodata.se wrote:
> Enrico Weigelt:
> ...
>> Let's just take some example: libsrvmgt with funcs like that:
>>
>> * srvmgt_daemonize()
>>   --> detach from controlling terminal, etc
> 
> Why do any monitor program need to know if the program has detached
> or not, the only thing it needs to know is the pid and the state,
> which would/could be provided by the *_state() functions,
> or am I wrong ?

That function should also do the daemonizing, so it doesn't need to do
be implemented individually anymore.

BTW: I'd add pidfile handling here, so it's just done once and for all.
Reduces repeated code between individual packages and gives dists a
central hook to do whatever tweaks they might wanna do. Might also come
handy when somebody wants to have per-user services.

>> * srvmgt_droppriv(...)
>>   --> drop root privileges (if we are still root)
>>   --> several versions, eg. with fetching the target uid/gid from env
> 
> Given the pid, it isn't that hard for a monitor to find the uid/gid of 
> the process, why has this have to go through the monitor ?

As above. It's not just notifying, it's doing that. If somebody wants
an explicit notification for whatever reason (maybe just as simple as
a syslog message), he can do it there, instead of within the
individal application.

>>  --> states could be eg.
>>  * SRVMGT_STATE_STARTUP -- still within the startup phase
>>  * SRVMGT_STATE_READY_LOCAL -- ready for local clients only
>>  * SRVMGT_STATE_READY_ALL   -- ready for all clients
>>  * SRVMGT_STATE_BUSY-- too busy to process new requests
>>  * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
>>  queued requests
>>  * SRVMGT_STATE_DEFERRED-- temporarily can't accept new
>>  requests (eg. overload)
>>  * SRVMGT_STATE_WAITING -- wait for resource (eg. printer
>>  needs paper or ink)
>>  * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
>>  fatal error)
> 
> Given the choises given, it seems that the target of the monitor is
> network servers. Couldn't the monitor find out the ready_local,
> ready_all and shutdown by itself by monitoring which ports are open ?

That's not easy - the monitor would need a lot of application specific
handling. For example sendmail doesn't close the inbound socket if it
just cant accept new mails temporarily - instead it just tells that
within the smtp handshake.

My goal here is to have an simple notification for simple scenarios,
where you don't have a full-blown network monitoring (eg. nagios).
The whole thing could also be useful for per-user services that might
not even be accessible from the outside.

> What's the difference between busy and deferred, seems to be the same 
> thing.

A little bit: busy means waiting for some external resource (eg.
printer ran out of paper), while deferred is due some load limit
(too high load, disk full, ...). Maybe we'd have to find better names.

> And, what if the monitor cannot trust the program it monitors ?

It's meant to be trusted (regarding security, etc) - it's just for
easier notification, so that user/operator has at least an idea what's
going on, and allows the monitor to defer startup of other depending
services, etc.

> But what if if we don't know if we can trust the walues provided by
> the program, the we have to check what the program is really doing,
> and then the lib have no value, isn't that so ?

That would be scenarios where any kind of notification to the service
monitor would be useless. My point is that there are indeed some
scenarios where notification can be helpful, and that's only what that
lib is for.


--mtx

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[Simon Hobson]

KatolaZ  wrote:

>> For start, we'd just write a small library, that logs to syslog,
>> perhaps maintains some pidfiles (maybe even a *compile-time* option
>> to route directly to libsystemd), then patch up packages that currently
>> use libsystemd to use our new one.
>> 
> 
> I personally don't see why one would like to redo libsystemd0 from
> scratch, as you seem so kee of doing. 
> 
> Go on down your path, but I suspect not many people would cheer at you
> in this camp...

I can see the merit - on two points, technical and political.

On the technical side, I can see the usefulness of a system wide standardised 
service status reporting - making it easy for one process to see if a service 
it depends on is actually running and ready (as opposed to, "has started"). I 
have a customer system I've inherited where it regularly fails to startup 
properly because Asterisk starts before MySQL has finished starting up.
For those of us who put consistency above boot speed, simply changing the init 
script so MySQL doesn't flag as "started" until the daemon is up and ready to 
accept requests would fix it; I can see how many would love to be able to have 
each init script just start and do a "wait for service X to be ready" step, so 
that init could just fire up all the scripts in parallel and they'd start as 
each dependency reached the available state.

On the political side, when the Poetteristas say how horrible startup scripts 
are, and how they don't support parallel/dependency startup - there's be the 
response that "actually, you don't need all that furball of crap called SystemD 
because this simple, clean, minimally interlinked, API does it".

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[karl]

Enrico Weigelt:
...
> Let's just take some example: libsrvmgt with funcs like that:
> 
> * srvmgt_daemonize()
>   --> detach from controlling terminal, etc

Why do any monitor program need to know if the program has detached
or not, the only thing it needs to know is the pid and the state,
which would/could be provided by the *_state() functions,
or am I wrong ?

> * srvmgt_droppriv(...)
>   --> drop root privileges (if we are still root)
>   --> several versions, eg. with fetching the target uid/gid from env

Given the pid, it isn't that hard for a monitor to find the uid/gid of 
the process, why has this have to go through the monitor ?

Also, given that you can do the above without this lib opens gives the
program the option too cheat, say, if the monitor wishes to have a
tight control of the process.

A reliable (well, we have to trust the kernel) way to get the pid
and uid of a sender is through signals, are there any other ways ?

> * srvmgt_report_state(...)
  --> report the service state to the supervisor
...

There could be something like *_a_would_like_to_be_monitored() and
*_I_wish_to_regain_my_free_will().

>  --> states could be eg.
>   * SRVMGT_STATE_STARTUP -- still within the startup phase
>   * SRVMGT_STATE_READY_LOCAL -- ready for local clients only
>   * SRVMGT_STATE_READY_ALL   -- ready for all clients
>   * SRVMGT_STATE_BUSY-- too busy to process new requests
>   * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
>  queued requests
>   * SRVMGT_STATE_DEFERRED-- temporarily can't accept new
>  requests (eg. overload)
>   * SRVMGT_STATE_WAITING -- wait for resource (eg. printer
>  needs paper or ink)
>   * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
>  fatal error)

Given the choises given, it seems that the target of the monitor is
network servers. Couldn't the monitor find out the ready_local,
ready_all and shutdown by itself by monitoring which ports are open ?

What's the difference between busy and deferred, seems to be the same 
thing.

///

And, what if the monitor cannot trust the program it monitors ?

Given the lib suggestion, it seems that we are trusting the program.
But what if if we don't know if we can trust the walues provided by
the program, the we have to check what the program is really doing,
and then the lib have no value, isn't that so ?

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[info at smallinnovations dot nl]

On 14-04-17 11:20, KatolaZ wrote:

On Fri, Apr 14, 2017 at 10:57:01AM +0200, Enrico Weigelt, metux IT consult 
wrote:

[cut]


* srvmgt_daemonize()
   --> detach from controlling terminal, etc
* srvmgt_droppriv(...)
   --> drop root privileges (if we are still root)
   --> several versions, eg. with fetching the target uid/gid from env
* srvmgt_report_state(...)
   --> report the service state to the supervisor
   --> states could be eg.
* SRVMGT_STATE_STARTUP -- still within the startup phase
* SRVMGT_STATE_READY_LOCAL -- ready for local clients only
* SRVMGT_STATE_READY_ALL   -- ready for all clients
* SRVMGT_STATE_BUSY-- too busy to process new requests
* SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
   queued requests
* SRVMGT_STATE_DEFERRED-- temporarily can't accept new
   requests (eg. overload)
* SRVMGT_STATE_WAITING -- wait for resource (eg. printer
   needs paper or ink)
* SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
   fatal error)

For start, we'd just write a small library, that logs to syslog,
perhaps maintains some pidfiles (maybe even a *compile-time* option
to route directly to libsystemd), then patch up packages that currently
use libsystemd to use our new one.


I personally don't see why one would like to redo libsystemd0 from
scratch, as you seem so kee of doing.

Go on down your path, but I suspect not many people would cheer at you
in this camp...

HND

KatolaZ
I am too do not see a reason to redo libsystemd0 but we can think about 
maintaining a Devuan version of libsystemd0. To ease the maintenance of 
packages depending on those basic api requests.
We simply cannot trust the debian version but a Devuan version should be 
trustworthy.


Grtz,

Nick
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]

[KatolaZ]

On Fri, Apr 14, 2017 at 10:57:01AM +0200, Enrico Weigelt, metux IT consult 
wrote:

[cut]

> * srvmgt_daemonize()
>   --> detach from controlling terminal, etc
> * srvmgt_droppriv(...)
>   --> drop root privileges (if we are still root)
>   --> several versions, eg. with fetching the target uid/gid from env
> * srvmgt_report_state(...)
>   --> report the service state to the supervisor
>   --> states could be eg.
>   * SRVMGT_STATE_STARTUP -- still within the startup phase
>   * SRVMGT_STATE_READY_LOCAL -- ready for local clients only
>   * SRVMGT_STATE_READY_ALL   -- ready for all clients
>   * SRVMGT_STATE_BUSY-- too busy to process new requests
>   * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing
>   queued requests
>   * SRVMGT_STATE_DEFERRED-- temporarily can't accept new
>   requests (eg. overload)
>   * SRVMGT_STATE_WAITING -- wait for resource (eg. printer
>   needs paper or ink)
>   * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some
>   fatal error)
> 
> For start, we'd just write a small library, that logs to syslog,
> perhaps maintains some pidfiles (maybe even a *compile-time* option
> to route directly to libsystemd), then patch up packages that currently
> use libsystemd to use our new one.
>

I personally don't see why one would like to redo libsystemd0 from
scratch, as you seem so kee of doing. 

Go on down your path, but I suspect not many people would cheer at you
in this camp...

HND

KatolaZ

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[ "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[   @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[ @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]


signature.asc
Description: Digital signature
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng