Re: [systemd-devel] Complex supervision structures/delegating watchdog?
On Mon, 16.02.15 21:36, Holger Hans Peter Freyther (hol...@freyther.de) wrote:
Sorry for the late reply, still catching up with piles of email queued
up.
On Mon, Feb 16, 2015 at 11:21:53AM +0100, Lennart Poettering wrote:
Is your pppd daemon itself also a systemd service?
What precisely does monitor consist of for this case?
Yes, the pppd is invoked by systemd as a service. What I am
doing right now is:
link.service:
ExecStart=pppd call uplink ... nodetach
/etc/ppp/ip-up.d/NN-linkmon:
linkmon -i $PPP_IFACE -d 8.8.8.8 -p `cat /run/${PPP_IFACE}.pid`
The monitoring right now involves simple ICMP request and
depending on the outcome I change the metric of one of the
default routes. So in case some amount of packet loss is
reached the linkmon will SIGKILL the pid provided.
We have watchdog support already, with sd_notify(0, WATCHDOG=1), and
WatchdogSec=. But that requires you to run your pppd as a service of
its own, to be useful.
I thought the sd_notify is only possible by the main
application that got started?
Well, that's the way this was intended originally, but you actually
use it differently, by using NotifyAccess=all (instead of
NotifyAccess=main).
That said given your usecase I don#t really see how the watchdog
systemd implements would be helpful. After all the key words here
really are app specific monitoring rather than just ping watchdog.
So far whenever the question of monitoring came up I just said: we
provide the apis, you implement the monitoring, without going into
more detail. I figure we should put together some wiki page that
explains exactly how we think a monitoring tool would make use of
systemd's functionality, and actually figure out if we really have
everything in place to make it work nicely, and fill the gaps where we
don't.
E.g. in the above case the linkmon would be a child of pppd. My
application wouldn't run until pppd has setup the link. This means I
would need to configure a high enough timeout to cope with a
potential bigger set-up time.
Ideally, pppd would simply report back when it finished setting up the
connection. Then, the monitoring daemon could use After= on the ppd
service, and would be started only after the connection up. And it
should use BindsTo= to glue itself to the pppd instances.
I figure making pppd report back when it completed setting up the
connection is easy, by making use of the normal detach mode with
Type=forking and a PID file. Even better of course would be to patch
pppd to use sd_notify(READY=1) so that this ugliness with double
forking would be unnecessary
Hope this makes some sense,
Lennart
--
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Complex supervision structures/delegating watchdog?
On Mon, Feb 16, 2015 at 11:21:53AM +0100, Lennart Poettering wrote:
Is your pppd daemon itself also a systemd service?
What precisely does monitor consist of for this case?
Yes, the pppd is invoked by systemd as a service. What I am
doing right now is:
link.service:
ExecStart=pppd call uplink ... nodetach
/etc/ppp/ip-up.d/NN-linkmon:
linkmon -i $PPP_IFACE -d 8.8.8.8 -p `cat /run/${PPP_IFACE}.pid`
The monitoring right now involves simple ICMP request and
depending on the outcome I change the metric of one of the
default routes. So in case some amount of packet loss is
reached the linkmon will SIGKILL the pid provided.
We have watchdog support already, with sd_notify(0, WATCHDOG=1), and
WatchdogSec=. But that requires you to run your pppd as a service of
its own, to be useful.
I thought the sd_notify is only possible by the main
application that got started? E.g. in the above case the
linkmon would be a child of pppd. My application wouldn't
run until pppd has setup the link. This means I would need
to configure a high enough timeout to cope with a potential
bigger set-up time.
One nice thing for an external watchdog started by the
NN-linkmon script would be that it would be under pid1
control (e.g. if it is crashing, the other service would
be taken down by systemd), I could have different privileges
for the monitoring system (currently at least one part of
it must be able to send a kill to a parent process).
Another neat feature would be if applications could
communicate some extra (custom) status to systemd. E.g.
in the case of pppd to indicate the state of the link-setup,
for something like our BTS process to indicate if it is
currently broadcasting or muted.
Thanks for listening
holger
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Complex supervision structures/delegating watchdog?
On Mon, Feb 16, 2015 at 10:36 PM, Holger Hans Peter Freyther
hol...@freyther.de wrote:
On Mon, Feb 16, 2015 at 11:21:53AM +0100, Lennart Poettering wrote:
Is your pppd daemon itself also a systemd service?
What precisely does monitor consist of for this case?
Yes, the pppd is invoked by systemd as a service. What I am
doing right now is:
link.service:
ExecStart=pppd call uplink ... nodetach
/etc/ppp/ip-up.d/NN-linkmon:
linkmon -i $PPP_IFACE -d 8.8.8.8 -p `cat /run/${PPP_IFACE}.pid`
The monitoring right now involves simple ICMP request and
depending on the outcome I change the metric of one of the
default routes. So in case some amount of packet loss is
reached the linkmon will SIGKILL the pid provided.
We have watchdog support already, with sd_notify(0, WATCHDOG=1), and
WatchdogSec=. But that requires you to run your pppd as a service of
its own, to be useful.
I thought the sd_notify is only possible by the main
application that got started? E.g. in the above case the
linkmon would be a child of pppd. My application wouldn't
run until pppd has setup the link. This means I would need
to configure a high enough timeout to cope with a potential
bigger set-up time.
NotifyAccess=
One nice thing for an external watchdog started by the
NN-linkmon script would be that it would be under pid1
control (e.g. if it is crashing, the other service would
be taken down by systemd), I could have different privileges
for the monitoring system (currently at least one part of
it must be able to send a kill to a parent process).
You can ask systemd to stop or reload a service.
Another neat feature would be if applications could
communicate some extra (custom) status to systemd. E.g.
in the case of pppd to indicate the state of the link-setup,
for something like our BTS process to indicate if it is
currently broadcasting or muted.
sd_notify(STATUS=Reticulating splines);
sd_notify(READY=1\nSTATUS=Connected.);
--
Mantas Mikulėnas graw...@gmail.com
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Complex supervision structures/delegating watchdog?
Hi, for one application we will spawn one or more pppd daemons. Once a link is up I would like to monitor them. The closest thing that I can do right now is systemctl start mon@$PPP_IFACE from within a /etc/ppp/ip-up.d/mon file. In case I consider the link broken or I want to bring it down, I don't know how to do it. What options do I have? I can look at /run/$PPP_IFACE.pid and then send a SIGHUP to the task (this requires that the monitor app is allowed to do that). What would be neat is that if I could spawn a watchdog from with-in /etc/ppp/ip-up.d/mon for the existing service and that when the application exits (or doesn't respond/send watchdog messages to systemd) the parent will be stopped. Is something like this already possible or wanted behavior? How would you handle that? holger ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Complex supervision structures/delegating watchdog?
On Sat, Feb 14, 2015 at 10:31 PM, Holger Freyther hol...@freyther.de wrote: Hi, for one application we will spawn one or more pppd daemons. Once a link is up I would like to monitor them. The closest thing that I can do right now is systemctl start mon@$PPP_IFACE from within a /etc/ppp/ip-up.d/mon file. In case I consider the link broken or I want to bring it down, I don't know how to do it. What options do I have? I can look at /run/$PPP_IFACE.pid and then send a SIGHUP to the task (this requires that the monitor app is allowed to do that). Pidfiles? systemctl kill -s SIGHUP pppd@$PPP_IFACE.service ExecReload=/bin/kill -HUP $MAINPID systemctl reload pppd@$PPP_IFACE.service What would be neat is that if I could spawn a watchdog from with-in /etc/ppp/ip-up.d/mon for the existing service and that when the application exits (or doesn't respond/send watchdog messages to systemd) the parent will be stopped. I wonder if you could have the pppd service automatically start the pppd-watchdog@ service using some combination of Before=, Requires=, and BindsTo= (or something similar that I'd forgotten). When the watchdog process exits, the pppd-watchdog@ service fails, causing pppd@ to also fail due to the Requires=. -- Mantas Mikulėnas graw...@gmail.com ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
