Re: [arch-general] Systemd email notifications
On Thursday 13 Feb 2014 17:58:05 Damjan Georgievski wrote: Yeah, I think it's possible to get systemd to poll a script, or there's always cron (or a timer unit) that should allow us to manually inspect a process and restart it if necessary. But it would be cooler if there were shortcuts to features that we see in Monit and other similar systems; something like this in a unit file: MaxMemoryThreshold=100M MaxMemoryCheckInterval=30 MaxMemoryIntervalThrehold=2 The memory is then checked every 30 seconds. When the unit exceeds this amount of RAM for 2 successive intervals, the unit is restarted. there is setting of ulimits in systemd, it's much more brutal though, will not wait for 30 seconds man systemd.exec Thank you; I didn't know about that, and it's very interesting. The limits are indeed a bit strict for server process monitoring. I imagine it would be really handy as a failsafe in embedded environments, though. The great benefit of them is that (as far as I can tell), it's the kernel that enforces the limits, so there's no polling involved. I suppose that the gold standard for monitoring would be a combination of the two: a userspace component (e.g. systemd or associated monitoring tool) registering to receive events from the kernel on certain resource thresholds for a process, so that e.g. time spent above RAM threshold could be monitored accurately in an event-based way, without polling. That would be awesome. Paul
[arch-general] Systemd email notifications
Hello all, Does anyone know of any standard system for receiving notifications from systemd for unit state changes? I currently use Monit for the monitoring of many processes, and it'll e-mail me when things happen (e.g. a process was restarted). Since switching to systemd, it's felt a bit silly that for several processes, I'm having Monit monitor them simply because systemd is unable to tell me it restarted a unit. Monit isn't actually required to keep those processes alive as it once was, because systemd can do that. Paul
Re: [arch-general] Systemd email notifications
Am 13.02.2014 13:04, schrieb Paul Gideon Dann: Does anyone know of any standard system for receiving notifications from systemd for unit state changes? I currently use Monit for the monitoring of many processes, and it'll e-mail me when things happen (e.g. a process was restarted). Since switching to systemd, it's felt a bit silly that for several processes, I'm having Monit monitor them simply because systemd is unable to tell me it restarted a unit. Monit isn't actually required to keep those processes alive as it once was, because systemd can do that. I'd place a bet on the systemd dbus API: IIRC, it exports the state of each unit as a property and then emits the standard org.freedesktop.DBus.Properties.PropertiesChanged signal when the state changes. So, your task would be to subscribe to that signal and act on it. This could be nicely done in python (and maybe someone has done it already). I just listed some properties using qdbus: $ qdbus --system org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/dbus_2eservice org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Unit ActiveState active $ qdbus --system org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/dbus_2eservice org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Unit SubState running $ qdbus --system org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/dbus_2eservice org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Service MainPID 391 Surely, the ActiveState, SubState and MainPID properties would change when something gets restarted or stopped, and you would receive the PropertiesChanged signal. signature.asc Description: OpenPGP digital signature
Re: [arch-general] Systemd email notifications
On Thursday 13 Feb 2014 13:35:59 Thomas Bächler wrote: Am 13.02.2014 13:04, schrieb Paul Gideon Dann: Does anyone know of any standard system for receiving notifications from systemd for unit state changes? I currently use Monit for the monitoring of many processes, and it'll e-mail me when things happen (e.g. a process was restarted). Since switching to systemd, it's felt a bit silly that for several processes, I'm having Monit monitor them simply because systemd is unable to tell me it restarted a unit. Monit isn't actually required to keep those processes alive as it once was, because systemd can do that. I'd place a bet on the systemd dbus API: IIRC, it exports the state of each unit as a property and then emits the standard org.freedesktop.DBus.Properties.PropertiesChanged signal when the state changes. So, your task would be to subscribe to that signal and act on it. This could be nicely done in python (and maybe someone has done it already). Agreed, but I'm baffled as to why there isn't already a well-known tool. To be honest, I'd have expected it to be important enough to be produced along-side the systemd project, probably with several backends for different notification systems. Paul
Re: [arch-general] Systemd email notifications
On Thursday 13 Feb 2014 14:21:36 ushi wrote:
Am 13.02.2014 13:04, schrieb Paul Gideon Dann:
Hello all,
Does anyone know of any standard system for receiving notifications
from systemd for unit state changes? I currently use Monit for the
monitoring of many processes, and it'll e-mail me when things
happen (e.g. a process was restarted). Since switching to systemd,
it's felt a bit silly that for several processes, I'm having Monit
monitor them simply because systemd is unable to tell me it
restarted a unit. Monit isn't actually required to keep those
processes alive as it once was, because systemd can do that.
Paul
Hey Paul,
Check out OnFailure=
[Unit]
Description=HTTP Service
OnFailure=mail-root@http.service
...
The mail-root@.service is a generic oneshot service that mails you
some stuff.
[Unit]
Description=Mailer
[Service]
Type=oneshot
ExecStart=/path/to/my/mail-script %i
[Install]
WantedBy=multi-user.target
And your mail-script is somthing like...
#!/usr/bin/env sh
systemctl status ${1}.service | \
mail -s Failed Service: ${1} ad...@example.com
See:
http://www.freedesktop.org/software/systemd/man/systemd.unit.html#OnFailure=
Thanks ushi, that's certainly something. It looks promising for custom unit
files, but not
so great for catching unexpected unit failures. I'll definitely keep that one
in mind,
though.
Paul
Re: [arch-general] Systemd email notifications
On 13 February 2014 13:35, Paul Gideon Dann pdgid...@gmail.com wrote:
On Thursday 13 Feb 2014 14:21:36 ushi wrote:
Am 13.02.2014 13:04, schrieb Paul Gideon Dann:
Hello all,
Does anyone know of any standard system for receiving notifications
from systemd for unit state changes? I currently use Monit for the
monitoring of many processes, and it'll e-mail me when things
happen (e.g. a process was restarted). Since switching to systemd,
it's felt a bit silly that for several processes, I'm having Monit
monitor them simply because systemd is unable to tell me it
restarted a unit. Monit isn't actually required to keep those
processes alive as it once was, because systemd can do that.
Paul
Hey Paul,
Check out OnFailure=
[Unit]
Description=HTTP Service
OnFailure=mail-root@http.service
...
The mail-root@.service is a generic oneshot service that mails you
some stuff.
[Unit]
Description=Mailer
[Service]
Type=oneshot
ExecStart=/path/to/my/mail-script %i
[Install]
WantedBy=multi-user.target
And your mail-script is somthing like...
#!/usr/bin/env sh
systemctl status ${1}.service | \
mail -s Failed Service: ${1} ad...@example.com
See:
http://www.freedesktop.org/software/systemd/man/systemd.unit.html#OnFailure=
Thanks ushi, that's certainly something. It looks promising for custom unit
files, but not
so great for catching unexpected unit failures. I'll definitely keep that
one in mind,
though.
You could use unit overrides[0] to add the OnFailure to provided
units. So should be able to set it up for any service you are
interested in receiving notifications for, I would assume.
[0] https://wiki.archlinux.org/index.php/Systemd#Editing_provided_unit_files
Re: [arch-general] Systemd email notifications
On Thu, Feb 13, 2014 at 1:35 PM, Thomas Bächler tho...@archlinux.org wrote:
Am 13.02.2014 13:04, schrieb Paul Gideon Dann:
Does anyone know of any standard system for receiving
notifications from systemd for unit state changes? I currently
use Monit for the monitoring of many processes, and it'll e-mail
me when things happen (e.g. a process was restarted). Since
switching to systemd, it's felt a bit silly that for several
processes, I'm having Monit monitor them simply because
systemd is unable to tell me it restarted a unit. Monit isn't
actually required to keep those processes alive as it once was,
because systemd can do that.
I'd place a bet on the systemd dbus API: IIRC, it exports the state of
each unit as a property and then emits the standard
org.freedesktop.DBus.Properties.PropertiesChanged signal when the state
changes.
So, your task would be to subscribe to that signal and act on it. This
could be nicely done in python (and maybe someone has done it already).
Ok... I'll take the chance to practice my DBus abilities...
It is a bit long, but it kind of works. Just replace the print() call
with your favourite sendmail function and you'll get a notification
every time any of the units specified in the command line changes
status.
HTH.
#!/usr/bin/python
from gi.repository import GObject
import sys
import dbus
from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)
bus = dbus.SystemBus()
systemd = bus.get_object('org.freedesktop.systemd1',
'/org/freedesktop/systemd1')
manager = dbus.Interface(systemd, 'org.freedesktop.systemd1.Manager')
def GetPropChanged(obj, iface, propName, changes, invalids):
if propName in invalids:
return obj.Get(iface, propName, dbus_interface=dbus.PROPERTIES_IFACE)
elif propName in changes:
return changes[propName]
else:
return None
def OnPropChanged(unit, name, iface, changes, invalids):
if iface != 'org.freedesktop.systemd1.Unit':
return
state = GetPropChanged(unit, iface, 'ActiveState', changes, invalids)
substate = GetPropChanged(unit, iface, 'SubState', changes, invalids)
if state or substate:
print('Status changed', name, state, substate)
for unitName in sys.argv[1:]:
unit = bus.get_object('org.freedesktop.systemd1', manager.GetUnit(unitName))
unit.connect_to_signal('PropertiesChanged', lambda a,b,c :
OnPropChanged(unit, unitName, a, b, c),
dbus_interface=dbus.PROPERTIES_IFACE)
loop = GObject.MainLoop()
loop.run()
Re: [arch-general] Systemd email notifications
Am 13.02.2014 16:05, schrieb Rodrigo Rivas: Ok... I'll take the chance to practice my DBus abilities... It is a bit long, but it kind of works. Just replace the print() call with your favourite sendmail function and you'll get a notification every time any of the units specified in the command line changes status. This isn't too long and it actually seems to work fine. I guess what Paul actually wants is a system where you would subscribe to all services, not just some of them. This should be possible as well with the API. signature.asc Description: OpenPGP digital signature
Re: [arch-general] Systemd email notifications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am 13.02.2014 16:11, schrieb Thomas Bächler: Am 13.02.2014 16:05, schrieb Rodrigo Rivas: Ok... I'll take the chance to practice my DBus abilities... It is a bit long, but it kind of works. Just replace the print() call with your favourite sendmail function and you'll get a notification every time any of the units specified in the command line changes status. This isn't too long and it actually seems to work fine. I guess what Paul actually wants is a system where you would subscribe to all services, not just some of them. This should be possible as well with the API. This works pretty good. It shouldn't be that hard to monitor all units using ListUnits(). http://www.freedesktop.org/wiki/Software/systemd/dbus/ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJS/OMhAAoJEAAoDO4PlX3grQAP/0TI/WjvFPmkkx7PvqRQ2Va4 Mz7hgLAfETjsB9SVZfSry/lpZ8xTrMJjYubtIAoyqvHx+nWowTAOz6toqNMOKSy/ bTTyzno1FUJ5C1NMzqcewFDEnDQuYuaK1V41Ff6qQ/HpybJNZq5wYe4mm4ATBU8U TQ7S57tg54VjsQ3QgoloWusAEw21AAtNUzqnDWS9iA2TRipk5eHoIVxRBI6ao2rZ GnNpori9Rl7Fn18qVtkA4HRc9FQg0e3DLB5YYRwvT4Z8zRhTQegujUZw4lbivvKL 1mTVqTtv/RVrt3GAft9/gqn9JhzSD0ucU+/2UciHUmKuYW3Ek1CV7bsmj8s3zggx sGt7IRTh5Ds5XKJF3fQHKkmLZKiUTBlh0hVouyCPdDbkY9HgG01MEO1HhOn4+9YV v/b2T/xLBFvgSycGwuU6yZBdqfOb6LqSSWhgTm/tqO4XqH5EuXskivSt/gGTPVmv HdVcGxeVFqUmCi0WDEdnAOlNMU5O5KxwPKEBhO4V8DjOOTvHjoIsg5CQMkrY7RNC ACEx/ZFXKPyQhksDeCOGPJ1WLDUj+EeVsew2EHCv29ilQQltgJvqk1l1zI5MD4qj +2suxy+2Nuxalgv0XoXTkFLJfqs7ee3f9EN3j6FIqOjcjv4o90xUAsEHJT1a4wEN 12sQQBTpFdjCUydCHGfD =9EOv -END PGP SIGNATURE-
Re: [arch-general] Systemd email notifications
On Thursday 13 Feb 2014 16:11:56 Thomas Bächler wrote: Am 13.02.2014 16:05, schrieb Rodrigo Rivas: Ok... I'll take the chance to practice my DBus abilities... It is a bit long, but it kind of works. Just replace the print() call with your favourite sendmail function and you'll get a notification every time any of the units specified in the command line changes status. This isn't too long and it actually seems to work fine. I guess what Paul actually wants is a system where you would subscribe to all services, not just some of them. This should be possible as well with the API. Yeah, though actually I'm just really surprised that, given the incredible administrative benefits of systemd, there isn't currently anything that leverages it for actual process monitoring and reporting. As far as I can tell, systemd is also not yet able to automatically restart bloated or stale services (e.g. worker instances that may go haywire). Hopefully these things will come along now that systemd --user is maturing. I'm actually pretty excited about user-systemd. I'm implementing monitoring for long- running Ruby processes for a Rails app, and although the go-to tool for this in the Ruby world is god, I've found it really icky and brittle, and really wish I could replace it with systemd, but for now I will make my peace. Paul
Re: [arch-general] Systemd email notifications
On Thursday 13 Feb 2014 16:05:05 Rodrigo Rivas wrote:
Ok... I'll take the chance to practice my DBus abilities...
It is a bit long, but it kind of works. Just replace the print() call
with your favourite sendmail function and you'll get a notification
every time any of the units specified in the command line changes
status.
HTH.
#!/usr/bin/python
from gi.repository import GObject
import sys
import dbus
from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)
bus = dbus.SystemBus()
systemd = bus.get_object('org.freedesktop.systemd1',
'/org/freedesktop/systemd1')
manager = dbus.Interface(systemd, 'org.freedesktop.systemd1.Manager')
def GetPropChanged(obj, iface, propName, changes, invalids):
if propName in invalids:
return obj.Get(iface, propName,
dbus_interface=dbus.PROPERTIES_IFACE) elif propName in changes:
return changes[propName]
else:
return None
def OnPropChanged(unit, name, iface, changes, invalids):
if iface != 'org.freedesktop.systemd1.Unit':
return
state = GetPropChanged(unit, iface, 'ActiveState', changes, invalids)
substate = GetPropChanged(unit, iface, 'SubState', changes, invalids)
if state or substate:
print('Status changed', name, state, substate)
for unitName in sys.argv[1:]:
unit = bus.get_object('org.freedesktop.systemd1',
manager.GetUnit(unitName)) unit.connect_to_signal('PropertiesChanged',
lambda a,b,c :
OnPropChanged(unit, unitName, a, b, c),
dbus_interface=dbus.PROPERTIES_IFACE)
loop = GObject.MainLoop()
loop.run()
Thanks Rodrigo; great work. I may come back to this in future if no official
solution
ever seems forthcoming.
Paul
Re: [arch-general] Systemd email notifications
On Thu, Feb 13, 2014 at 1:29 PM, Paul Gideon Dann pdgid...@gmail.com wrote: Yeah, though actually I'm just really surprised that, given the incredible administrative benefits of systemd, there isn't currently anything that leverages it for actual process monitoring and reporting. As far as I can tell, systemd is also not yet able to automatically restart bloated or stale services (e.g. worker instances that may go haywire). Hopefully these things will come along now that systemd --user is maturing. I think you can rely on software or hardware watchdogs, which are supported by systemd. http://0pointer.de/blog/projects/watchdog.html -- A: Because it obfuscates the reading. Q: Why is top posting so bad? For more information, please read: http://idallen.com/topposting.html --- Denis A. Altoe Falqueto Linux user #524555 ---
Re: [arch-general] Systemd email notifications
On Thu, Feb 13, 2014 at 5:19 PM, Paul Gideon Dann pdgid...@gmail.com wrote: On Thursday 13 Feb 2014 13:36:35 Denis A. Altoé Falqueto wrote: On Thu, Feb 13, 2014 at 1:29 PM, Paul Gideon Dann pdgid...@gmail.com wrote: Yeah, though actually I'm just really surprised that, given the incredible administrative benefits of systemd, there isn't currently anything that leverages it for actual process monitoring and reporting. As far as I can tell, systemd is also not yet able to automatically restart bloated or stale services (e.g. worker instances that may go haywire). Hopefully these things will come along now that systemd --user is maturing. I think you can rely on software or hardware watchdogs, which are supported by systemd. Yeah, I think it's possible to get systemd to poll a script, or there's always cron (or a timer unit) that should allow us to manually inspect a process and restart it if necessary. But it would be cooler if there were shortcuts to features that we see in Monit and other similar systems; something like this in a unit file: MaxMemoryThreshold=100M MaxMemoryCheckInterval=30 MaxMemoryIntervalThrehold=2 The memory is then checked every 30 seconds. When the unit exceeds this amount of RAM for 2 successive intervals, the unit is restarted. Paul Well, you have the LimitAS= option. I've used it with Restart=on-failure with some services. If I understand it correctly, it will not restart the service when the memory limit is reached, but `malloc()` will return an error. Since many programs are not prepared for that, they segfault and then they are restarted automatically. Not very elegant, but you don't need the polling interval. Rodrigo
