A short followup: we got rid of the notification delay by using a wrapper script around the check_mk script to force asynchronous notifications. Our average service check delay went from 40s down to 0.6s.
check_mk_notify_wrapper: #!/bin/bash # This is a wrapper script for "check_mk --notify" to decouple # Icinga from check_mk. The check_mk script blocks the whole # Icinga core. Thus, we start check_mk as a background # child. Note: Icinga uses popen (execute a script and fetch the # output into a pipe) to execute this script. Thus, we have to # redirect the outputs to some files as otherwise the # pipe would also wait for all the childrens to execute. check_mk --notify >>/tmp/wrapper.log 2>&1 & Of course the notification in Icinga must be configured to use this wrapper script instead of calling "check_mk --notify" directly. regards Klaus On 26.02.2015 17:08, Michael Friedrich wrote: > Am 26.02.2015 um 17:01 schrieb Klaus Darilion: >> Hi! >> >> We use Icinga1 with check_mk's flexible notifications. > > Not sure how that works, but I could imagine that this is just one > massive script call with all the required runtime macros. > >> Usually the >> Reaper easily reaps 1000 results/second, except when there are >> notifications: > > Icinga1 doesn't run asynchronously, but everything one after the other > in the event loop. Therefore the reaper thread actually waits for the > "reaping event". If other events block, the reaper is not capable of > continueing to collect check results. > > I'd analyze the notification runtime (put a time in front and log it > somewhere), as well as how much of them are triggered at once. Maybe too > many contacts are notified, or anything else is blocking the daemon/script. > >> >> [1424962674.037492] [256.1] [pid=19422] Running command '...' check_mk >> --notify'... >> [1424962674.823460] [256.1] [pid=19422] Execution time=0.785 sec, early >> timeout=0, result=0, output=(null) >> [1424962675.023625] [016.0] [pid=19422] Finished reaping 966 check >> results >> >> Thus, I think our performance problems (Icinga is slow in reading the >> result files) is caused by this notifications. It seems that the Reaper >> waits for the external command to finish. > > External commands are different, you mean the notification command being > executed. Same goes for writing performance data files btw too. Or NEB > modules subscribing to actions and blocking the core from event execution. > >> >> Are there some ways to speed up the notifications, e.g. asynchronously >> by not waiting for script execution? > > Never used checkmk's notification tool, but I could imagine that it's > fairly easy to add these notifications into a spool directory and > process that separately (Bash/Perl or similar). > > Kind regards, > Michael > > > -- > Michael Friedrich, DI (FH) > Application Developer > > NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg > Tel: +49 911 92885-0 | Fax: +49 911 92885-77 > GF: Julian Hein, Bernd Erk | AG Nuernberg HRB18461 > http://www.netways.de | [email protected] > > ** CeBIT 2015 - 16.-20. März 2015 - http://www.netways.de/cebit ** > ** OSDC 2015 - April - osdc.de ** > ** Puppet Camp Berlin 2015 - April - netways.de/puppetcamp ** > ** OSBConf 2015 - September - osbconf.org ** > _______________________________________________ > icinga-users mailing list > [email protected] > https://lists.icinga.org/mailman/listinfo/icinga-users _______________________________________________ icinga-users mailing list [email protected] https://lists.icinga.org/mailman/listinfo/icinga-users
