I'm not sure if anyone else is doing this but I thought I'd share a
neat little trick I did. Our Icinga setup is quite large, 7300+ hosts
and 16000 services and they mostly all come from an Oracle database.
We run scripts that pulls the data every 4 hours and reloads Icinga
automatically. Of course, there are often inconsistencies (mostly
child/parent relationships) so we need to know when something is
broken. Since I get over 100+ emails from Icinga per day, I
automatically put them in a folder that I don't look at very often, so
even when my script runs every 4 hours, these failure emails go into
this folder too.
To remedy this situation, I created a passive service for my Icinga
host called 'Icinga Config'. When my script runs to update my config,
at the end it checks if there were any errors and then sends a passive
check to this service. Since I run Nagstamon on my desktop, I see the
alert right away when it breaks.
Examples:
define service{
use server_icinga
service_description Icinga Config
active_checks_enabled 0
notifications_enabled 1
flap_detection_enabled 0
check_freshness 0
check_command check_icinga_stats!0
hostgroup_name server_icinga
servicegroups icinga
}
Then my shell script that runs every 4 hours (just the last part):
# Event handler to submit check results
EVENTHANDLER="/usr/lib64/icinga/eventhandlers/submit_check_result"
# Set ICINGACHK
ICINGACHK="/var/spool/icinga/icinga.chk"
if [ -f $ICINGACHK ]; then
$EVENTHANDLER etna "Icinga Config" 2 "CRITICAL - Please Verify
Icinga Config"
else
$EVENTHANDLER etna "Icinga Config" 0 "OK - Icinga Config is Good"
fi
Enjoy!
- Zachary
------------------------------------------------------------------------------
Own the Future-Intel® Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
_______________________________________________
icinga-users mailing list
icinga-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/icinga-users
