Hi - just realized in the archive that for some reason this wasn't sent as plain text but html (and I'm not sure if it just shows up as and empty mail in the archive, or if that's maybe the reason no one replied), sorry about that. The original e-mail and attachment follows:
I have a reoccuring problem and maybe I've looked at it too long, but I just can't find any possible cause. Regularly (sometimes every 3 hours, sometimes once a day) the check latencies in Icinga increase significantly. My usual average is about ~1 second or less and the increased goes up to 300 seconds. Host Check latencies seem to increase faster and higher, so if host check latency is as 300, service check will be at around 100. Setup: RHEL 6.1 Icinga 1.9.0 with idoutils compiled from source mySQL is running on the same machine pnp4nagios is also running on the same maschine nothing else running 48 GB Ram 2.8 Ghz Intel Xeon E5440 146 GB Raid 1 with 2,5" HP 10k sas drives ~1000 Hosts - checked every 5 Minutes ~10000 Checks - around 6000 checked every 5 minutes, the rest spread over the day (some only once a day) I've seen this happen 3 times today, each time it slowly increased over about 10 minutes till. Sometimes it decreases without user interaction as well. I monitored the last increase and looked at CPU Load => always around 2-4 cpu wait => around 5% Memory: 6 GB really used, 40 GB cached no swap used disks usage - all nowhere close to full, both block and inode open files: around 7k for the whole system, 475 for icinga ulimits: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 386031 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 2048 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited everything under /usr/local/icinga/var/spool/ is on ramdisks (checkresults,temp_path,debug_file) icinga.cfg is attached. Any suggestions on what could be causing this or what else I could monitor to troubleshoot this are greatly appreciated.
icinga.cfg
Description: Binary data
------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users
