On 08.08.2013 09:03, Gerd Radecke wrote: > Hi - just realized in the archive that for some reason this wasn't > sent as plain text but html (and I'm not sure if it just shows up as > and empty mail in the archive, or if that's maybe the reason no one > replied), sorry about that. The original e-mail and attachment > follows: > > I have a reoccuring problem and maybe I've looked at it too long, but > I just can't find any possible cause. Regularly (sometimes every 3 > hours, sometimes once a day) the check latencies in Icinga increase > significantly. My usual average is about ~1 second or less and the > increased goes up to 300 seconds. Host Check latencies seem to > increase faster and higher, so if host check latency is as 300, > service check will be at around 100. > > Setup: > RHEL 6.1 > Icinga 1.9.0 with idoutils compiled from source
1.9.1 / 1.9.2 fixed some bugs concerning idoutils. did you verify that ido2db is running w/o problems? > mySQL is running on the same machine > pnp4nagios is also running on the same maschine > nothing else running > 48 GB Ram > 2.8 Ghz Intel Xeon E5440 > 146 GB Raid 1 with 2,5" HP 10k sas drives > > ~1000 Hosts - checked every 5 Minutes > ~10000 Checks - around 6000 checked every 5 minutes, the rest spread > over the day (some only once a day) > > > I've seen this happen 3 times today, each time it slowly increased > over about 10 minutes till. Sometimes it decreases without user > interaction as well. > > I monitored the last increase and looked at > CPU Load => always around 2-4 > cpu wait => around 5% > Memory: 6 GB really used, 40 GB cached > no swap used > disks usage - all nowhere close to full, both block and inode > open files: around 7k for the whole system, 475 for icinga > > > ulimits: > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 386031 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 2048 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > everything under /usr/local/icinga/var/spool/ is on ramdisks > (checkresults,temp_path,debug_file) > > icinga.cfg is attached. > > Any suggestions on what could be causing this or what else I could > monitor to troubleshoot this are greatly appreciated. > > > > ------------------------------------------------------------------------------ > Get 100% visibility into Java/.NET code with AppDynamics Lite! > It's a free troubleshooting tool designed for production. > Get down to code-level detail for bottlenecks, with<2% overhead. > Download for free and get started troubleshooting in minutes. > http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk > > > > _______________________________________________ > icinga-users mailing list > icinga-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/icinga-users -- DI (FH) Michael Friedrich mail: michael.friedr...@gmail.com twitter: https://twitter.com/dnsmichi jabber: dnsmi...@jabber.ccc.de irc: irc.freenode.net/icinga dnsmichi icinga open source monitoring position: lead core developer url: https://www.icinga.org ------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk _______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users
