On Thu, 1 Nov 2012, Zachary McGibbon wrote:
> Has anyone seen something like this? The host went down, then came back up
> but the service check took 43 minutes to run.
Without more information, specifically other things that your
monitoring host was doing at the time other than restarting Icinga
and whether you're running idoutils, we can't really say much.
If you're running idoutils, the main scheduling loop won't start
until the database entries have been recereated, and that can take
significant amounts of time especially if there is contention for
the database. This can be exacerbated if you have more than one
instance of Icinga pointed at the same database engine.
As a guess, since you restart your process every day, today's
behaviour was anomalous, so something must have perturbed it. Do
you have sar configured to snapshot performance every so often? If
so, you can see what the machine was doing between 12:14 and 12:53
in terms of CPU, memory, and I/O usage -- what that will not tell
you, however, is whether something was perrturbing another host -- a
database host, for instance.
Here's a startup sequence on my Icinga (with idoutils) system at
home:
[2012-10-28 10:19:30] Icinga 1.8.0 starting... (PID=9809)
[2012-10-28 10:19:30] Local time is Sun Oct 28 10:19:30 EDT 2012
[2012-10-28 10:19:30] LOG VERSION: 2.0
[2012-10-28 10:19:30] idomod: IDOMOD 1.8.0 (10-18-2012) Copyright(c)
2005-2008 Ethan Galstad, Copyright(c) 2009-2012 Icinga Development Team
(https://www.icinga.org)
[2012-10-28 10:19:30] idomod: Successfully connected to data sink. 0 queued
items to flush.
[2012-10-28 10:19:30] Event broker module 'IDOMOD' version '1.8.0' from
'/usr/local/icinga/lib/idomod.so' initialized successfully.
[2012-10-28 10:19:30] Finished daemonizing... (New PID=9811)
[2012-10-28 10:19:37] Event loop started..
Note that my very limited system takes seven full seconds to get
from instantiation to the main event loop -- and that's down to the
idoutils repopulating the database. This delay can be extensive in
large installations (like yours) and, in particular, if there is
contention on the database in the form of table locks. NO CHECKS
will be run until the event loop starts, so we particularly need
that timestamp.
Cheers!
+------------------------------------------------+---------------------+
| Carl Richard Friend (UNIX Sysadmin) | West Boylston |
| Minicomputer Collector / Enthusiast | Massachusetts, USA |
| mailto:crfri...@rcn.com +---------------------+
| http://users.rcn.com/crfriend/museum | ICBM: 42:22N 71:47W |
+------------------------------------------------+---------------------+
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
icinga-users mailing list
icinga-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/icinga-users
