On Fri, 12 Dec 2014, Michael Martinez wrote:
I've got an icinga 1.8 installation that has the following latencies:
Active Service Latency: 0.000 / 1390.955 / 1077.921 sec
Active Host Latency: 0.000 / 1383.793 / 1351.764 sec
633 hosts. 3,800 active service checks.
The server is Redhat linux as a Virtual Machine on a hypervisor. Yes,
I know we shouldn't run Icinga on a Vm, but before you assume this is
the problem, note that there are no bottlenecks whatsoever on the
server.
Running a monitoring application from a VM is perfectly legal and
sometimes practical if certain conditions are met.
First and foremost, you need to take a look at what the hypervisor
view of the situtation is. If there are more VMs in execution than
there are cores available, then contention is a very real probability,
and that will wreak havoc with timing-sensitive operations. How busy
are your "neighbour VMs"? The only place to see this in all its bald
detail is on the hypervisor.
Of particular note here is that the rightmost columns in vmstat are
worse than useless in a VM environment that's oversubscribed; the VM
only knows what *it* sees -- and that may not be reality because other
VMs are likely stealing cycles that would otherwise be available.
I have been trying unsuccessfully to reduce this latency so
that all checks are completed within roughly a 5-minute
timeframe, but I have been unable to get anything better
than the 1000 second latencies seen above.
Here are a few ideas to instrument:
1) Dump performance data to disk and see what the trend for
latency is following a restart of Icinga.
2) Watch the performance of the internal "wall-clock" to see
how well it tracks the physical clock on the wall. (A VM's
wall-clock, being virtual, may well be very, very, wrong if
there is serious contention for cores.)
3) Ascertain what form of time-remediation is being performed
by the hypervisor. TANSTAAFL ("There Ain't No Such Thing As
A Free Lunch") is pertinent here; cycles used by the hypervisor
to service another VM are not going to be ones available to
yours -- and the effect is cumulative. Since Icinga uses the
system's clock, if that clock is losing time, or is inaccurate
for any other reason, the latency numbers will be inaccurate
as well.
4) Look to see if some services are more affected than other
services. Sometimes this can yield clues.
5) Are you running IDO? If so, how healthy is the database?
Newer versions of Icinga feed IDO via a FIFO which tends to
mitigate blockages in the event-broker for IDO-related
events.
There are a few. With any luck, they may be of help and/or
real assistance.
Cheers!
+------------------------------------------------+---------------------+
| Carl Richard Friend (UNIX Sysadmin) | West Boylston |
| Minicomputer Collector / Enthusiast | Massachusetts, USA |
| mailto:[email protected] +---------------------+
| http://users.rcn.com/crfriend/museum | ICBM: 42:22N 71:47W |
+------------------------------------------------+---------------------+
_______________________________________________
icinga-users mailing list
[email protected]
https://lists.icinga.org/mailman/listinfo/icinga-users
