Hi, The check latency is ok on the satellites, which are doing the checks and send this checks as passiv checks to the central system. We have some checks, where the execution time is long, for example "jmx monitoring". But as I said these checks are doing the satellites and not the central system.
@Bernd - Have you tried to modify reaper_frequenzy? We use the following values: check_result_reaper_frequency=10 max_check_result_reaper_time=30 - Check out processlist in mysql (show full processlist) to check if database is the root of all evil :-) During the event_broker runtime, there is only one process, that updates,query the mysql db - I think you need to identify the root cause using stats of mysql and system The queries/s of the mysql are around 50/s and the overall load is around 4.0 BR, Jens -----Original Message----- From: Stock, Daniel [mailto:d.st...@ing-diba.de] Sent: Monday, May 23, 2011 15:48 To: icinga-users@lists.sourceforge.net Subject: Re: [icinga-users] distributed icinga system, decreased peformance if event_broker ndomod is used Hi Jens, we had a similar Problem. We did a review of the check that where being executed and removed a few that where taking to long. It seems they would hold up the waiting line for other checks. Since then we haven't had any problems with the latency. Aloha Dan Stock -----Ursprüngliche Nachricht----- Von: Bernd Erk [mailto:bernd....@netways.de] Gesendet: Montag, 23. Mai 2011 15:29 An: icinga-users@lists.sourceforge.net Betreff: Re: [icinga-users] distributed icinga system, decreased peformance if event_broker ndomod is used Hi Jens, the questions is, if the database is responsible for the problem. Perhaps it is a general load problem with the check execution on the master. - Have you tried to modify reaper_frequenzy? - Check out processlist in mysql (show full processlist) to check if database is the root of all evil :-) - I think you need to identify the root cause using stats of mysql and system Regards Bernd ________________________________________ Von: Mueller, Jens [jens.muel...@navteq.com] Gesendet: Montag, 23. Mai 2011 15:00 An: icinga-users@lists.sourceforge.net Betreff: [icinga-users] distributed icinga system, decreased peformance if event_broker ndomod is used Hi, We are using icinga (distributed monitoring setup (central instance with two satellites) ) to monitor around 500 server and 5900 services at the moment. Software version: Icinga-version 1.3.0 Ido2db version 1.3.0 Used hardware: Central server (HA) 2x Quad-Core Opteron 32GB RAM 600GB SAS Raid 1 Satellite (doing the checks for the services/hardware) 2x Quad-Core Opteron 32GB RAM SAS drives In the beginning we used the event_broker idomod to store config/status data in a mysql db. After the number of checks were growing, we saw that the scheduling queue is more and more behind the time, which leads to check latencies greater then 3k sec. Also the freshness checks for the passiv chekcs are not working anymore and so the active checks of the icinga server itself (checks from icinga-cluster) will not be processed. We tried to reduce the amount of data, that will be written to the mysql db, but this didn’t change anything. I tired the following things to find out the root cause of this problem: · Decreased the amount of data that is written to the mysql db · Enabled debug logging of icinga/ido2db, but didn’t find any errors/warnings · Disabled mysql replication to the standby node · Tuned mysql settings Overall this didn’t really fixed the problem, is the size of data/monitoring checks maybe to much for the ido2db ? I doubt or hope that ☺ Has/had anybody of you, maybe a similar problem with icinga and ido2db ? BR, Jens Müller ________________________________ The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users ING-DiBa AG, Frankfurt am Main. Registernummer HRB 7727, Handelsregister Amtsgericht Frankfurt am Main. Vorstand: Roland Boekhout (Vorsitzender), Bas Brouwers, Bernd Geilen, Katharina Herrmann, Martin Krebs, Herbert Willius. Aufsichtsrat: Ben Tellings (Vorsitzender) Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users
