On 21.03.2013 16:13, Michael Friedrich wrote: > On 21.03.2013 15:25, Marco Hoyer wrote: >> Hello folks, >> >> I'm currently planning a setup of Icinga checking some thousands of >> services (maybe 8-10k) on round about 1500 hosts. >> There is no big problem at all regarding the amount of checks but on >> icinga reload there is a wait time of actually 25s where there is >> nothing going on (with 300 hosts and 3000 services). > > > 25s isn't that much. my dev box with a variety of config specialities > for testing all possible configurations takes ~80s with 4k services on a > dualcore vm. no tuning applied to mysql 5.1.66 on debian squeeze. > > for the problem itsself - there have been so many discussions in the > past on the mailinglists / forums / dev tracker, that i won't repeat > them here. > > for the configuration reload in general, this is required and the > problem is well known. > > more information to be found here: https://dev.icinga.org/issues/1934 > > >> Ido2db is initializing the status database with the new config during >> this time. The problem is, that I would like to reload icinga >> everytime a config-change is made. Therefor a wait time of maybe 5s >> would be acceptable. >> Is there anyone having a similar setup and maybe an idea, how to fix >> this? > > that's not fixable within the current architecture. the core does not > send diffs, as it just does not do any on a reload. so the application > (ido2db) is left alone doing update/insert logic. bad design, but > requires a full rewrite which won't happen within 1.x > > again, read #1934 for detailed thoughts and information. > >> The mysql database storing the icinga status db is actually capable of >> ap. 1200 queries/sec which seems to be very well. >> We think about using SSD storage for it to increase iops but there >> should be a better solution than throwing hardware on it. > > ssd might be fast, but also broken fast enough with many write ops over > time. you've been warned. > > >> >> My ideas: >> - I would like to reduce the amount of data to be loaded by reducing >> the data_processing_opts value in idomod.cfg. May there be a good >> value for icinga-web usage? > > the one which is set by default. read the configuration comments, as > well as follow the wiki for some more proposals on tuning that.
seems i wrote a wiki entry for that as well. someone should monitor my wiki article count... https://wiki.icinga.org/display/howtos/Special+IDOUtils+data_processing_options > >> - Another idea was to use set >> "clean_realtime_tables_on_core_startup=0" and >> "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no >> big effort. Are these parameters working properly? > > they are tagged experimental as for the reason being exactly that. i had > no further reports expect one bug to those options, so unless people > report it working properly, i have no intention to remove the > experimental tag at all. > >> >> Is there anything left I can tune to get a better performance? > > show some love to your database, and invest some time in reading tuning > guides. the icinga wiki holds a bunch of such, which i have collected > over the past years in order to resolve the general interest "where is > the tuning guide?". > > well, read and learn here: > https://wiki.icinga.org/display/howtos/Performance+Tuning > >> >> To prevent some questions: >> We generate icinga config-snipplets on the monitored system and push >> it to the icinga instances. In case of an update on maybe 50 webserver >> nodes there may be some config we need very fast to be active in icinga. >> To prevent an ongoing reload of icinga, we do it only once per update >> transaction. It means that an update on all webservers only causes one >> reload. But doing CLD on many of our systems means that there might be >> many updates >> and therefor many reloads of icinga. >> >> Used versions (compiled from source for RHEL6 and packaged as RPM >> using the supplied spec file): >> Icinga 1.8.4 >> Icinga-Web 1.8.2 > > 1.9 will provide some - again tagged experimental - options to tune the > reload in terms of making it faster. > > https://dev.icinga.org/issues/3527 > https://dev.icinga.org/issues/3533 > > still, the real bottleneck is the database and even if the socket_queue > will help resolve minimize reloading time, it will > > a) cut ido2db more cpu cycles and memory > b) create sort of a proxy which means buffering the data, while waiting > for the db to finish > c) the data arrival in the database will still be slow, so web guis > might still show old data during that update process > > since it's another component between icinga and the database, it might > contain unwanted behaviour, so it will be experimental and turned off by > default. > > the transaction patch encapsulates a config object and its relations > (contacts, dependencies, etc) into a single transaction. this is not > perfect playing but still shows some huge performance increasements in > special environments. still, in my tests it did not help that much, so > it's experimential either way - and turned off by default. > > though, 1.9 isn't release yet, but you may test the beta version, which > is due 1 week before release (planned 25.4.) > > kind regards, > Michael > > > PS: Icinga2 will add ido support via compat module too, in order to stay > compatible with Icinga Web. That's a todo for m3. > > -- DI (FH) Michael Friedrich mail: michael.friedr...@gmail.com twitter: https://twitter.com/dnsmichi jabber: dnsmi...@jabber.ccc.de irc: irc.freenode.net/icinga dnsmichi icinga open source monitoring position: lead core developer url: https://www.icinga.org ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar _______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users
