Am 21.03.2013 um 16:13 schrieb Michael Friedrich: > On 21.03.2013 15:25, Marco Hoyer wrote: >> Hello folks, >> >> I'm currently planning a setup of Icinga checking some thousands of services >> (maybe 8-10k) on round about 1500 hosts. >> There is no big problem at all regarding the amount of checks but on icinga >> reload there is a wait time of actually 25s where there is nothing going on >> (with 300 hosts and 3000 services). > > > 25s isn't that much. my dev box with a variety of config specialities > for testing all possible configurations takes ~80s with 4k services on a > dualcore vm. no tuning applied to mysql 5.1.66 on debian squeeze.
If the goal is to have double the hosts and services per host, a reload may take 50s. That's a big problem if you have to reload quite often. Our infrastructure is in a process of permanent change. Additional load causes the creation of new machines, apps get updates some times a day. All this may cause icinga config changes. > > for the problem itsself - there have been so many discussions in the > past on the mailinglists / forums / dev tracker, that i won't repeat > them here. I know about some of them but things change and there might be something I haven't heard about. > > for the configuration reload in general, this is required and the > problem is well known. > > more information to be found here: https://dev.icinga.org/issues/1934 > > >> Ido2db is initializing the status database with the new config during this >> time. The problem is, that I would like to reload icinga everytime a >> config-change is made. Therefor a wait time of maybe 5s would be acceptable. >> Is there anyone having a similar setup and maybe an idea, how to fix this? > > that's not fixable within the current architecture. the core does not > send diffs, as it just does not do any on a reload. so the application > (ido2db) is left alone doing update/insert logic. bad design, but > requires a full rewrite which won't happen within 1.x > > again, read #1934 for detailed thoughts and information. > >> The mysql database storing the icinga status db is actually capable of ap. >> 1200 queries/sec which seems to be very well. >> We think about using SSD storage for it to increase iops but there should be >> a better solution than throwing hardware on it. > > ssd might be fast, but also broken fast enough with many write ops over > time. you've been warned. > > >> >> My ideas: >> - I would like to reduce the amount of data to be loaded by reducing the >> data_processing_opts value in idomod.cfg. May there be a good value for >> icinga-web usage? > > the one which is set by default. read the configuration comments, as > well as follow the wiki for some more proposals on tuning that. I did but am not informed enough about the side-effects on icinga-web. There is something exported by ido2db, which seems to be unused in icinga-web like the command-definitions but before changing this and running into a trap sometimes later, I wanted to ask somebody knowing about the requirements of icinga-web to ido2db. > >> - Another idea was to use set "clean_realtime_tables_on_core_startup=0" and >> "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no big >> effort. Are these parameters working properly? > > they are tagged experimental as for the reason being exactly that. i had > no further reports expect one bug to those options, so unless people > report it working properly, i have no intention to remove the > experimental tag at all. > >> >> Is there anything left I can tune to get a better performance? > > show some love to your database, and invest some time in reading tuning > guides. the icinga wiki holds a bunch of such, which i have collected > over the past years in order to resolve the general interest "where is > the tuning guide?". > > well, read and learn here: > https://wiki.icinga.org/display/howtos/Performance+Tuning > >> >> To prevent some questions: >> We generate icinga config-snipplets on the monitored system and push it to >> the icinga instances. In case of an update on maybe 50 webserver nodes there >> may be some config we need very fast to be active in icinga. >> To prevent an ongoing reload of icinga, we do it only once per update >> transaction. It means that an update on all webservers only causes one >> reload. But doing CLD on many of our systems means that there might be many >> updates >> and therefor many reloads of icinga. >> >> Used versions (compiled from source for RHEL6 and packaged as RPM using the >> supplied spec file): >> Icinga 1.8.4 >> Icinga-Web 1.8.2 > > 1.9 will provide some - again tagged experimental - options to tune the > reload in terms of making it faster. > > https://dev.icinga.org/issues/3527 > https://dev.icinga.org/issues/3533 > > still, the real bottleneck is the database and even if the socket_queue > will help resolve minimize reloading time, it will > > a) cut ido2db more cpu cycles and memory > b) create sort of a proxy which means buffering the data, while waiting > for the db to finish > c) the data arrival in the database will still be slow, so web guis > might still show old data during that update process > > since it's another component between icinga and the database, it might > contain unwanted behaviour, so it will be experimental and turned off by > default. > > the transaction patch encapsulates a config object and its relations > (contacts, dependencies, etc) into a single transaction. this is not > perfect playing but still shows some huge performance increasements in > special environments. still, in my tests it did not help that much, so > it's experimential either way - and turned off by default. > > though, 1.9 isn't release yet, but you may test the beta version, which > is due 1 week before release (planned 25.4.) > > kind regards, > Michael > > > PS: Icinga2 will add ido support via compat module too, in order to stay > compatible with Icinga Web. That's a todo for m3. > Thank you for your help and some useful hints. I will have a look at Icinga 1.9 to see if there will be some improvement. kind regards Marco > > -- > DI (FH) Michael Friedrich > > mail: michael.friedr...@gmail.com > twitter: https://twitter.com/dnsmichi > jabber: dnsmi...@jabber.ccc.de > irc: irc.freenode.net/icinga dnsmichi > > icinga open source monitoring > position: lead core developer > url: https://www.icinga.org > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_mar > _______________________________________________ > icinga-users mailing list > icinga-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/icinga-users ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar _______________________________________________ icinga-users mailing list icinga-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/icinga-users
