This issue is definitely a problem. I have a support ticket in with Puppet Labs about the same thing. My CPU remains at 100% almost constantly and it slows things down significantly. If you strace it you can see that very little appears to be going on. This is absolutely not normal behavior. Even when I had 1 client checking in I had all cores fully used.
On Wed, Dec 15, 2010 at 12:15 PM, Brice Figureau < brice-pup...@daysofwonder.com> wrote: > On Wed, 2010-12-15 at 05:28 -0800, Chris wrote: > > > > On Dec 15, 12:42 pm, Brice Figureau <brice-pup...@daysofwonder.com> > > wrote: > > > On Tue, 2010-12-14 at 00:24 -0800, Chris wrote: > > > > Hi > > > > > > > I recently upgraded my puppet masters (and clients) from 0.24.8 to > > > > 2.6.4 > > > > > > > Previously, my most busy puppet master would hover around about 0.9 > > > > load average, after the upgrade, its load hovers around 5 > > > > > > > I am running passenger and mysql based stored configs. > > > > > > > Checking my running processes, ruby (puppetmasterd) shoots up to 99% > > > > cpu load and stays there for a few seconds before dropping again. > > > > Often there are 4 of these running simultaneously, pegging each core > > > > at 99% cpu. > > > > > > I would say it is perfectly normal. Compiling the catalog is a hard and > > > complex problem and requires CPU. > > > > > > The difference between 0.24.8 and 2.6 (or 0.25 for what matters) is > that > > > some performance issues have been fixed. Those issues made the master > be > > > more I/O bound under 0.24, but now mostly CPU bound in later versions. > > > > If we were talking about only cpu usage, I would agree with you. But > > in this case, the load average of the machine has gone up over 5x. > > And as high load average indicates processes not getting enough > > runtime, in this case it is an indication to me that 2.6 is performing > > worse than 0.24 (previously, on average, all processes got enough > > runtime and did not have to wait for system resources, now processes > > are sitting in the run queue, waiting to get a chance to run) > > Load is not necessarily an indication of an issue. It can also mean some > tasks are waiting for I/O not only CPU. > The only real issue under load is if service time is beyond an > admissible value, otherwise you can't say it's bad or not. > If you see some hosts reporting timeouts, then it's an indication that > service time is not good :) > > BTW, do you run your mysql storedconfig instance on the same server? > You can activate thin_storeconfigs to reduce the load on the mysql db. > > > > > > > I don't really get what is the issue about using 100% of CPU? > > Thats not the issue, just an indication of what is causing it > > > > > > > > You're paying about the same price when your CPU is used and when it's > > > idle, so that shouldn't make a difference :) > > Generally true, but this is a on VM which is also running some of my > > radius and proxy instances, amongst others. > > > > > > > > If that's an issue, reduce the concurrency of your setup (run less > > > compilation in parallel, implement splay time, etc...). > > splay has been enabled since 0.24 > > > > My apache maxclients is set to 15 to limit concurrency. > > I think this is too many except if you have 8 cores. As Trevor said in > another e-mail in this thread, 2PM/Core is the best. > > Now it all depends on your number of nodes and sleeptime. I suggest you > use ext/puppet-load to find your setup real concurrency. > -- > Brice Figureau > Follow the latest Puppet Community evolutions on www.planetpuppet.org! > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-us...@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com<puppet-users%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
