ok, so I have been tinkering a bit more with this and I my problems start when I upgrade to puppet 3.7.5. So 3.7.4 works fine, but 3.7.5 and 3.8.x is showing some really terrible performance for me.
Running puppetserver (jvm) with 3.7.4, 9 jruby instances and 12G Xmx I get really good performance, catalog compilation times ranging from 1-3s. Upgrade to 3.7.5, restart puppetserver, and the compilation takes forever (40-90s) which in turn causes the server to get overloaded with puppet agents which causes a lot of failed runs. There only error messages in the logs are "java.nio.channels.WritePendingException: null" which probably is some timeout error caused by the excessive load on the server. I can't see anything in the release notes for 3.7.5 that would explain this, but someone out there is sure to have a better eye for that (I hope). I also added this to https://tickets.puppetlabs.com/browse/SERVER-819, since it seems to be a similar problem even though I do not use puppetdb. I'm hoping that someone out there can think of any reason for my problems. Regards Adam On Thursday, September 17, 2015 at 7:03:50 AM UTC+2, Adam Winberg wrote: > > I actually tried the --profile option but never got it to actually record > any 'PROFILE' log data. The logdest and debug flags works as expected, but > no 'PROFILE' entries in the logfile. > > On Thu, Sep 17, 2015 at 1:02 AM, Josh Cooper <[email protected]> wrote: > >> Hi Adam, >> >> On Wed, Sep 16, 2015 at 3:47 AM, Adam Winberg <[email protected]> >> wrote: >> >>> Upgraded from 3.7.3 to 3.8.2 and have after that had very slow >>> compilation and file serving from the puppet master. We're running our >>> puppetmaster with apache/passenger, ca. 900 hosts. Complete puppet runs >>> earlier took 10-15s and after update take 30-300s. Very unstable >>> performance. >>> >>> Even running with just a few clients the runtimes are really slow. >>> Decreasing the 'PassengerMaxPoolSize' makes the runs faster, but then it >>> cant cope with an increased load. >>> >>> I have also tried with the jvm puppetserver with roughly the same >>> performance (or lack thereof). I noticed the same pattern, if i set >>> 'max-active-instances' in puppetserver.conf to 2, I get acceptable >>> performance with a handful of clients, but with increased load it doesnt >>> cope. If I raise the 'max-active-instances' the performance is even worse. >>> >>> This is running on an vmware server, 8 cpu and 12GB RAM. >>> >>> Cant find anything in the release notes that explain this performance >>> drops, does anyone recognize the problem? >>> >>> I have now downgraded to 3.7.3 and my puppet runs are as fast as ever >>> again. >>> >>> Any help appreciated. >>> >>> Regards >>> Adam >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> I would take a look at the `--profile` option as described in >> https://puppetlabs.com/blog/tune-puppet-performance-profiler, making >> sure to update config.ru as needed. >> >> Josh >> >> -- >> Josh Cooper >> Developer, Puppet Labs >> >> *PuppetConf 2015 <http://2015.puppetconf.com/>** is right around the >> corner! Join us October 5-9 in Portland, OR. **Register now >> <https://puppetconf2015.eventbrite.com/>**.* >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/95f54835-40bf-4b9e-9ece-e9c0187876be%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
