for posterity - a new puppetlabs ticket was created for this, https://tickets.puppetlabs.com/browse/PUP-5380, where the root cause of the problem is pinpointed.
On Tue, Oct 13, 2015 at 3:34 PM, Adam Winberg <[email protected]> wrote: > ok, so I have been tinkering a bit more with this and I my problems start > when I upgrade to puppet 3.7.5. So 3.7.4 works fine, but 3.7.5 and 3.8.x is > showing some really terrible performance for me. > > Running puppetserver (jvm) with 3.7.4, 9 jruby instances and 12G Xmx I get > really good performance, catalog compilation times ranging from 1-3s. > Upgrade to 3.7.5, restart puppetserver, and the compilation takes forever > (40-90s) which in turn causes the server to get overloaded with puppet > agents which causes a lot of failed runs. There only error messages in the > logs are > "java.nio.channels.WritePendingException: null" > which probably is some timeout error caused by the excessive load on the > server. > > I can't see anything in the release notes for 3.7.5 that would explain > this, but someone out there is sure to have a better eye for that (I hope). > > I also added this to https://tickets.puppetlabs.com/browse/SERVER-819, > since it seems to be a similar problem even though I do not use puppetdb. > I'm hoping that someone out there can think of any reason for my problems. > > Regards > Adam > > > > On Thursday, September 17, 2015 at 7:03:50 AM UTC+2, Adam Winberg wrote: >> >> I actually tried the --profile option but never got it to actually record >> any 'PROFILE' log data. The logdest and debug flags works as expected, but >> no 'PROFILE' entries in the logfile. >> >> On Thu, Sep 17, 2015 at 1:02 AM, Josh Cooper <[email protected]> wrote: >> >>> Hi Adam, >>> >>> On Wed, Sep 16, 2015 at 3:47 AM, Adam Winberg <[email protected]> >>> wrote: >>> >>>> Upgraded from 3.7.3 to 3.8.2 and have after that had very slow >>>> compilation and file serving from the puppet master. We're running our >>>> puppetmaster with apache/passenger, ca. 900 hosts. Complete puppet runs >>>> earlier took 10-15s and after update take 30-300s. Very unstable >>>> performance. >>>> >>>> Even running with just a few clients the runtimes are really slow. >>>> Decreasing the 'PassengerMaxPoolSize' makes the runs faster, but then it >>>> cant cope with an increased load. >>>> >>>> I have also tried with the jvm puppetserver with roughly the same >>>> performance (or lack thereof). I noticed the same pattern, if i set >>>> 'max-active-instances' in puppetserver.conf to 2, I get acceptable >>>> performance with a handful of clients, but with increased load it doesnt >>>> cope. If I raise the 'max-active-instances' the performance is even worse. >>>> >>>> This is running on an vmware server, 8 cpu and 12GB RAM. >>>> >>>> Cant find anything in the release notes that explain this performance >>>> drops, does anyone recognize the problem? >>>> >>>> I have now downgraded to 3.7.3 and my puppet runs are as fast as ever >>>> again. >>>> >>>> Any help appreciated. >>>> >>>> Regards >>>> Adam >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> I would take a look at the `--profile` option as described in >>> https://puppetlabs.com/blog/tune-puppet-performance-profiler, making >>> sure to update config.ru as needed. >>> >>> Josh >>> >>> -- >>> Josh Cooper >>> Developer, Puppet Labs >>> >>> *PuppetConf 2015 <http://2015.puppetconf.com/>** is right around the >>> corner! Join us October 5-9 in Portland, OR. **Register now >>> <https://puppetconf2015.eventbrite.com/>**.* >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com >>> <https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/95f54835-40bf-4b9e-9ece-e9c0187876be%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/95f54835-40bf-4b9e-9ece-e9c0187876be%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFCL3Hug%3D0pZ79D9Xdotxi8YPivxi97%3D_F1muW3gRw_YGGdgCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
