Chris, Hiera 3 + hiera-eyaml may also be contributing to the slowness. Here is one ticket (related to SERVER-1922) that indicated moving to hiera 5 improved compile times substantially: https://tickets.puppetlabs.com/browse/SERVER-1919
-Haus On Tue, Jan 9, 2018 at 11:36 AM, Matthaus Owens <[email protected]> wrote: > Chris, > To better help you, it would be great to know a few more things about > your installation. First question: are you running puppetserver 5.0.0 > or something later in the 5.x series (and is it the same on all > servers)? Second, what version of the puppet-agent are on those > servers? puppetserver 5.1.3 included a fix for > https://tickets.puppetlabs.com/browse/SERVER-1922 which should improve > performance some. > To dig into what may be causing the compiles to be slower, I would > recommend first checking out the client metrics. > https://puppet.com/docs/puppetserver/5.1/http_client_metrics.html has > some details, and I would be interested in the client metrics that > page lists under the /puppet/v3/catalog. They are PuppetDB related > requests, and as that was also upgraded alongside puppetserver it > would be good to eliminate PuppetDB as a contributor. PuppetDB > slowness can show up as slow catalog compiles, which in turn will hold > jrubies for longer and might explain some of what you are seeing. > > On Mon, Jan 8, 2018 at 7:31 PM, chris smith <[email protected]> wrote: >> Hi there, >> >> I recently did an upgrade from puppetserver 2.7.2 to puppetserver 5.0 and >> performance has bottomed out pretty terribly. Agents and puppetdb also got >> updated. Compiling the catalog on the server used to take 10-20 seconds, now >> they are taking 90-150 seconds and agent runs are taking 30+ minutes (used >> to be a couple of minutes). >> >> The architecture is distributed, with: >> * a central ca, running puppetserver, puppetdb, postgres 9.6 >> * separate puppetservers replicated in other datacentres. These are also >> running puppetdb, pointing writes to the central ca, but reads go to a >> locally replicated database >> >> Other servers (agents) point to the replicated puppetservers to do all of >> the work. >> >> The puppetservers were tuned (upped jvm memory, set max-instances). >> >> The architecture hasn't changed since the upgrade. >> >> The puppetservers are still running hiera3 configs, they haven't been >> converted yet (it's on the list, but from what I read it wasn't a >> showstopper). We have a reasonable amount of encrypted yaml files (using >> hiera-eyaml-gpg), though this was the same as pre-upgrade and hasn't changed >> significantly. >> >> Since the upgrade, I've tried re-tuning the jvm settings, changing >> max-instances and not having much luck at all. I found the experimental >> dashboard on the puppetservers and they show that there are no free jruby's, >> but there has to be something I'm missing that's causing that to happen. >> >> I'm lost on what to look at next, is there an easy way to peak inside jruby >> to see what's taking so long? >> >> Any suggestions would be great. >> >> Cheers, >> Chris. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/4d0dc37f-c07e-4f8c-8323-44a90d68b208%40googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACD%3DwAfJveNpAzekCvjeP8aZ7ma4mD4%2B1KC9JKE4mqELbktB4g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
