Chris, Good to know that PuppetDB isn't the cause here. One thing worth doing on your servers is moving forward to Puppet-agent 5.3.2. There was a performance regression in Puppet 5 https://tickets.puppetlabs.com/browse/PUP-8009 around internationalization and modules (something like 30% in some cases). You might also need to set `disable_i18n=true` in the [master] section of your puppet.conf. To see what puppetserver is doing and how long it takes you can set `profile=true` in the [master] section of puppet.conf (I would recommend only doing this on one of your servers, as it will generate a lot of log output per puppet run). The profiling results will be logged at the info level to /var/log/puppetlabs/puppetserver/puppetserver.log, so you could then inspect the timing of different parts of the agent lifecycle.
On Tue, Jan 9, 2018 at 7:11 PM, Chris Smith <[email protected]> wrote: > Hi, > > Thanks for your help. > > On 10/01/18 06:36, Matthaus Owens wrote: >> Chris, >> To better help you, it would be great to know a few more things about >> your installation. First question: are you running puppetserver 5.0.0 >> or something later in the 5.x series (and is it the same on all >> servers)? Second, what version of the puppet-agent are on those >> servers? puppetserver 5.1.3 included a fix for >> https://tickets.puppetlabs.com/browse/SERVER-1922 which should improve >> performance some. > > Hm. Interesting, thanks. I'll check out what a 5.0 -> 5.1 upgrade will do. > >> >> Hiera 3 + hiera-eyaml may also be contributing to the slowness. Here >> is one ticket (related to SERVER-1922) that indicated moving to hiera >> 5 improved compile times substantially: >> https://tickets.puppetlabs.com/browse/SERVER-1919 > > Also interesting but as noted on the last comment, a lot of the > structure was changed so that might not all have been hiera3 -> hiera5. > >> To dig into what may be causing the compiles to be slower, I would >> recommend first checking out the client metrics. >> https://puppet.com/docs/puppetserver/5.1/http_client_metrics.html has >> some details, and I would be interested in the client metrics that >> page lists under the /puppet/v3/catalog. They are PuppetDB related >> requests, and as that was also upgraded alongside puppetserver it >> would be good to eliminate PuppetDB as a contributor. PuppetDB >> slowness can show up as slow catalog compiles, which in turn will hold >> jrubies for longer and might explain some of what you are seeing. > > puppetservers are all the same. > > We upgraded to: > # /opt/puppetlabs/server/bin/puppetserver -v > puppetserver version: 5.0.0 > > puppetdb is this, it should have been 5.0 as well but I stuffed it up. > # /opt/puppetlabs/server/bin/puppetdb -v > puppetdb version: 5.1.3 > > > agents are all: > # /opt/puppetlabs/puppet/bin/puppet --version > 5.0.0 > > > The metrics say > > { > "route-id": "puppet-v3-file_metadata-/*/", > "count": 9373, > "mean": 10217, > "aggregate": 95763941 > }, > { > "route-id": "puppet-v3-catalog-/*/", > "count": 828, > "mean": 94773, > "aggregate": 78472044 > }, > { > "route-id": "puppet-v3-node-/*/", > "count": 831, > "mean": 62709, > "aggregate": 52111179 > }, > { > "route-id": "puppet-v3-file_metadatas-/*/", > "count": 4714, > "mean": 9288, > "aggregate": 43783632 > }, > { > "route-id": "puppet-v3-report-/*/", > "count": 780, > "mean": 3433, > "aggregate": 2677740 > }, > > > > "http-client-metrics": [ > { > "count": 821, > "mean": 48, > "aggregate": 39408, > "metric-name": > "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.command.replace_catalog.full-response", > "metric-id": [ > "puppetdb", > "command", > "replace_catalog" > ] > }, > { > "count": 832, > "mean": 25, > "aggregate": 20800, > "metric-name": > "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.command.replace_facts.full-response", > "metric-id": [ > "puppetdb", > "command", > "replace_facts" > ] > }, > { > "count": 780, > "mean": 19, > "aggregate": 14820, > "metric-name": > "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.command.store_report.full-response", > "metric-id": [ > "puppetdb", > "command", > "store_report" > ] > }, > { > "count": 215, > "mean": 43, > "aggregate": 9245, > "metric-name": > "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.facts.find.full-response", > "metric-id": [ > "puppetdb", > "facts", > "find" > ] > } > ] > > > So I think that's showing it's quick to pass it off to puppetdb when > it's storing changes. Yea none of those client metrics are worrying, they are all less than 50ms. > > puppetdb logs are telling me that 'replace catalog' is taking 2-3 > seconds, and 'replace facts' is taking 10-20 seconds (previous puppetdb > wasn't logging the time taken, so I can't compare). > > I tried changing puppetdb logging to debug but it doesn't tell me what > it's doing with those 'replace' commands (I don't think, I might've > missed it). I haven't found a way to manually process one of those > files, do you know if there is a way to do it? I don't know offhand unfortunately. > > I've set up postgres logging to alert for queries over 200ms (both on > the primary & replica) and I get very little (a couple of queries every > now and then), so I don't think it's the database. > > > Cheers, > Chris. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/1b549321-8db4-c352-5b75-ebe8eb0e5a79%40gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACD%3DwAfXMXa89Jjy-vf_2baqERsvyGvJJ85_Qsu19smEU4hDgA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
