Issue #21835 has been updated by Christian Flamm.
Plz **ignore result 2** from the previous post - I believe something went wrong
during this test run.
I've created another setup with just one puppetmaster and one puppet agent.
High system load created by serving other agents in parallel is not an issue in
this isolated setup (there aren't any). Agents catalog is the biggest one of
our catalogs (1000+ resources).
- **Scenario (A)** 1 puppetmaster and 1 agent both at v3.2.3:
`config_retrieval` time between **25sec - 35sec**
Most interestingly looking lines are:
PROFILE [70082058459200] 1.4.4.1 Called include: took 4.7933 seconds
PROFILE [70082058459200] 1.4.4.3 Called include: took 13.8960 seconds
PROFILE [70082058459200] 1.4.6.1.1 Evaluated definitions: took 3.9402
seconds
PROFILE [70082058459200] 1.4.6.2.1 Evaluated definitions: took 1.4371
seconds
PROFILE [70082058459200] 1.6 Rendered result in
Puppet::Network::Format[pson]: took 1.6592 seconds
PROFILE [70082043372920] 1 Processed request PUT
/production/report/<puppetagent>: took 6.2024 seconds
I guess at least 1.4.* is part of `config_retrieval`?
- **Scenario (B)** 1 puppetmaster and 1 agent both at v3.1.1:
`config_retrieval` time between **10sec - 15sec**
A shame I can't have profiling output, but what I guess must be recognized is
that **1.4.4.3** of Scenario (A) is almost whole `config_retrieval` time or
sometimes even more than `config_retrieval` in Scenario (B).
----------------------------------------
Bug #21835: Heavy load increase after upgrading to Puppet 3.2
https://projects.puppetlabs.com/issues/21835#change-95358
* Author: Christian Flamm
* Status: Needs More Information
* Priority: Normal
* Assignee: Charlie Sharpsteen
* Category:
* Target version:
* Affected Puppet version: 3.2.3
* Keywords: puppet rack passenger performance load
* Branch:
----------------------------------------
After upgrading from Puppet 3.1.1 to 3.2.x (x in {1,2,3}) I noticed a huge load
increase on the puppetmaster machine. It's heavy enough that for every new
3.2.x version I tried it spooked me and I ran back to 3.1.1 as quickly as I
could.
Using apache/rack/passenger on an 8 core puppet master machine its load is
usually between 3 and 4 (using 3.1.1). After the upgrade the load jumps to
values between 7 and 12 (using 3.2.x). This overload situation then causes
everything to slow down massively - especially agent's config_retrieval time
(factor 2-3).
Please have a look a the attached diagrams. Those diagrams each show 100
measures every 10s. The only thing that has changed between these two
situations is an upgrade of puppet-server and puppet from 3.1.1 to 3.2.3. The
values are:
- **Load** ... load, 1 minute average
- **\#Passenger Processes** ... output of 'passenger-status', "General
information", "Processes"
- **\#Clients ... output** of 'passenger-status --show=requests', "Clients"
Does puppet 3.2 contain any new processing-heavy features that are enabled per
default?
> uname -a
Linux <some agent> 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/redhat-release
CentOS release 6.3 (Final)
> rpm -qa | egrep "puppet|ruby|httpd"
rubygem-rake-0.8.7-2.1.el6.noarch
ruby-mysql-2.8.2-1.el6.x86_64
libselinux-ruby-2.0.94-5.3.el6.x86_64
rubygem-fastthread-1.0.7-2.el6.x86_64
rubygem-mongrel-1.1.5-3.el6.x86_64
ruby-libs-1.8.7.352-10.el6_4.x86_64
ruby-irb-1.8.7.352-10.el6_4.x86_64
httpd-2.2.15-26.el6.centos.x86_64
puppetlabs-release-6-7.noarch
puppet-3.1.1-1.el6.noarch
ruby-augeas-0.4.1-1.el6.x86_64
ruby-shadow-1.4.1-13.el6.x86_64
rubygems-1.3.7-1.el6.noarch
rubygem-gem_plugin-0.2.3-3.el6.noarch
rubygem-daemons-1.0.10-2.el6.noarch
ruby-1.8.7.352-10.el6_4.x86_64
ruby-rdoc-1.8.7.352-10.el6_4.x86_64
httpd-tools-2.2.15-26.el6.centos.x86_64
httpd-devel-2.2.15-26.el6.centos.x86_64
ruby-devel-1.8.7.352-10.el6_4.x86_64
puppet-dashboard-1.2.23-1.el6.noarch
ruby-rgen-0.6.5-1.el6.noarch
puppet-server-3.1.1-1.el6.noarch
rubygem-json-1.4.6-1.el6.x86_64
> gem list
*** LOCAL GEMS ***
cgi_multipart_eof_fix (2.5.0)
daemon_controller (1.1.2)
daemons (1.1.9, 1.0.10)
fastthread (1.0.7)
gem_plugin (0.2.3)
json (1.4.6)
mongrel (1.1.5)
passenger (4.0.10, 4.0.5, 3.0.19)
rack (1.5.2)
rake (10.0.4, 0.8.7)
> cat /etc/httpd/conf.d/puppetmaster
LoadModule passenger_module
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.10/buildout/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10
PassengerDefaultRuby /usr/bin/ruby
PassengerStatThrottleRate 120
PassengerHighPerformance On
PassengerMaxPoolSize 12
PassengerMaxRequests 1000
PassengerPoolIdleTime 600
Listen 8140
<VirtualHost *:8140>
SSLEngine On
# Only allow high security cryptography. Alter if needed for
compatibility.
SSLProtocol All -SSLv2
SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/<puppetmaster>.lan.pem
SSLCertificateKeyFile
/var/lib/puppet/ssl/private_keys/<puppetmaster>.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
<Directory /usr/share/puppet/rack/puppetmasterd/>
Options None
AllowOverride None
Order Allow,Deny
Allow from All
</Directory>
</VirtualHost>
> cat /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
modulepath = /opt/xxx/dev/puppet/modules
manifestdir = /opt/xxx/dev/puppet/manifests
manifest = /opt/xxx/dev/puppet/manifests/site.pp
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = <puppetmaster>
report = true
splaylimit = 0
runinterval = 30
[master]
certname=<puppetmaster>
reports = http,log,tagmail
reportdir = /var/lib/puppet/reports/upload
reporturl = http://<puppetmaster>:3000/reports
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/groups/opt_out.
