Issue #21835 has been reported by Christian Flamm.
----------------------------------------
Bug #21835: Heavy load increase after upgrading to Puppet 3.2
https://projects.puppetlabs.com/issues/21835
* Author: Christian Flamm
* Status: Unreviewed
* Priority: Normal
* Assignee:
* Category:
* Target version:
* Affected Puppet version: 3.2.3
* Keywords: puppet rack passenger performance load
* Branch:
----------------------------------------
After upgrading from Puppet 3.1.1 to 3.2.x (x in {1,2,3}) I noticed a huge load
increase on the puppetmaster machine. It's heavy enough that for every new
3.2.x version I tried it spooked me and I ran back to 3.1.1 as quickly as I
could.
Using apache/rack/passenger on an 8 core puppet master machine its load is
usually between 3 and 4 (using 3.1.1). After the upgrade the load jumps to
values between 7 and 12 (using 3.2.x). This overload situation then causes
everything to slow down massively - especially agent's config_retrieval time
(factor 2-3).
Please have a look a the attached diagrams. Those diagrams each show 100
measures every 10s. The only thing that has changed between these two
situations is an upgrade of puppet-server and puppet from 3.1.1 to 3.2.3. The
values are:
- **Load** ... load, 1 minute average
- **\#Passenger Processes** ... output of 'passenger-status', "General
information", "Processes"
- **\#Clients ... output** of 'passenger-status --show=requests', "Clients"
Does puppet 3.2 contain any new processing-heavy features that are enabled per
default?
> uname -a
Linux <some agent> 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/redhat-release
CentOS release 6.3 (Final)
> rpm -qa | egrep "puppet|ruby|httpd"
rubygem-rake-0.8.7-2.1.el6.noarch
ruby-mysql-2.8.2-1.el6.x86_64
libselinux-ruby-2.0.94-5.3.el6.x86_64
rubygem-fastthread-1.0.7-2.el6.x86_64
rubygem-mongrel-1.1.5-3.el6.x86_64
ruby-libs-1.8.7.352-10.el6_4.x86_64
ruby-irb-1.8.7.352-10.el6_4.x86_64
httpd-2.2.15-26.el6.centos.x86_64
puppetlabs-release-6-7.noarch
puppet-3.1.1-1.el6.noarch
ruby-augeas-0.4.1-1.el6.x86_64
ruby-shadow-1.4.1-13.el6.x86_64
rubygems-1.3.7-1.el6.noarch
rubygem-gem_plugin-0.2.3-3.el6.noarch
rubygem-daemons-1.0.10-2.el6.noarch
ruby-1.8.7.352-10.el6_4.x86_64
ruby-rdoc-1.8.7.352-10.el6_4.x86_64
httpd-tools-2.2.15-26.el6.centos.x86_64
httpd-devel-2.2.15-26.el6.centos.x86_64
ruby-devel-1.8.7.352-10.el6_4.x86_64
puppet-dashboard-1.2.23-1.el6.noarch
ruby-rgen-0.6.5-1.el6.noarch
puppet-server-3.1.1-1.el6.noarch
rubygem-json-1.4.6-1.el6.x86_64
> gem list
*** LOCAL GEMS ***
cgi_multipart_eof_fix (2.5.0)
daemon_controller (1.1.2)
daemons (1.1.9, 1.0.10)
fastthread (1.0.7)
gem_plugin (0.2.3)
json (1.4.6)
mongrel (1.1.5)
passenger (4.0.10, 4.0.5, 3.0.19)
rack (1.5.2)
rake (10.0.4, 0.8.7)
> cat /etc/httpd/conf.d/puppetmaster
LoadModule passenger_module
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.10/buildout/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10
PassengerDefaultRuby /usr/bin/ruby
PassengerStatThrottleRate 120
PassengerHighPerformance On
PassengerMaxPoolSize 12
PassengerMaxRequests 1000
PassengerPoolIdleTime 600
Listen 8140
<VirtualHost *:8140>
SSLEngine On
# Only allow high security cryptography. Alter if needed for
compatibility.
SSLProtocol All -SSLv2
SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/<puppetmaster>.lan.pem
SSLCertificateKeyFile
/var/lib/puppet/ssl/private_keys/<puppetmaster>.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
<Directory /usr/share/puppet/rack/puppetmasterd/>
Options None
AllowOverride None
Order Allow,Deny
Allow from All
</Directory>
</VirtualHost>
> cat /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
modulepath = /opt/xxx/dev/puppet/modules
manifestdir = /opt/xxx/dev/puppet/manifests
manifest = /opt/xxx/dev/puppet/manifests/site.pp
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = <puppetmaster>
report = true
splaylimit = 0
runinterval = 30
[master]
certname=<puppetmaster>
reports = http,log,tagmail
reportdir = /var/lib/puppet/reports/upload
reporturl = http://<puppetmaster>:3000/reports
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/groups/opt_out.
