Issue #21835 has been updated by Christian Flamm.
Charlie Sharpsteen wrote:
> If the performance hit is re-producible using a single agent checking into a
> master running under webrick, then using `git bisect` may be a quick way to
> nail down what is causing this---especially since we know it had to be
> something between 3.1.1 and 3.2.0. There was a period of time around 3.2.0
> where Passenger support was bent out of shape, so bisecting under webrick
> would be recommended in this case.
Problem is not apache/rack/passenger specific. Just tested using plain old
webrick: the `config_retrieval` value is **~30sec vs. ~4sec**.
- `puppet-3.2.3-1.el6` and `puppet-server-3.2.3-1.el6`
events:
failure: 0
success: 0
total: 0
resources:
skipped: 6
out_of_sync: 0
failed_to_restart: 0
failed: 0
restarted: 0
total: 1038
scheduled: 0
changed: 0
time:
service: 1.079802
package: 0.59133
cron: 0.004248
group: 0.008652
exec: 2.515222
last_run: 1375197060
config_retrieval: 30.381078004837 <--
user: 0.025736
file: 3.699435
filebucket: 0.000345
total: 38.315582004837
mount: 0.009734
version:
puppet: "3.2.3"
config: 1375197002
changes:
total: 0
- `puppet-3.1.1-1.el6` and `puppet-server-3.1.1-1.el6`
changes:
total: 0
time:
group: 0.007988
filebucket: 0.000176
user: 0.018322
service: 0.473722
cron: 0.00279
package: 0.351063
last_run: 1375200204
exec: 2.869731
mount: 0.007087
file: 2.7071
total: 10.5817079714813
config_retrieval: 4.14372897148132 <--
version:
puppet: "3.1.1"
config: 1375197002
events:
success: 0
failure: 0
total: 0
resources:
failed: 0
changed: 0
scheduled: 0
out_of_sync: 0
skipped: 6
restarted: 0
failed_to_restart: 0
total: 1038
Since there's no profiling debug output for Puppet 3.1.1 I guess it doesn't
make sense to upload server debug output... just can't really be compared
(keeping them anyway). What's your recommendation about how to proceed?
Btw: tiny correction (may be not important): first version I've tried was
3.2.**1**.
----------------------------------------
Bug #21835: Heavy load increase after upgrading to Puppet 3.2
https://projects.puppetlabs.com/issues/21835#change-95718
* Author: Christian Flamm
* Status: Needs More Information
* Priority: Normal
* Assignee: Christian Flamm
* Category: compiler
* Target version:
* Affected Puppet version: 3.2.3
* Keywords: rack passenger performance load compile
* Branch:
----------------------------------------
After upgrading from Puppet 3.1.1 to 3.2.x (x in {1,2,3}) I noticed a huge load
increase on the puppetmaster machine. It's heavy enough that for every new
3.2.x version I tried it spooked me and I ran back to 3.1.1 as quickly as I
could.
Using apache/rack/passenger on an 8 core puppet master machine its load is
usually between 3 and 4 (using 3.1.1). After the upgrade the load jumps to
values between 7 and 12 (using 3.2.x). This overload situation then causes
everything to slow down massively - especially agent's config_retrieval time
(factor 2-3).
Please have a look a the attached diagrams. Those diagrams each show 100
measures every 10s. The only thing that has changed between these two
situations is an upgrade of puppet-server and puppet from 3.1.1 to 3.2.3. The
values are:
- **Load** ... load, 1 minute average
- **\#Passenger Processes** ... output of 'passenger-status', "General
information", "Processes"
- **\#Clients ... output** of 'passenger-status --show=requests', "Clients"
Does puppet 3.2 contain any new processing-heavy features that are enabled per
default?
> uname -a
Linux <some agent> 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/redhat-release
CentOS release 6.3 (Final)
> rpm -qa | egrep "puppet|ruby|httpd"
rubygem-rake-0.8.7-2.1.el6.noarch
ruby-mysql-2.8.2-1.el6.x86_64
libselinux-ruby-2.0.94-5.3.el6.x86_64
rubygem-fastthread-1.0.7-2.el6.x86_64
rubygem-mongrel-1.1.5-3.el6.x86_64
ruby-libs-1.8.7.352-10.el6_4.x86_64
ruby-irb-1.8.7.352-10.el6_4.x86_64
httpd-2.2.15-26.el6.centos.x86_64
puppetlabs-release-6-7.noarch
puppet-3.1.1-1.el6.noarch
ruby-augeas-0.4.1-1.el6.x86_64
ruby-shadow-1.4.1-13.el6.x86_64
rubygems-1.3.7-1.el6.noarch
rubygem-gem_plugin-0.2.3-3.el6.noarch
rubygem-daemons-1.0.10-2.el6.noarch
ruby-1.8.7.352-10.el6_4.x86_64
ruby-rdoc-1.8.7.352-10.el6_4.x86_64
httpd-tools-2.2.15-26.el6.centos.x86_64
httpd-devel-2.2.15-26.el6.centos.x86_64
ruby-devel-1.8.7.352-10.el6_4.x86_64
puppet-dashboard-1.2.23-1.el6.noarch
ruby-rgen-0.6.5-1.el6.noarch
puppet-server-3.1.1-1.el6.noarch
rubygem-json-1.4.6-1.el6.x86_64
> gem list
*** LOCAL GEMS ***
cgi_multipart_eof_fix (2.5.0)
daemon_controller (1.1.2)
daemons (1.1.9, 1.0.10)
fastthread (1.0.7)
gem_plugin (0.2.3)
json (1.4.6)
mongrel (1.1.5)
passenger (4.0.10, 4.0.5, 3.0.19)
rack (1.5.2)
rake (10.0.4, 0.8.7)
> cat /etc/httpd/conf.d/puppetmaster
LoadModule passenger_module
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.10/buildout/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10
PassengerDefaultRuby /usr/bin/ruby
PassengerStatThrottleRate 120
PassengerHighPerformance On
PassengerMaxPoolSize 12
PassengerMaxRequests 1000
PassengerPoolIdleTime 600
Listen 8140
<VirtualHost *:8140>
SSLEngine On
# Only allow high security cryptography. Alter if needed for
compatibility.
SSLProtocol All -SSLv2
SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/<puppetmaster>.lan.pem
SSLCertificateKeyFile
/var/lib/puppet/ssl/private_keys/<puppetmaster>.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
<Directory /usr/share/puppet/rack/puppetmasterd/>
Options None
AllowOverride None
Order Allow,Deny
Allow from All
</Directory>
</VirtualHost>
> cat /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
modulepath = /opt/xxx/dev/puppet/modules
manifestdir = /opt/xxx/dev/puppet/manifests
manifest = /opt/xxx/dev/puppet/manifests/site.pp
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = <puppetmaster>
report = true
splaylimit = 0
runinterval = 30
[master]
certname=<puppetmaster>
reports = http,log,tagmail
reportdir = /var/lib/puppet/reports/upload
reporturl = http://<puppetmaster>:3000/reports
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/groups/opt_out.
