Issue #21835 has been updated by Charlie Sharpsteen.
Christian Flamm wrote: > Since there's no profiling debug output for Puppet 3.1.1 I guess it doesn't > make sense to upload server debug output... just can't really be compared > (keeping them anyway). What's your recommendation about how to proceed? I would try a git bisect with the puppet master running under webrick. This has a chance of nailing down the exact commit that caused the performance to change. The setup goes like this: <pre> # Halt the currently running puppet master service puppetmaster stop # Or stop passenger if running under that. # Ensure you have git installed yum install -y git # Check out puppet source code into a subdirectory mkdir puppetlabs (cd puppetlabs && git clone https://github.com/puppetlabs/puppet) # Add a stub to profile.d that shims the Puppet checkout into the PATH and RUBYLIB tee /etc/profile.d/envpuppet.sh << EOF export ENVPUPPET_BASEDIR=$PWD/puppetlabs eval \$(\$ENVPUPPET_BASEDIR/puppet/ext/envpuppet) EOF # Re-load the shell exec bash -i </pre> Now, you are ready to bisect: <pre> cd puppetlabs/puppet # Step 0: Start a bisect between versions 3.2.1 and 3.1.1 git bisect start 3.2.1 3.1.1 # Step 1: Start the puppet master puppet master --debug --trace --no-daemonize # Step 2: Run the agent and note the catalog compilation time. # Step 3: Kill the master with CTRL-C # If compilation time from Step 2 was excessive: git bisect bad # Otherwise: git bisect good # Go to Step 1. </pre> For changes between 3.2.1 and 3.1.1, that process should converge in ~9 iterations. To clean up after bisecting: <pre> # Remove the envpuppet.sh shim rm /etc/profile.d/envpuppet.sh # Remove the puppet source checkout # Restart the regular Puppet master. </pre> > Btw: tiny correction (may be not important): first version I've tried was > 3.2.**1**. Thanks for confirming the earliest version you are observing this with. We made some changes to manifest loading in 3.2.3, but if you are seeing the load increase with 3.2.1, then we can rule that out. ---------------------------------------- Bug #21835: Heavy load increase after upgrading to Puppet 3.2 https://projects.puppetlabs.com/issues/21835#change-95736 * Author: Christian Flamm * Status: Needs More Information * Priority: Normal * Assignee: Christian Flamm * Category: compiler * Target version: * Affected Puppet version: 3.2.3 * Keywords: rack passenger performance load compile * Branch: ---------------------------------------- After upgrading from Puppet 3.1.1 to 3.2.x (x in {1,2,3}) I noticed a huge load increase on the puppetmaster machine. It's heavy enough that for every new 3.2.x version I tried it spooked me and I ran back to 3.1.1 as quickly as I could. Using apache/rack/passenger on an 8 core puppet master machine its load is usually between 3 and 4 (using 3.1.1). After the upgrade the load jumps to values between 7 and 12 (using 3.2.x). This overload situation then causes everything to slow down massively - especially agent's config_retrieval time (factor 2-3). Please have a look a the attached diagrams. Those diagrams each show 100 measures every 10s. The only thing that has changed between these two situations is an upgrade of puppet-server and puppet from 3.1.1 to 3.2.3. The values are: - **Load** ... load, 1 minute average - **\#Passenger Processes** ... output of 'passenger-status', "General information", "Processes" - **\#Clients ... output** of 'passenger-status --show=requests', "Clients" Does puppet 3.2 contain any new processing-heavy features that are enabled per default? > uname -a Linux <some agent> 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux > cat /etc/redhat-release CentOS release 6.3 (Final) > rpm -qa | egrep "puppet|ruby|httpd" rubygem-rake-0.8.7-2.1.el6.noarch ruby-mysql-2.8.2-1.el6.x86_64 libselinux-ruby-2.0.94-5.3.el6.x86_64 rubygem-fastthread-1.0.7-2.el6.x86_64 rubygem-mongrel-1.1.5-3.el6.x86_64 ruby-libs-1.8.7.352-10.el6_4.x86_64 ruby-irb-1.8.7.352-10.el6_4.x86_64 httpd-2.2.15-26.el6.centos.x86_64 puppetlabs-release-6-7.noarch puppet-3.1.1-1.el6.noarch ruby-augeas-0.4.1-1.el6.x86_64 ruby-shadow-1.4.1-13.el6.x86_64 rubygems-1.3.7-1.el6.noarch rubygem-gem_plugin-0.2.3-3.el6.noarch rubygem-daemons-1.0.10-2.el6.noarch ruby-1.8.7.352-10.el6_4.x86_64 ruby-rdoc-1.8.7.352-10.el6_4.x86_64 httpd-tools-2.2.15-26.el6.centos.x86_64 httpd-devel-2.2.15-26.el6.centos.x86_64 ruby-devel-1.8.7.352-10.el6_4.x86_64 puppet-dashboard-1.2.23-1.el6.noarch ruby-rgen-0.6.5-1.el6.noarch puppet-server-3.1.1-1.el6.noarch rubygem-json-1.4.6-1.el6.x86_64 > gem list *** LOCAL GEMS *** cgi_multipart_eof_fix (2.5.0) daemon_controller (1.1.2) daemons (1.1.9, 1.0.10) fastthread (1.0.7) gem_plugin (0.2.3) json (1.4.6) mongrel (1.1.5) passenger (4.0.10, 4.0.5, 3.0.19) rack (1.5.2) rake (10.0.4, 0.8.7) > cat /etc/httpd/conf.d/puppetmaster LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10/buildout/apache2/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10 PassengerDefaultRuby /usr/bin/ruby PassengerStatThrottleRate 120 PassengerHighPerformance On PassengerMaxPoolSize 12 PassengerMaxRequests 1000 PassengerPoolIdleTime 600 Listen 8140 <VirtualHost *:8140> SSLEngine On # Only allow high security cryptography. Alter if needed for compatibility. SSLProtocol All -SSLv2 SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP SSLCertificateFile /var/lib/puppet/ssl/certs/<puppetmaster>.lan.pem SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/<puppetmaster>.pem SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem SSLVerifyClient optional SSLVerifyDepth 1 SSLOptions +StdEnvVars +ExportCertData RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/ <Directory /usr/share/puppet/rack/puppetmasterd/> Options None AllowOverride None Order Allow,Deny Allow from All </Directory> </VirtualHost> > cat /etc/puppet/puppet.conf [main] logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl modulepath = /opt/xxx/dev/puppet/modules manifestdir = /opt/xxx/dev/puppet/manifests manifest = /opt/xxx/dev/puppet/manifests/site.pp [agent] classfile = $vardir/classes.txt localconfig = $vardir/localconfig server = <puppetmaster> report = true splaylimit = 0 runinterval = 30 [master] certname=<puppetmaster> reports = http,log,tagmail reportdir = /var/lib/puppet/reports/upload reporturl = http://<puppetmaster>:3000/reports -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/groups/opt_out.
