Issue #21835 has been updated by Charlie Sharpsteen.
Christian Flamm wrote: > No idea what's going on here... I would guess that something in your manifests is programmatically constructing invalid class references. We are still very interested in examining your manifests---this could help us make specific recommendations for your situation and possibly come up with a better design for the type loader that resolves this problem. If it is possible to share your manifests but sanitization isn't feasible, I can arrange access to the secure drop box that we use when receiving this sort of information from commercial customers. ---------------------------------------- Bug #21835: Heavy load increase after upgrading to Puppet 3.2 https://projects.puppetlabs.com/issues/21835#change-96049 * Author: Christian Flamm * Status: Needs More Information * Priority: Normal * Assignee: Christian Flamm * Category: compiler * Target version: * Affected Puppet version: 3.2.3 * Keywords: performance load compile hostclass definition * Branch: ---------------------------------------- After upgrading from Puppet 3.1.1 to 3.2.x (x in {1,2,3}) I noticed a huge load increase on the puppetmaster machine. It's heavy enough that for every new 3.2.x version I tried it spooked me and I ran back to 3.1.1 as quickly as I could. Using apache/rack/passenger on an 8 core puppet master machine its load is usually between 3 and 4 (using 3.1.1). After the upgrade the load jumps to values between 7 and 12 (using 3.2.x). This overload situation then causes everything to slow down massively - especially agent's config_retrieval time (factor 2-3). Please have a look a the attached diagrams. Those diagrams each show 100 measures every 10s. The only thing that has changed between these two situations is an upgrade of puppet-server and puppet from 3.1.1 to 3.2.3. The values are: - **Load** ... load, 1 minute average - **\#Passenger Processes** ... output of 'passenger-status', "General information", "Processes" - **\#Clients ... output** of 'passenger-status --show=requests', "Clients" Does puppet 3.2 contain any new processing-heavy features that are enabled per default? > uname -a Linux <some agent> 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux > cat /etc/redhat-release CentOS release 6.3 (Final) > rpm -qa | egrep "puppet|ruby|httpd" rubygem-rake-0.8.7-2.1.el6.noarch ruby-mysql-2.8.2-1.el6.x86_64 libselinux-ruby-2.0.94-5.3.el6.x86_64 rubygem-fastthread-1.0.7-2.el6.x86_64 rubygem-mongrel-1.1.5-3.el6.x86_64 ruby-libs-1.8.7.352-10.el6_4.x86_64 ruby-irb-1.8.7.352-10.el6_4.x86_64 httpd-2.2.15-26.el6.centos.x86_64 puppetlabs-release-6-7.noarch puppet-3.1.1-1.el6.noarch ruby-augeas-0.4.1-1.el6.x86_64 ruby-shadow-1.4.1-13.el6.x86_64 rubygems-1.3.7-1.el6.noarch rubygem-gem_plugin-0.2.3-3.el6.noarch rubygem-daemons-1.0.10-2.el6.noarch ruby-1.8.7.352-10.el6_4.x86_64 ruby-rdoc-1.8.7.352-10.el6_4.x86_64 httpd-tools-2.2.15-26.el6.centos.x86_64 httpd-devel-2.2.15-26.el6.centos.x86_64 ruby-devel-1.8.7.352-10.el6_4.x86_64 puppet-dashboard-1.2.23-1.el6.noarch ruby-rgen-0.6.5-1.el6.noarch puppet-server-3.1.1-1.el6.noarch rubygem-json-1.4.6-1.el6.x86_64 > gem list *** LOCAL GEMS *** cgi_multipart_eof_fix (2.5.0) daemon_controller (1.1.2) daemons (1.1.9, 1.0.10) fastthread (1.0.7) gem_plugin (0.2.3) json (1.4.6) mongrel (1.1.5) passenger (4.0.10, 4.0.5, 3.0.19) rack (1.5.2) rake (10.0.4, 0.8.7) > cat /etc/httpd/conf.d/puppetmaster LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10/buildout/apache2/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10 PassengerDefaultRuby /usr/bin/ruby PassengerStatThrottleRate 120 PassengerHighPerformance On PassengerMaxPoolSize 12 PassengerMaxRequests 1000 PassengerPoolIdleTime 600 Listen 8140 <VirtualHost *:8140> SSLEngine On # Only allow high security cryptography. Alter if needed for compatibility. SSLProtocol All -SSLv2 SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP SSLCertificateFile /var/lib/puppet/ssl/certs/<puppetmaster>.lan.pem SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/<puppetmaster>.pem SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem SSLVerifyClient optional SSLVerifyDepth 1 SSLOptions +StdEnvVars +ExportCertData RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/ <Directory /usr/share/puppet/rack/puppetmasterd/> Options None AllowOverride None Order Allow,Deny Allow from All </Directory> </VirtualHost> > cat /etc/puppet/puppet.conf [main] logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl modulepath = /opt/xxx/dev/puppet/modules manifestdir = /opt/xxx/dev/puppet/manifests manifest = /opt/xxx/dev/puppet/manifests/site.pp [agent] classfile = $vardir/classes.txt localconfig = $vardir/localconfig server = <puppetmaster> report = true splaylimit = 0 runinterval = 30 [master] certname=<puppetmaster> reports = http,log,tagmail reportdir = /var/lib/puppet/reports/upload reporturl = http://<puppetmaster>:3000/reports -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/groups/opt_out.
