Issue #21835 has been updated by Charlie Sharpsteen.
Christian Flamm wrote: > Checked out 3.2.3 and added the extra line to my puppet.conf. > `config_retrieval` time is now consistently below 15sec. Two things: > > 1. Might I somehow get in trouble when using `ignoremissingtypes = true`? Toggling this option to true re-enables an optimization that was on by default in puppet versions 3.0 -- 3.1.1. There should be no change in behavior, other than the addition of some warning messages in the master logs that call out counter-intutive behavior produced by the optimization (much more on that later). Some background: the optimization in question was added in [242692e](https://github.com/puppetlabs/puppet/commit/242692e) and was designed to improve the performance of pre-2.6 manifests that had not been re-factored to take advantage of the module system and autoloader. The problem with the optimization is that it worked by having the compiler silently quit looking for classes or defined types that were noted as missing on previous catalog runs. The compiler would continue to ignore missing types and classes until a restart or environment reload occurred. This behavior resulted in confusing error messages that were difficult to debug: #16568 (and many related issues). The patch I added in [0255778](https://github.com/puppetlabs/puppet/commit/0255778) made two alterations to the optimization: - A warning message was added when class or type lookups were skipped, because silent behavior is difficult to detect and debug. - The optimization was disabled by default so that users coming from 2.7.x wouldn't have to train themselves to bounce the Puppetmaster every time they corrected a missing class error. Which brings us to: > Within minutes puppet log gets flooded with literally 100,000s of lines à la > Not attempting to load definition... This is logging far beyond the scale I was anticipating when I added the warning message. I will look into throttling that warning or removing it until we can find a saner way to call out the behavior of this optimization. All the noise is coming from a single log call in the TypeCollection if you want to knock it out while running Puppet 3.2.3 with ignoremissingtypes enabled: <https://github.com/puppetlabs/puppet/blob/3.2.3/lib/puppet/resource/type_collection.rb#L205> However, that noise is calling out places where the type loader is being invoket to find things that do not exist. It is worth looking at the warning messages being emitted to see if there are unimplemented types or classes in your manifests that can be pruned. In fact, we would be very interested in examining production manifests that are generating thousands of bogus calls to the type loader. This would help us design a better optimization for this case, and possibly a better type loader. If it is possible to share the manifests but not feasible to sanitize them, you could use the secure delivery mechanism that commercial support uses for this kind of information. ---------------------------------------- Bug #21835: Heavy load increase after upgrading to Puppet 3.2 https://projects.puppetlabs.com/issues/21835#change-95868 * Author: Christian Flamm * Status: Needs More Information * Priority: Normal * Assignee: Christian Flamm * Category: compiler * Target version: * Affected Puppet version: 3.2.3 * Keywords: rack passenger performance load compile * Branch: ---------------------------------------- After upgrading from Puppet 3.1.1 to 3.2.x (x in {1,2,3}) I noticed a huge load increase on the puppetmaster machine. It's heavy enough that for every new 3.2.x version I tried it spooked me and I ran back to 3.1.1 as quickly as I could. Using apache/rack/passenger on an 8 core puppet master machine its load is usually between 3 and 4 (using 3.1.1). After the upgrade the load jumps to values between 7 and 12 (using 3.2.x). This overload situation then causes everything to slow down massively - especially agent's config_retrieval time (factor 2-3). Please have a look a the attached diagrams. Those diagrams each show 100 measures every 10s. The only thing that has changed between these two situations is an upgrade of puppet-server and puppet from 3.1.1 to 3.2.3. The values are: - **Load** ... load, 1 minute average - **\#Passenger Processes** ... output of 'passenger-status', "General information", "Processes" - **\#Clients ... output** of 'passenger-status --show=requests', "Clients" Does puppet 3.2 contain any new processing-heavy features that are enabled per default? > uname -a Linux <some agent> 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux > cat /etc/redhat-release CentOS release 6.3 (Final) > rpm -qa | egrep "puppet|ruby|httpd" rubygem-rake-0.8.7-2.1.el6.noarch ruby-mysql-2.8.2-1.el6.x86_64 libselinux-ruby-2.0.94-5.3.el6.x86_64 rubygem-fastthread-1.0.7-2.el6.x86_64 rubygem-mongrel-1.1.5-3.el6.x86_64 ruby-libs-1.8.7.352-10.el6_4.x86_64 ruby-irb-1.8.7.352-10.el6_4.x86_64 httpd-2.2.15-26.el6.centos.x86_64 puppetlabs-release-6-7.noarch puppet-3.1.1-1.el6.noarch ruby-augeas-0.4.1-1.el6.x86_64 ruby-shadow-1.4.1-13.el6.x86_64 rubygems-1.3.7-1.el6.noarch rubygem-gem_plugin-0.2.3-3.el6.noarch rubygem-daemons-1.0.10-2.el6.noarch ruby-1.8.7.352-10.el6_4.x86_64 ruby-rdoc-1.8.7.352-10.el6_4.x86_64 httpd-tools-2.2.15-26.el6.centos.x86_64 httpd-devel-2.2.15-26.el6.centos.x86_64 ruby-devel-1.8.7.352-10.el6_4.x86_64 puppet-dashboard-1.2.23-1.el6.noarch ruby-rgen-0.6.5-1.el6.noarch puppet-server-3.1.1-1.el6.noarch rubygem-json-1.4.6-1.el6.x86_64 > gem list *** LOCAL GEMS *** cgi_multipart_eof_fix (2.5.0) daemon_controller (1.1.2) daemons (1.1.9, 1.0.10) fastthread (1.0.7) gem_plugin (0.2.3) json (1.4.6) mongrel (1.1.5) passenger (4.0.10, 4.0.5, 3.0.19) rack (1.5.2) rake (10.0.4, 0.8.7) > cat /etc/httpd/conf.d/puppetmaster LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10/buildout/apache2/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.10 PassengerDefaultRuby /usr/bin/ruby PassengerStatThrottleRate 120 PassengerHighPerformance On PassengerMaxPoolSize 12 PassengerMaxRequests 1000 PassengerPoolIdleTime 600 Listen 8140 <VirtualHost *:8140> SSLEngine On # Only allow high security cryptography. Alter if needed for compatibility. SSLProtocol All -SSLv2 SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP SSLCertificateFile /var/lib/puppet/ssl/certs/<puppetmaster>.lan.pem SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/<puppetmaster>.pem SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem SSLVerifyClient optional SSLVerifyDepth 1 SSLOptions +StdEnvVars +ExportCertData RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/ <Directory /usr/share/puppet/rack/puppetmasterd/> Options None AllowOverride None Order Allow,Deny Allow from All </Directory> </VirtualHost> > cat /etc/puppet/puppet.conf [main] logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl modulepath = /opt/xxx/dev/puppet/modules manifestdir = /opt/xxx/dev/puppet/manifests manifest = /opt/xxx/dev/puppet/manifests/site.pp [agent] classfile = $vardir/classes.txt localconfig = $vardir/localconfig server = <puppetmaster> report = true splaylimit = 0 runinterval = 30 [master] certname=<puppetmaster> reports = http,log,tagmail reportdir = /var/lib/puppet/reports/upload reporturl = http://<puppetmaster>:3000/reports -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/groups/opt_out.
