Issue #23066 has been updated by Charlie Sharpsteen. File sharpie-tst_generate-0.1.0.tar.gz added
---------------------------------------- Bug #23066: Qualified variable lookups are very slow under Puppet 2.7+ unless prefixed with :: https://projects.puppetlabs.com/issues/23066#change-99403 * Author: Charlie Sharpsteen * Status: Unreviewed * Priority: Normal * Assignee: * Category: * Target version: * Affected Puppet version: * Keywords: * Branch: ---------------------------------------- This ticket concerns a performance regression that occurs when variable lookups are qualified as suggested by the [language documentation](http://docs.puppetlabs.com/puppet/3/reference/lang_variables.html#naming): > Qualified variable names are prefixed with the name of their scope and the > `::` (double colon) namespace separator. (For example, the `$vhostdir` > variable from the `apache::params` class would be > `$apache::params::vhostdir`.) The regression affects the 3.x and 2.7.x branches of Puppet. Consider the following node definitions in `site.pp`: <pre> node badnode { include 'tst_lookup::badclass' } node goodnode { include 'tst_lookup::goodclass' } </pre> These definitions reference a `tst_lookup` module with the following `params` class: <pre> class tst_lookup::params { $basedir = '/tmp/type_collection_tst' $user = 'tstusr' $ugroup = 'tstugroup' } </pre> `tst_lookup::goodclass` and `tst_lookup::badclass` are defined by ERB templates that generate 1000 file resources each referencing variables in `tst_lookup::params`. `badclass` references the variables in the manner suggested by the docs linked above. `goodclass` adds a leading `::` to the variable references. <pre> # badclass.pp.erb class tst_lookup::badclass { include tst_lookup::params group { "${tst_lookup::params::ugroup}": ensure => present, } user { "${tst_lookup::params::user}": ensure => present, gid => $tst_lookup::params::ugroup, } file { "${tst_lookup::params::basedir}": ensure => directory, owner => $tst_lookup::params::user, group => $tst_lookup::params::ugroup, } <% (1..1000).to_a.each do |i| %> file{ "${tst_lookup::params::basedir}/file_<%= i %>": ensure => file, content => "This is a test file.\n", owner => $tst_lookup::params::user, group => $tst_lookup::params::ugroup, require => File["${tst_lookup::params::basedir}"], } <% end %> } </pre> <pre> # goodclass.pp.erb class tst_lookup::goodclass { include tst_lookup::params group { "${::tst_lookup::params::ugroup}": ensure => present, } user { "${::tst_lookup::params::user}": ensure => present, gid => $::tst_lookup::params::ugroup, } file { "${::tst_lookup::params::basedir}": ensure => directory, owner => $::tst_lookup::params::user, group => $::tst_lookup::params::ugroup, } <% (1..1000).to_a.each do |i| %> file{ "${::tst_lookup::params::basedir}/file_<%= i %>": ensure => file, content => "This is a test file.\n", owner => $::tst_lookup::params::user, group => $::tst_lookup::params::ugroup, require => File["${::tst_lookup::params::basedir}"], } <% end %> } </pre> These templates can be turned into an actual Puppet manifests via: <pre> erb badclass.pp.erb > badclass.pp erb goodclass.pp.erb > goodclass.pp </pre> (A pre-built module is also attached to this ticket.) The performance characteristics of each class can be observed using the `puppet master --compile` command. ## 3.x <pre> # puppet --version 3.3.1 # time puppet master --compile badnode 2>&1 | grep 'Compiled catalog for' Notice: Compiled catalog for badnode in environment production in 19.65 seconds # puppet master --compile goodnode 2>&1 | grep 'Compiled catalog for' Notice: Compiled catalog for goodnode in environment production in 3.64 seconds </pre> ## 2.7.x <pre> # puppet --version 2.7.23 # puppet master --compile badnode 2>&1 | grep 'Compiled catalog for' notice: Compiled catalog for badnode in environment production in 67.04 seconds # puppet master --compile goodnode 2>&1 | grep 'Compiled catalog for' notice: Compiled catalog for goodnode in environment production in 4.89 seconds </pre> ## 2.6.x <pre> # puppet --version 2.6.18 # puppet master --compile badnode 2>&1 | grep 'Compiled catalog for' notice: Compiled catalog for badnode in environment production in 3.49 seconds # puppet master --compile goodnode 2>&1 | grep 'Compiled catalog for' notice: Compiled catalog for goodnode in environment production in 3.45 seconds </pre> The performance regression appears to be due to the [resolve_namespaces method in typeloader.rb](https://github.com/puppetlabs/puppet/blob/3.3.1/lib/puppet/resource/type_collection.rb#L164-L197). This method constructs a list of namespaces to search, starting with the local scope and proceeds to add namespace components until it reaches global scope. This method was added in [6b1dd81](https://github.com/puppetlabs/puppet/commit/6b1dd81799a44288287d9ab0cdf46afa3aaf090a) to address #4472 and #4483 --- but the behavior implemented is probably obsolete given the retirement of dynamic scope. Some examples of the behavior of `resolve_namespaces` under 3.3.x: Searching for `$::tst_lookup::params` in `class tst_lookup::goodclass` searches `["tst_lookup::params"]` and takes ~3 seconds for 4006 variable lookups. Searching for `$tst_lookup::params` in `class tst_lookup::badclass` searches `["tst_lookup::badclass::tst_lookup::params","tst_lookup::tst_lookup::params","tst_lookup::params"]` and takes ~20 seconds for 4006 variable lookups. Searching for `$tst_lookup::params` in `class tst_lookup::foo::worseclass` searches `["tst_lookup::foo::worseclass::tst_lookup::params","tst_lookup::foo::tst_lookup::params","tst_lookup::tst_lookup::params","tst_lookup::params"]` and takes ~33 seconds for 4006 variable lookups. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/groups/opt_out.
