Issue #21729 has been updated by Patrick Hemmer. File duplicate_collection.patch added Affected Puppet version changed from 3.2.2 to 3.3.0
Attached is a patch I came up with to solve the issue. It basically keeps a history of searches that have been sent to the indirector, whenever a search is a duplicate of a previous search, it skips the rest of the code and acts as if no resources were found (which would be the end result anyway as all resources would end up being duplicates and thus discarded). This was the only solution I saw. It doesn't look like there is a clean way to prevent the duplicate collectors being created, or being evaluated multiple times (as it's apparently being done deliberately). ---------------------------------------- Bug #21729: duplicate resource collections during catalog compilation https://projects.puppetlabs.com/issues/21729#change-98762 * Author: Patrick Hemmer * Status: Accepted * Priority: Normal * Assignee: Deepak Giridharagopal * Category: compiler * Target version: * Affected Puppet version: 3.3.0 * Keywords: puppetdb, exported resources, collection, compiler * Branch: ---------------------------------------- While attempting to troubleshoot performance issues with puppetdb, I noticed that during a single catalog compilation, the same resources are trying to be collected multiple times. In my case, what is normally 4 collections turns into 28. This is potentially the source of my performance issue when multiple nodes check in and hammer puppetdb with all these duplicate collections. Here is a very simple test showing a single resource being collected twice: <pre> rm /tmp/testcompile; puppet apply --noop --profile --debug --logdest /tmp/testcompile -e 'Ssh_authorized_key <<||>>'; grep Collect /tmp/testcompile 2013-07-12 00:10:08 +0000 Scope(Class[main]) (debug): Collected 1 Ssh_authorized_key resource in 0.18 seconds 2013-07-12 00:10:08 +0000 Scope(Class[main]) (debug): Collected 0 Ssh_authorized_key resources in 0.14 seconds </pre> (attached is the full version of this log file) Catalog generation for a real node looks like this (this is from a `puppet master --compile NODENAME --profile --debug --logdest /tmp/compile`): <pre> # grep Collected /tmp/compile 2013-07-11 22:35:06 +0000 Scope(Class[Common::Hosts]) (debug): Collected 1 Host resource in 0.32 seconds 2013-07-11 22:35:06 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 1 Ssh_authorized_key resource in 0.15 seconds 2013-07-11 22:35:06 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 1 File resource in 0.15 seconds 2013-07-11 22:35:06 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 0.15 seconds 2013-07-11 22:35:09 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host resources in 1.09 seconds 2013-07-11 22:35:10 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 Ssh_authorized_key resources in 1.31 seconds 2013-07-11 22:35:11 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 File resources in 0.84 seconds 2013-07-11 22:35:12 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 0.59 seconds 2013-07-11 22:35:13 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host resources in 0.35 seconds 2013-07-11 22:35:14 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 Ssh_authorized_key resources in 0.40 seconds 2013-07-11 22:35:14 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 File resources in 0.57 seconds 2013-07-11 22:35:15 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 0.50 seconds 2013-07-11 22:35:16 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host resources in 0.47 seconds 2013-07-11 22:35:16 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 Ssh_authorized_key resources in 0.76 seconds 2013-07-11 22:35:18 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 File resources in 1.46 seconds 2013-07-11 22:35:19 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 1.20 seconds 2013-07-11 22:35:20 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host resources in 0.95 seconds 2013-07-11 22:35:21 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 Ssh_authorized_key resources in 1.15 seconds 2013-07-11 22:35:23 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 File resources in 1.18 seconds 2013-07-11 22:35:24 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 1.52 seconds 2013-07-11 22:35:25 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host resources in 1.15 seconds 2013-07-11 22:35:26 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 Ssh_authorized_key resources in 0.91 seconds 2013-07-11 22:35:27 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 File resources in 1.06 seconds 2013-07-11 22:35:28 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 1.02 seconds 2013-07-11 22:35:29 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host resources in 1.01 seconds 2013-07-11 22:35:31 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 Ssh_authorized_key resources in 1.94 seconds 2013-07-11 22:35:33 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0 File resources in 2.17 seconds 2013-07-11 22:35:35 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0 Ssh_authorized_key resources in 1.77 seconds </pre> Puppet 3.2.2 PuppetDB 1.3.2 Ruby 1.9.3p429 Ubuntu 12.04 -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/groups/opt_out.
