Issue #21729 has been updated by Lee Lowder.
Status changed from Unreviewed to Accepted
Assignee set to Deepak Giridharagopal
Keywords changed from puppetdb, exported resources, collection to puppetdb,
exported resources, collection, compiler
Using the following code with a 2 VM setup:
test_duplicates/manifests/init.pp:
class test_duplicates{
@@file {
"/tmp/foo_${::hostname}.txt":
ensure => present,
content => $::fqdn,
}
}
site.pp :
node default {
include test_duplicates
}
node master {
include test_duplicates
File <<| |>>
}
Running:
vagrant@master:~$ sudo puppet master --compile master --profile --debug
--logdest /tmp/compile
followed by a grep on Collect:
vagrant@master:~$ grep Collect /tmp/compile
2013-07-13 22:44:08 +0000 Scope(Node[master]) (debug): Collected 2 File
resources in 0.39 seconds
2013-07-13 22:44:11 +0000 Scope(Node[master]) (debug): Collected 0 File
resources in 0.33 seconds
2013-07-13 22:44:12 +0000 Scope(Node[master]) (debug): Collected 0 File
resources in 0.35 seconds
2013-07-13 22:44:12 +0000 Scope(Node[master]) (debug): Collected 0 File
resources in 0.33 seconds
2013-07-13 22:44:12 +0000 Scope(Node[master]) (debug): Collected 0 File
resources in 0.35 seconds
----------------------------------------
Bug #21729: duplicate resource collections during catalog compilation
https://projects.puppetlabs.com/issues/21729#change-94709
* Author: Patrick Hemmer
* Status: Accepted
* Priority: Normal
* Assignee: Deepak Giridharagopal
* Category: compiler
* Target version:
* Affected Puppet version: 3.2.2
* Keywords: puppetdb, exported resources, collection, compiler
* Branch:
----------------------------------------
While attempting to troubleshoot performance issues with puppetdb, I noticed
that during a single catalog compilation, the same resources are trying to be
collected multiple times. In my case, what is normally 4 collections turns into
28. This is potentially the source of my performance issue when multiple nodes
check in and hammer puppetdb with all these duplicate collections.
Here is a very simple test showing a single resource being collected twice:
<pre>
rm /tmp/testcompile; puppet apply --noop --profile --debug --logdest
/tmp/testcompile -e 'Ssh_authorized_key <<||>>'; grep Collect /tmp/testcompile
2013-07-12 00:10:08 +0000 Scope(Class[main]) (debug): Collected 1
Ssh_authorized_key resource in 0.18 seconds
2013-07-12 00:10:08 +0000 Scope(Class[main]) (debug): Collected 0
Ssh_authorized_key resources in 0.14 seconds
</pre>
(attached is the full version of this log file)
Catalog generation for a real node looks like this (this is from a `puppet
master --compile NODENAME --profile --debug --logdest /tmp/compile`):
<pre>
# grep Collected /tmp/compile
2013-07-11 22:35:06 +0000 Scope(Class[Common::Hosts]) (debug): Collected 1 Host
resource in 0.32 seconds
2013-07-11 22:35:06 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 1
Ssh_authorized_key resource in 0.15 seconds
2013-07-11 22:35:06 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 1
File resource in 0.15 seconds
2013-07-11 22:35:06 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 0.15 seconds
2013-07-11 22:35:09 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host
resources in 1.09 seconds
2013-07-11 22:35:10 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
Ssh_authorized_key resources in 1.31 seconds
2013-07-11 22:35:11 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
File resources in 0.84 seconds
2013-07-11 22:35:12 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 0.59 seconds
2013-07-11 22:35:13 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host
resources in 0.35 seconds
2013-07-11 22:35:14 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
Ssh_authorized_key resources in 0.40 seconds
2013-07-11 22:35:14 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
File resources in 0.57 seconds
2013-07-11 22:35:15 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 0.50 seconds
2013-07-11 22:35:16 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host
resources in 0.47 seconds
2013-07-11 22:35:16 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
Ssh_authorized_key resources in 0.76 seconds
2013-07-11 22:35:18 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
File resources in 1.46 seconds
2013-07-11 22:35:19 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 1.20 seconds
2013-07-11 22:35:20 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host
resources in 0.95 seconds
2013-07-11 22:35:21 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
Ssh_authorized_key resources in 1.15 seconds
2013-07-11 22:35:23 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
File resources in 1.18 seconds
2013-07-11 22:35:24 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 1.52 seconds
2013-07-11 22:35:25 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host
resources in 1.15 seconds
2013-07-11 22:35:26 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
Ssh_authorized_key resources in 0.91 seconds
2013-07-11 22:35:27 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
File resources in 1.06 seconds
2013-07-11 22:35:28 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 1.02 seconds
2013-07-11 22:35:29 +0000 Scope(Class[Common::Hosts]) (debug): Collected 0 Host
resources in 1.01 seconds
2013-07-11 22:35:31 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
Ssh_authorized_key resources in 1.94 seconds
2013-07-11 22:35:33 +0000 Scope(Class[Common::Rundeck]) (debug): Collected 0
File resources in 2.17 seconds
2013-07-11 22:35:35 +0000 Scope(Class[Common::User::Root]) (debug): Collected 0
Ssh_authorized_key resources in 1.77 seconds
</pre>
Puppet 3.2.2
PuppetDB 1.3.2
Ruby 1.9.3p429
Ubuntu 12.04
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/groups/opt_out.
