Issue #9671 has been updated by Josh Cooper.
Ran some performance tests using 2000 empty files in a single directory with
the following manifest:
<pre>
file { "/Users/josh/tmp/scale2000":
ensure => directory,
owner => josh,
group => staff,
mode => 644,
recurse => true,
}
</pre>
The time to apply the manifest increased from 7.7 seconds in 2.6.2 to 85
seconds in 2.7.1. The time increased again to 135 seconds in 2.7.6:
<pre>
0 ~/work/puppet [2.6.2] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
real 0m7.741s
user 0m6.666s
sys 0m0.754s
0 ~/work/puppet [2.7.1] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 84.84 seconds
real 1m30.931s
user 1m29.997s
sys 0m0.908s
0 ~/work/puppet [2.7.3] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 86.09 seconds
real 1m32.362s
user 1m31.418s
sys 0m0.905s
0 ~/work/puppet [2.7.4] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 82.36 seconds
real 1m28.420s
user 1m27.498s
sys 0m0.900s
0 ~/work/puppet [2.7.5] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 84.73 seconds
real 1m30.912s
user 1m29.851s
sys 0m0.920s
0 ~/work/puppet [2.7.6] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 135.25 seconds
real 2m21.242s
user 2m18.146s
sys 0m3.001s
</pre>
I believe the initial performance decrease is related to changes to the graph,
especially commit:2a6c6cb8fabf82d2f2127c90db670c1a856427c5 for issue #5200,
which is where lines 171-174 in transaction.rb were added. More research is
needed to confirm, but we know it's not due to the Windows work.
The second performance decrease is due to
commit:1300e0afeb297d0f150cadf698e1f997dae7345c specifically this change:
<pre>
- # a lone slash as :name indicates a root dir on windows
- if value[:name] == '/'
- basedir
- else
- ::File.join( basedir, value[:name] )
- end
+
+ ::File.expand_path ::File.join( basedir, value[:name] )
</pre>
In other words, if I undo just those lines, performance goes from 135 seconds
back to 85. I'll talk to Nick tomorrow about this.
Another concerning issue is that I have just a single directory -- it's not
nested. Simply removing recurse => true from the manifest, improves performance
considerably for all versions: 1.8 seconds in 2.6.2 and 2.0 seconds in 2.7.1
and up. Here's the manifest:
<pre>
file { "/Users/josh/tmp/scale2000":
ensure => directory,
owner => josh,
group => staff,
mode => 644,
}
</pre>
And the results:
<pre>
0 ~/work/puppet [2.6.2] [1.8.7@noFeatures] $ puppet apply
~/work/manifests/9671-whit.pp
real 0m1.847s
user 0m1.184s
sys 0m0.644s
0 ~/work/puppet [2.7.1] [1.8.7@noFeatures] $ puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 0.51 seconds
real 0m2.068s
user 0m1.349s
sys 0m0.644s
0 ~/work/puppet [2.7.6] [1.8.7@noFeatures] $ time puppet apply
~/work/manifests/9671-whit.pp
notice: Finished catalog run in 0.51 seconds
real 0m2.039s
user 0m1.338s
sys 0m0.639s
</pre>
----------------------------------------
Bug #9671: transaction eval_generate slow on many files in recurse managed
directory
https://projects.puppetlabs.com/issues/9671
Author: Simon Effenberg
Status: Needs More Information
Priority: Normal
Assignee: Josh Cooper
Category: file
Target version: 2.7.x
Affected Puppet version: 2.7.3
Keywords: transaction file recurse
Branch:
Since 2.7 (jump from 2.6.2 to 2.7.3) a file resource for a directory with many
files in it is really really too slow.
<pre>
file { "/var/www/munin":
ensure => directory,
owner => munin,
group => munin,
mode => 644,
recurse => true,
require => [
Class['munin::master'],
]
}
</pre>
In the directory there are ~19000 files and in the
/usr/lib/ruby/1.8/transaction.rb in lines 171-174:
<pre>
made.each do |res|
add_conditional_directed_dependency(made.find { |r| r != res && r.name ==
res.name[0,r.name.length]} || resource, res)
add_conditional_directed_dependency(res, sentinal, default_label)
end
</pre>
the first add_conditional_directed_dependency needs a time between 0.2 and 1.5s
which is, let's say an avg (it is growing over the time) of 0.4s and this
*19000 files means, that the resource needs >2h... maybe this is not changeable
but i wants to know if i have to remove the recurse => true for this or if this
is a "bug"
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
