Issue #2630 has been updated by Daniel Pittman. Target version changed from 2.7.x to 2.7.12
---------------------------------------- Refactor #2630: Sourced files are checksummed twice https://projects.puppetlabs.com/issues/2630#change-57169 Author: Josh Anderson Status: Closed Priority: Normal Assignee: Category: file Target version: 2.7.12 Affected Puppet version: 0.25.0 Keywords: Branch: Sourced files are being checksummed twice, doubling the processing time for these resources. This really adds up if you have a lot of files (or a few large files). To see for yourself, add a *puts* or *debug* call to the *checksum_file* function in util/checksums.rb: <pre> def checksum_file(digest, filename, lite = false) puts "DEBUG: checksumming %s" % filename </pre> Stack traces generated from *checksum_file* show that it's being called from both /type/file/content.rb and type/file/checksum.rb. Here's a sample run with stack traces: <pre> # puppetd --no-daemonize --verbose --onetime --tags filetest notice: Ignoring --listen on onetime run info: Retrieving plugin DEBUG: checksumming /var/puppet/lib/facter/servertype.rb DEBUG: checksumming /var/puppet/lib/facter/serverenv.rb DEBUG: checksumming /var/puppet/lib/facter/mailstore.rb DEBUG: checksumming /var/puppet/lib/facter/macosx_minorversion.rb DEBUG: checksumming /var/puppet/lib/facter/servernet.rb DEBUG: checksumming /var/puppet/lib/facter/zonename.rb DEBUG: checksumming /var/puppet/lib/facter/sparseroot.rb DEBUG: checksumming /var/puppet/lib/facter/imsversion.rb DEBUG: checksumming /var/puppet/lib/facter/servertype.rb DEBUG: checksumming /var/puppet/lib/facter/serverenv.rb DEBUG: checksumming /var/puppet/lib/facter/mailstore.rb DEBUG: checksumming /var/puppet/lib/facter/macosx_minorversion.rb DEBUG: checksumming /var/puppet/lib/facter/servernet.rb DEBUG: checksumming /var/puppet/lib/facter/zonename.rb DEBUG: checksumming /var/puppet/lib/facter/sparseroot.rb DEBUG: checksumming /var/puppet/lib/facter/imsversion.rb DEBUG: checksumming /var/puppet/lib/facter/imsversion.rb DEBUG: checksumming /var/puppet/lib/facter/imsversion.rb DEBUG: checksumming /var/puppet/lib/facter/macosx_minorversion.rb DEBUG: checksumming /var/puppet/lib/facter/macosx_minorversion.rb DEBUG: checksumming /var/puppet/lib/facter/mailstore.rb DEBUG: checksumming /var/puppet/lib/facter/mailstore.rb DEBUG: checksumming /var/puppet/lib/facter/serverenv.rb DEBUG: checksumming /var/puppet/lib/facter/serverenv.rb DEBUG: checksumming /var/puppet/lib/facter/servernet.rb DEBUG: checksumming /var/puppet/lib/facter/servernet.rb DEBUG: checksumming /var/puppet/lib/facter/servertype.rb DEBUG: checksumming /var/puppet/lib/facter/servertype.rb DEBUG: checksumming /var/puppet/lib/facter/sparseroot.rb DEBUG: checksumming /var/puppet/lib/facter/sparseroot.rb DEBUG: checksumming /var/puppet/lib/facter/zonename.rb DEBUG: checksumming /var/puppet/lib/facter/zonename.rb info: Loading facts in imsversion info: Loading facts in macosx_minorversion info: Loading facts in mailstore info: Loading facts in serverenv info: Loading facts in servernet info: Loading facts in servertype info: Loading facts in sparseroot info: Loading facts in zonename info: Loading facts in macosx_minorversion info: Loading facts in imsversion info: Loading facts in servertype info: Loading facts in mailstore info: Loading facts in serverenv info: Loading facts in servernet info: Loading facts in sparseroot info: Loading facts in zonename info: Loading facts in imsversion info: Loading facts in macosx_minorversion info: Loading facts in mailstore info: Loading facts in serverenv info: Loading facts in servernet info: Loading facts in servertype info: Loading facts in sparseroot info: Loading facts in zonename info: Loading facts in macosx_minorversion info: Loading facts in imsversion info: Loading facts in servertype info: Loading facts in mailstore info: Loading facts in serverenv info: Loading facts in servernet info: Loading facts in sparseroot info: Loading facts in zonename info: Caching catalog for puppetm.foo.com info: Applying configuration version '1252622612' DEBUG: checksumming /var/tmp/test.tar #<ZeroDivisionError: divided by 0> /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util/checksums.rb:79:in `/' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util/checksums.rb:79:in `checksum_file' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util/checksums.rb:29:in `md5_file' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file/checksum.rb:153:in `send' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file/checksum.rb:153:in `getsum' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file/checksum.rb:216:in `retrieve' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:810:in `currentpropvalues' /opt/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `inject' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:802:in `each' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:802:in `inject' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:802:in `currentpropvalues' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:790:in `retrieve' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file.rb:638:in `retrieve' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:719:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:60:in `apply' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:249:in `eval_children_and_apply_resource' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:422:in `thinmark' /opt/ruby/lib/ruby/1.8/benchmark.rb:308:in `realtime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:421:in `thinmark' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:248:in `eval_children_and_apply_resource' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:205:in `eval_resource' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:294:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:422:in `thinmark' /opt/ruby/lib/ruby/1.8/benchmark.rb:308:in `realtime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:421:in `thinmark' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:293:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:287:in `collect' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:287:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:139:in `apply' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/configurer.rb:147:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:177:in `benchmark' /opt/ruby/lib/ruby/1.8/benchmark.rb:308:in `realtime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:176:in `benchmark' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/configurer.rb:146:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:21:in `lock' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run' /opt/ruby/lib/ruby/1.8/sync.rb:230:in `synchronize' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:130:in `with_client' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:51:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application/puppetd.rb:103:in `onetime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `send' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `run_command' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `exit_on_fail' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run' /usr/bin/puppetd:159 DEBUG: checksumming /var/tmp/test.tar #<ZeroDivisionError: divided by 0> /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util/checksums.rb:79:in `/' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util/checksums.rb:79:in `checksum_file' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util/checksums.rb:29:in `md5_file' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file/content.rb:122:in `send' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file/content.rb:122:in `retrieve' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:810:in `currentpropvalues' /opt/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `inject' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:802:in `each' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:802:in `inject' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:802:in `currentpropvalues' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:790:in `retrieve' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type/file.rb:638:in `retrieve' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/type.rb:719:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:60:in `apply' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:249:in `eval_children_and_apply_resource' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:422:in `thinmark' /opt/ruby/lib/ruby/1.8/benchmark.rb:308:in `realtime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:421:in `thinmark' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:248:in `eval_children_and_apply_resource' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:205:in `eval_resource' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:294:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:422:in `thinmark' /opt/ruby/lib/ruby/1.8/benchmark.rb:308:in `realtime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:421:in `thinmark' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:293:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:287:in `collect' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/transaction.rb:287:in `evaluate' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:139:in `apply' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/configurer.rb:147:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:177:in `benchmark' /opt/ruby/lib/ruby/1.8/benchmark.rb:308:in `realtime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/util.rb:176:in `benchmark' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/configurer.rb:146:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:21:in `lock' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run' /opt/ruby/lib/ruby/1.8/sync.rb:230:in `synchronize' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:130:in `with_client' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/agent.rb:51:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application/puppetd.rb:103:in `onetime' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `send' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `run_command' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `exit_on_fail' /opt/ruby/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run' /usr/bin/puppetd:159 DEBUG: checksumming /var/tmp/blah/g.txt DEBUG: checksumming /var/tmp/blah/a.txt DEBUG: checksumming /var/tmp/blah/d.txt DEBUG: checksumming /var/tmp/blah/b.txt DEBUG: checksumming /var/tmp/blah/e.txt DEBUG: checksumming /var/tmp/blah/c.txt DEBUG: checksumming /var/tmp/blah/f.txt DEBUG: checksumming /var/tmp/blah/g.txt DEBUG: checksumming /var/tmp/blah/a.txt DEBUG: checksumming /var/tmp/blah/d.txt DEBUG: checksumming /var/tmp/blah/b.txt DEBUG: checksumming /var/tmp/blah/e.txt DEBUG: checksumming /var/tmp/blah/c.txt DEBUG: checksumming /var/tmp/blah/f.txt notice: Finished catalog run in 10.65 seconds </pre> Please note that stack traces were only generated for one sourced file in the test module. It seems like this double checksumming could be eliminated with some clever refactoring. Checksum.rb already caches the value it generates (and source.rb uses that value), but content.rb is calling *<checksum type>_file* directly rather than asking for the value of the checksum parameter. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
