Issue #3558 has been updated by James Turnbull. Status changed from Unreviewed to Needs design decision Assigned to set to Luke Kanies
---------------------------------------- Bug #3558: Performance improvement in lexer.rb http://projects.puppetlabs.com/issues/3558 Author: Ken Barber Status: Needs design decision Priority: Normal Assigned to: Luke Kanies Category: parser Target version: Affected version: 0.25.5 Keywords: Branch: Dev thread here: http://groups.google.com/group/puppet-dev/browse_thread/thread/4b4e356721b67ea Basically we noticed that under our run conditions: Test manifests: This is with 55 *.pp files ... totalling 63526 Puppet dsl lines across all files. System: Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz, 3G ram, Ubuntu 10.x (64 bit) We were getting long wait times in parsing our manifests. We found that by changing a single line in the lexer.rb from using += to << we got a massive improvement in speed: <snip> # this is probably pretty damned inefficient... # it'd be nice not to have to load the whole file first... def file=(file) @file = file @line = 1 File.open(file) { |of| str = "" - of.each { |line| str += line } + of.each { |line| str << line } @scanner = StringScanner.new(str) } </snip> For example - without the change the parsing was taking us 3 minutes to complete on my desktop: <snip> kbar...@purple:~/tmp/puppet/lak/jaro$ time ~/tmp/puppet/lak/real-puppet/bin/puppet --noop ~/tmp/puppet/lak/jaro/manifests/site.pp --modulepath=/home/ken/tmp/puppet/lak/jaro/modules/ Could not find default node or by name with 'purple.rim.net, purple.rim, purple' on node purple.rim.net real 2m59.575s user 2m54.580s sys 0m4.950s kbar...@purple:~/tmp/puppet/lak/jaro$ </snip> After the change from += to << - it was averaging about ~18s: <snip> kbar...@purple:~/tmp/puppet/lak/jaro$ time ~/tmp/puppet/lak/real-puppet/bin/puppet --noop ~/tmp/puppet/lak/jaro/manifests/site.pp --modulepath=/home/ken/tmp/puppet/lak/jaro/modules/ Could not find default node or by name with 'purple.rim.net, purple.rim, purple' on node purple.rim.net real 0m18.183s user 0m16.250s sys 0m1.900s kbar...@purple:~/tmp/puppet/lak/jaro$ More samples: 18.218 18.025 18.154 </snip> As suggested in thread, we also tested MR's patch instead: <snip> --- a/lib/puppet/parser/lexer.rb +++ b/lib/puppet/parser/lexer.rb @@ -267,11 +267,7 @@ class Puppet::Parser::Lexer def file=(file) @file = file @line = 1 - File.open(file) { |of| - str = "" - of.each { |line| str += line } - @scanner = StringScanner.new(str) - } + @scanner = StringScanner.new(File.read(file)) end def find_string_token </snip> Which seemed to average ~17-18s (slightly faster then the previous suggestion): <snip> kbar...@purple:~/tmp/puppet/lak/jaro$ time ~/tmp/puppet/lak/real-puppet/bin/puppet --noop ~/tmp/puppet/lak/jaro/manifests/site.pp --modulepath=/home/ken/tmp/puppet/lak/jaro/modules/ Could not find default node or by name with 'purple.rim.net, purple.rim, purple' on node purple.rim.net More samples: real 0m17.936s user 0m16.020s sys 0m1.930s kbar...@purple:~/tmp/puppet/lak/jaro$ 17.936 18.011 17.945 </snip> In summary - the results were a ~90% improvement which is pretty damn good. Can you guys review and apply one of these patches? It would be great to see this in 0.25.6 and 2.6.0 ... Thanks. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
