Issue #3558 has been updated by Luke Kanies. Status changed from Needs design decision to Ready for Checkin Assigned to changed from Luke Kanies to Markus Roberts
IMO this is ready and should be included. ---------------------------------------- Bug #3558: Performance improvement in lexer.rb http://projects.puppetlabs.com/issues/3558 Author: Ken Barber Status: Ready for Checkin Priority: Normal Assigned to: Markus Roberts Category: parser Target version: Affected version: 0.25.5 Keywords: Branch: Dev thread here: http://groups.google.com/group/puppet-dev/browse_thread/thread/4b4e356721b67ea Basically we noticed that under our run conditions: Test manifests: This is with 55 *.pp files ... totalling 63526 Puppet dsl lines across all files. System: Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz, 3G ram, Ubuntu 10.x (64 bit) We were getting long wait times in parsing our manifests. We found that by changing a single line in the lexer.rb from using += to << we got a massive improvement in speed: <snip> # this is probably pretty damned inefficient... # it'd be nice not to have to load the whole file first... def file=(file) @file = file @line = 1 File.open(file) { |of| str = "" - of.each { |line| str += line } + of.each { |line| str << line } @scanner = StringScanner.new(str) } </snip> For example - without the change the parsing was taking us 3 minutes to complete on my desktop: <snip> kbar...@purple:~/tmp/puppet/lak/jaro$ time ~/tmp/puppet/lak/real-puppet/bin/puppet --noop ~/tmp/puppet/lak/jaro/manifests/site.pp --modulepath=/home/ken/tmp/puppet/lak/jaro/modules/ Could not find default node or by name with 'purple.rim.net, purple.rim, purple' on node purple.rim.net real 2m59.575s user 2m54.580s sys 0m4.950s kbar...@purple:~/tmp/puppet/lak/jaro$ </snip> After the change from += to << - it was averaging about ~18s: <snip> kbar...@purple:~/tmp/puppet/lak/jaro$ time ~/tmp/puppet/lak/real-puppet/bin/puppet --noop ~/tmp/puppet/lak/jaro/manifests/site.pp --modulepath=/home/ken/tmp/puppet/lak/jaro/modules/ Could not find default node or by name with 'purple.rim.net, purple.rim, purple' on node purple.rim.net real 0m18.183s user 0m16.250s sys 0m1.900s kbar...@purple:~/tmp/puppet/lak/jaro$ More samples: 18.218 18.025 18.154 </snip> As suggested in thread, we also tested MR's patch instead: <snip> --- a/lib/puppet/parser/lexer.rb +++ b/lib/puppet/parser/lexer.rb @@ -267,11 +267,7 @@ class Puppet::Parser::Lexer def file=(file) @file = file @line = 1 - File.open(file) { |of| - str = "" - of.each { |line| str += line } - @scanner = StringScanner.new(str) - } + @scanner = StringScanner.new(File.read(file)) end def find_string_token </snip> Which seemed to average ~17-18s (slightly faster then the previous suggestion): <snip> kbar...@purple:~/tmp/puppet/lak/jaro$ time ~/tmp/puppet/lak/real-puppet/bin/puppet --noop ~/tmp/puppet/lak/jaro/manifests/site.pp --modulepath=/home/ken/tmp/puppet/lak/jaro/modules/ Could not find default node or by name with 'purple.rim.net, purple.rim, purple' on node purple.rim.net More samples: real 0m17.936s user 0m16.020s sys 0m1.930s kbar...@purple:~/tmp/puppet/lak/jaro$ 17.936 18.011 17.945 </snip> In summary - the results were a ~90% improvement which is pretty damn good. Can you guys review and apply one of these patches? It would be great to see this in 0.25.6 and 2.6.0 ... Thanks. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
