Jira (PUP-11073) Reduce memory usage

[Josh Cooper (Jira)]

Title: Message Title

Josh Cooper commented on PUP-11073   Re: Reduce memory usage I produced the list above by running the full_catalog benchmark: $ cd ~/work/puppet $ rbenv shell 2.5.8 $ git submodule update --init --recursive $ bundle install --with development $ bundle exec rake benchmark:full_catalog:memory_profile ... Generated memory profile: /home/josh/work/puppet/mem_profile_157882 which shows about 46MB - 3MB = 43MB of string allocations. Note the benchmark uses a very small number of facts: https://github.com/puppetlabs/puppet/blob/a68574e063d74eda66d7e26ad493ad8372a3803d/benchmarks/full_catalog/benchmarker.rb#L40-L75, so memory allocations for real catalog compilations are much worse. $ cat /home/josh/work/puppet/mem_profile_157882 ... allocated memory by class ----------------------------------- 46541290 String ... retained memory by class ----------------------------------- 3179506 String where objects are allocated in: allocated objects by location ----------------------------------- 102870 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:531 99060 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:554 56909 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:708 53340 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:529 53340 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:530 53340 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:552 49530 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:553 41910 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:496 38100 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:497 36270 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:711 33372 /home/josh/work/puppet/lib/puppet/resource.rb:651 32169 /home/josh/work/puppet/lib/puppet/concurrent/thread_local_singleton.rb:5 30036 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:737 29592 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:730 27831 /home/josh/work/puppet/lib/puppet/parser/scope.rb:497 26670 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:494 26350 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:84 25549 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/pathname.rb:42 25545 /home/josh/work/puppet/lib/puppet/resource.rb:306 23696 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/pathname.rb:46 18706 /home/josh/work/puppet/lib/puppet/parser/templatewrapper.rb:85 16540 /home/josh/work/puppet/lib/puppet/parser/resource.rb:221 16453 /home/josh/work/puppet/lib/puppet/resource.rb:418 15260 /home/josh/work/puppet/lib/puppet/resource.rb:687 14454 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:538 14192 /home/josh/work/puppet/lib/puppet/type.rb:442 13938 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:779 13633 /home/josh/work/puppet/lib/puppet/resource.rb:642 12618 /home/josh/work/puppet/lib/puppet/pops/evaluator/runtime3_support.rb:321 11376 /home/josh/work/puppet/lib/puppet/resource/catalog.rb:169 11376 /home/josh/work/puppet/lib/puppet/resource/catalog.rb:92 11296 /home/josh/work/puppet/lib/puppet/pops/types/type_formatter.rb:25 10723 /home/josh/work/puppet/lib/puppet/concurrent/thread_local_singleton.rb:8 10615 /home/josh/work/puppet/lib/puppet/pops/visitor.rb:46 10580 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/ripper/sexp.rb:33 10333 /home/josh/work/puppet/lib/puppet/resource/type_collection.rb:232 9878 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:37 9780 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:458 9756 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/erb.rb:873 9521 /home/josh/work/puppet/lib/puppet/pops/types/type_formatter.rb:652 9426 /home/josh/work/puppet/lib/puppet/pops/loader/typed_name.rb:19 9398 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/ripper/sexp.rb:106 9353 /home/josh/work/puppet/lib/puppet/parser/templatewrapper.rb:86 9278 /home/josh/work/puppet/lib/puppet/pops/loader/typed_name.rb:29 8982 /home/josh/work/puppet/lib/puppet/pops/visitor.rb:23 8711 /home/josh/work/puppet/lib/puppet/pops/parser/locator.rb:78 8627 /home/josh/work/puppet/lib/puppet/util/tagging.rb:113 8076 /home/josh/work/puppet/lib/puppet/parser/resource.rb:208 7961 /home/josh/work/puppet/lib/puppet/parser/scope.rb:435 7928 /home/josh/work/puppet/lib/puppet/type/file.rb:47 Also the "Allocated String Report" shows where unnecessary allocations are made: Allocated String Report ----------------------------------- 40454 "::" 27831 /home/josh/work/puppet/lib/puppet/parser/scope.rb:497 ... 29980 " " 19966 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:708 .. 24715 "" 5612 /home/josh/work/puppet/lib/puppet/resource.rb:642 2112 /home/josh/work/puppet/lib/puppet/pops/puppet_stack.rb:33 ... 13639 "_" 9353 /home/josh/work/puppet/lib/puppet/parser/templatewrapper.rb:85 3810 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:530 ... 10723 ".singleton" 10723 /home/josh/work/puppet/lib/puppet/concurrent/thread_local_singleton.rb:5 ... Add Comment   This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.400951.1622060066000.93114.1627491180117%40Atlassian.JIRA.