Jira (PDB-165) storeconfigs face excessive memory requirement

[Giuseppe Lavagetto (JIRA)]

Title: Message Title

Giuseppe Lavagetto commented on PDB-165   Re: storeconfigs face excessive memory requirement Just for the record, I can report this again and it's not like a one-time bug, it's just the way the code is written: if you have more than a few hundred nodes, you're going to hit this too. I am in the process of migrating the Wikimedia Foundation puppet from storeconfigs to puppetDB, which is part of our effort to progressively move to puppet 4.0. I stumbled upon this same problem and had to allow the process to use almost 50 GB of memory, so I went to look at the code and fixed the issue: --- storeconfigs.orig.rb 2016-08-24 09:04:48.428728886 +0000 +++ storeconfigs.rb 2016-08-24 09:31:20.463976572 +0000 @@ -35,16 +35,13 @@ begin Puppet::Rails.connect - # Fetch all nodes, including exported resources and their params - nodes = Puppet::Rails::Host.all(:include => {:resources => [:param_values, :puppet_tags]}, - :conditions => {:resources => {:exported => true}}) - - catalogs = nodes.map { |node| node_to_catalog_hash(node) } - catalog_dir = File.join(workdir, 'catalogs') FileUtils.mkdir(catalog_dir) + # Fetch all nodes, including exported resources and their params + Puppet::Rails::Host.find_each(:include => {:resources => [:param_values, :puppet_tags]}, + :conditions => {:resources => {:exported => true}}, batch_size: 1) do |node| + catalog = node_to_catalog_hash(node) - catalogs.each do |catalog| filename = File.join(catalog_dir, "#{catalog[:data][:name]}.json") File.open(filename, 'w') do |file| Add Comment   This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.