Issue #13097 has been updated by Daniel Pittman. Assignee deleted (Daniel Pittman)
---------------------------------------- Bug #13097: Filebucket reads entire files into memory https://projects.puppetlabs.com/issues/13097#change-92302 * Author: Evan Mezeske * Status: Accepted * Priority: Normal * Assignee: * Category: filebucket * Target version: * Affected Puppet version: * Keywords: * Branch: ---------------------------------------- During backup/restore, Puppet::FileBucket::Dipper uses IO.binread() with no length parameter to read the entire file resource in question into memory at one time. This is very simple, but it causes problems when Puppet is used to manage large files. To compound the problem, Puppet::FileBucketFile.verify_identical_file! also reads any backed-up copy of the file fully into memory. Thus, in the worst case, Puppet might read two full copies of a large file into memory at one time. This problem can result in Puppet daemon processes taking up large amounts of RAM. Even though this RAM may be reused by the Ruby interpreter, it does not seem to be released back to the OS. I propose that it would be better to operate on the files in question in small chunks, only reading a couple kilobytes into memory at once. This is a common practice. For instance, Puppet::Util::Checksums.checksum_file reads the file in small chunks for performing the MD5. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
