Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Eric Thompson updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Eric Thompson QA Contact: EricThompson QA Status: Reviewed Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Eric Thompson commented on an issue Re: FileBucket should not keep files in memory this looks like spec level can cover. anyone have concerns with pulling this through FR to resolved? Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Henrik Lindberg commented on an issue Re: FileBucket should not keep files in memory There is no new / altered functionality. Measuring that it uses less memory is very difficult in practice. It is possible to assert that it uses the new method by turning on verbose output when doing filebucket operations - every time it says it is using a file instead of a string once copy in memory was avoided. (The amount saved drowns in static from the rest of the system unless using huge files). Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Eric Thompson assigned an issue to Eric Thompson Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Eric Thompson Assignee: EricThompson Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Joshua Partlow commented on an issue Re: FileBucket should not keep files in memory Merged the filebucket filecontents patch always closing stream to master in 34e55da] Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Kylo Ginsberg updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Kylo Ginsberg Sprint: 2014-08-20 ,2014-09-03 Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Joshua Partlow assigned an issue to Henrik Lindberg Puppet::FileBucket::File::FileContents.stream is causing a problem on Windows in lib/puppet/indirector/file_bucket_file/file.rb:131 because the bucket_file.stream is left open. https://jenkins.puppetlabs.com/job/Puppet-Acceptance-Windows-master-vcloud-and-github/label=acc-coord,platform=win2008r2,ruby_arch=rubyx86/401/consoleFull (6541 and 7101 failures) Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Joshua Partlow Assignee: HenrikLindberg Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Henrik Lindberg commented on an issue Re: FileBucket should not keep files in memory PR merged to master. Follow up fix also made in a22bdca780ee8874fb868ff32c0f4a09e44fbd5b (file mode for creating a stream was r instead of the required rb for binary mode). Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Henrik Lindberg commented on an issue Re: FileBucket should not keep files in memory While it seems at first that it should be a simple thing to measure the memory consumption before / after it really requires something that is long time running, and comparing against historical memory consumption for the same scenario. Reliably measuring how much memory is used requires either sampling of the OS recording of data (i.e. like what top does), or that the code is running under a profiler. Smaller sets of data (many megabytes of data) cannot be observed above background noise. I used a 500 Mb file and this takes a long time to process when running normally. When running in a profiler it is prohibitively slow to profile. In order to measure how much memory that is allocated, the ruby runtime must run without GC. This means it will continue to grow and reach a very large size. When running with GC, it may not be noticed under test load if the changes have an effect. The amount of garbage generated when processing very large files makes it very slow and problematic to measure. If someone has ideas how these tests should be performed I would be very happy to hear them. For now I think it is best to verify that the changes I am about to make a PR for does not break anything. I am convinced that it allocates less memory than before in a number of situations. The real problem here is that the code paths through the indirector makes it very difficult to do efficient streaming. While the changes i am about to put in a PR will reduce the amount of memory, it is only a partial fix. A major rewrite of the involved code paths will be necessary to completely get rid of having large data blobs in memory. I do not see that being possible for Puppet 3.7 or Puppet 4.0. Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Henrik Lindberg commented on an issue Re: FileBucket should not keep files in memory I now have an implementation that uses a file whenever it can (it is not streaming a request to a file first, so there is still one copy in memory though). I have trouble finding a use case that I can use to measure memory consumption. Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Henrik Lindberg commented on an issue Re: FileBucket should not keep files in memory So far, I have validated my work on this ticket by : info logging that agent backup computes the checksum using the file method instead of on a string a backup followed by a restore yields identical bits it passes unit and integration tests Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Andy Parker assigned an issue to Henrik Lindberg Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Andy Parker Assignee: HenrikLindberg Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Kylo Ginsberg updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Kylo Ginsberg Sprint: Week2014- 7 08 - 23 06 Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Josh Cooper updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Josh Cooper Sprint: Week2014-7-23to2014-8-6 Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Adrien Thebo updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Adrien Thebo Component/s: Client Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Adrien Thebo updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Adrien Thebo Story Points: 2 Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Kylo Ginsberg updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Kylo Ginsberg Component/s: NetworkingServices Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Eric Sorenson updated an issue Puppet / PUP-1044 FileBucket should not keep files in memory Change By: Eric Sorenson Fix Version/s: 3.7.0 Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Eric Sorenson commented on an issue Re: FileBucket should not keep files in memory Pulling this in for 3.7... it was estimated at 2 story points and has been open since 2011. Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Josh Cooper commented on an issue Re: FileBucket should not keep files in memory Dup of PUP-741? Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title Henrik Lindberg commented on an issue Re: FileBucket should not keep files in memory The unwanted behavior seems to be in Puppet::FileBucket::Dipper and in FileBucket::File. The dipper reads the entire contents from files using binread, and asks for the content from a FileBucket::File as a String. The checksum calculation can be made on a file instead of content, by calling a checksum method with _file appended instead of just the method named after the type of checksum. The Puppet::Util::Checksums performs efficient checksum calculation by reading 4096 byte sized chunks. (That could also be optimized to read the file system's most optimal size which is its disk buffer size (which often is 2x4096) - with a system that uses 1x4096 blocks there is very little speed improvement using 2x4096, but it makes a difference for those that do use 2x4096). Add Comment Puppet / PUP-1044 FileBucket should not keep files in memory It should not be necessary to load entire files into memory to checksum them. This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you
Jira (PUP-1044) FileBucket should not keep files in memory
Title: Message Title redmine.exporter created an issue Puppet / PUP-1044 FileBucket should not keep files in memory Issue Type: New Feature Assignee: Unassigned Created: 16/Dec/13 3:02 AM Labels: redmine Priority: Normal Reporter: redmine.exporter It should not be necessary to load entire files into memory to checksum them. Add Comment This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)
