On Dec 23, 2010, at 4:58, Trevor Vaughan <[email protected]> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Brice, > > Thanks for the feedback, this is good stuff! > >> >> That's more or less what rsync does. For sourced files we could even use >> HTTP If-Modified-Since and/or If-None-Match to perform the check (and >> thus the check would be done server side). > > Yes, I briefly looked at the Rsync algorithm papers to see if I could > figure out how to re-implement it in Ruby but just using the native > Rsync libraries might be a better call. However, that would introduce an > external dependency. > >> >>> 4) For ultimate speed, a direct comparison should be an option as a >>> checksum type. Directly comparing the content of the in-memory file >>> and the target file appears to be twice as fast as an MD5 checksum. >>> This would not be feasible for a 'source'. >> >> That might be faster, but please don't re-introduce the slurp the whole >> file in memory syndrom. > > It seems that MD5 might be doing it anyway. When I tried a block-wise > 'comp', it was *much* slower and I think it was even slower than MD5 (or > close anyway) which means that MD5 is reading the whole blob into memory > to work on it anyway! If we're going to take the memory hit, let's just > take it and compare the two items. Is this an md5 script you wrote, or are you using the Puppet code? We've worked to add 'stream' checksum types that checksum the file a bit at a time. I expect that most of those are actually a good bit slower than just reading the whole thing in and checksumming, but they're faster by being less ram-efficient. >> That's really something I'd like to work on. Unfortunately this is >> really complex stuff. The file type is one of the biggest type and even >> though I already worked on it, I'm not sure I grasped enough to be able >> to fully refactor it for a different inner working. > > Completely agreed. I'll do what I can to help, but my outside time is > severely limited. -- http://puppetlabs.com/ | +1-615-594-8199 | @puppetmasterd -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
