On Wed, Sep 14, 2016 at 4:58 PM, Jeff King <p...@peff.net> wrote:
> There's a fancy new compression algorithm called "zstd". The
> idea is that it's supposed to get similar compression ratios
> to zlib, but with much faster compression and decompression
> times. And on top of that, a nice sliding scale to trade off
> size versus time on the compression side.
> The zstd site at https://facebook.github.io/zstd/ claims
> close to 3x speedup for both compression and decompression
> versus zlib, with similar compression ratios. There are
> other fast algorithms (like lz4), but they usually compress
> much worse (follow the link above for a nice table of
> Since any git operations that have to access objects need to
> do a zlib inflate, in theory we can speed up everything by
> using zstd. And then on the packing side, use higher
> compression levels when making on-disk packfiles (which will
> be accessed many times) and lower ones when making loose
> objects, or deflating packed objects on the fly when serving
> The catch, of course, is that it's a new incompatible
> format. This would be a pretty huge change and totally break
> backwards compatibility for git, not just on disk but
> on-the-wire as well. So my goal here was not a finished
> product but just a quick experiment to see if it did indeed
> bring the promise speedups.
> Disappointingly, the answer seems to be "no".
After having looked at the data, I disagree with the conclusion.
And for that I think we need to reason about the frequency
of the operations happening.
* As an enduser, happily hacking away at one repository,
I probably do not care about the pack size on disk as much
as I care about timing of the local operations. And I assume
that for each repack we have about 1000 reads (log/rev-list)
The 1000 is a wild speculation without any data to back it up.
So as an end user I'd be happy about [zstd, ~5]
For the end user LZ4 seems to be the best solution if it were available.
* As a service provider, I know we have a lot more reads than
writes, and repacking is annoying. Also at that scale the disk
isn't negligible cheap. So we need to weigh the numbers differently,
but how? I suspect depending on the weighting it could still be
considered beneficial to go with zstd5. (No hard numbers here)