Hi,

On Mon, Feb 17, 2014 at 4:30 AM, Thomas Mueller <[email protected]> wrote:
> If the tar file only contains a few entries, then I think it's not a
> problem. If the tar file contains many entries, then I think this by
> itself is an issue, as maintaining such an index is problematic.

I'm expecting a normal 256MB tar file to contain 1k - 10k segments, so
an index for such a file would range from a few dozen to a few hundred
kilobytes, i.e. no more than 1% of the required space.

I'm thinking of only adding such an index at the end of a tar file
once it's filled up or explicitly closed, until then the index can be
kept in memory or created by a startup scan.

As the tar files are append-only in nature, index maintenance
shouldn't be too much of a problem. Once the index is written at the
end of a tar file, there'll be no need to update it as new segments
will be written to the next tar file.

> Some statistics about segement sizes based on one example repository:

Thanks for the summary!

As a general rule I've tried to fit as much data in a segment as
possible (including merging multiple small commits in one segment) in
an attempt to keep the median segment size fairly large (> 1kB) as the
tar file format requires at least 1kB of disk space for each non-empty
entry. There's still a few ideas left on how to increase segment
sizes, if it turns out that doing so would be beneficial also for
indexing.

BR,

Jukka Zitting

Reply via email to