Hi, On Mon, Feb 17, 2014 at 4:30 AM, Thomas Mueller <[email protected]> wrote: > If the tar file only contains a few entries, then I think it's not a > problem. If the tar file contains many entries, then I think this by > itself is an issue, as maintaining such an index is problematic.
I'm expecting a normal 256MB tar file to contain 1k - 10k segments, so an index for such a file would range from a few dozen to a few hundred kilobytes, i.e. no more than 1% of the required space. I'm thinking of only adding such an index at the end of a tar file once it's filled up or explicitly closed, until then the index can be kept in memory or created by a startup scan. As the tar files are append-only in nature, index maintenance shouldn't be too much of a problem. Once the index is written at the end of a tar file, there'll be no need to update it as new segments will be written to the next tar file. > Some statistics about segement sizes based on one example repository: Thanks for the summary! As a general rule I've tried to fit as much data in a segment as possible (including merging multiple small commits in one segment) in an attempt to keep the median segment size fairly large (> 1kB) as the tar file format requires at least 1kB of disk space for each non-empty entry. There's still a few ideas left on how to increase segment sizes, if it turns out that doing so would be beneficial also for indexing. BR, Jukka Zitting
