This RFC includes a new way to index the objects in multiple packs
using one file, called the multi-pack index (MIDX).
The commits are split into parts as follows:
[01] - A full design document.
[02] - The full file format for MIDX files.
[03] - Creation of core.midx config setting.
[04-12] - Creation of "midx" builtin that writes, reads, and deletes
MIDX files.
[13-18] - Consume MIDX files for abbreviations and object loads.
The main goals of this RFC are:
* Determine interest in this feature.
* Find other use cases for the MIDX feature.
* Design a proper command-line interface for constructing and checking
MIDX files. The current "midx" builtin is probably inadequate.
* Determine what additional changes are needed before the feature can
be merged. Specifically, I'm interested in the interactions with
repack and fsck. The current patch also does not update the MIDX on
a fetch (which adds a packfile) but would be valuable. Whenever
possible, I tried to leave out features that could be added in a
later patch.
* Consider splitting this patch into multiple patches, such as:
i. The MIDX design document.
ii. The command-line interface for building and reading MIDX files.
iii. Integrations with abbreviations and object lookups.
Please do not send any style nits to this patch, as I expect the code to
change dramatically before we consider merging.
I created three copies of the Linux repo with 1, 24, and 127 packfiles
each using 'git repack -adfF --max-pack-size=[64m|16m]'. These copies
gave significant performance improvements on the following comand:
git log --oneline --raw --parents
Num Packs | Before MIDX | After MIDX | Rel % | 1 pack %
----------+-------------+------------+--------+----------
1 | 35.64 s | 35.28 s | -1.0% | -1.0%
24 | 90.81 s | 40.06 s | -55.9% | +12.4%
127 | 257.97 s | 42.25 s | -83.6% | +18.6%
The last column is the relative difference between the MIDX-enabled repo
and the single-pack repo. The goal of the MIDX feature is to present the
ODB as if it was fully repacked, so there is still room for improvement.
Changing the command to
git log --oneline --raw --parents --abbrev=40
has no observable difference (sub 1% change in all cases). This is likely
due to the repack I used putting commits and trees in a small number of
packfiles so the MRU cache workes very well. On more naturally-created
lists of packfiles, there can be up to 20% improvement on this command.
We are using a version of this patch with an upcoming release of GVFS.
This feature is particularly important in that space since GVFS performs
a "prefetch" step that downloads a pack of commits and trees on a daily
basis. These packfiles are placed in an alternate that is shared by all
enlistments. Some users have 150+ packfiles and the MRU misses and
abbreviation computations are significant. Now, GVFS manages the MIDX file
after adding new prefetch packfiles using the following command:
git midx --write --update-head --delete-expired --pack-dir=<alt>
As that release deploys we will gather more specific numbers on the
performance improvements and report them in this thread.
Derrick Stolee (18):
docs: Multi-Pack Index (MIDX) Design Notes
midx: specify midx file format
midx: create core.midx config setting
midx: write multi-pack indexes for an object list
midx: create midx builtin with --write mode
midx: add t5318-midx.sh test script
midx: teach midx --write to update midx-head
midx: teach git-midx to read midx file details
midx: find details of nth object in midx
midx: use existing midx when writing
midx: teach git-midx to clear midx files
midx: teach git-midx to delete expired files
t5318-midx.h: confirm git actions are stable
midx: load midx files when loading packs
midx: use midx for approximate object count
midx: nth_midxed_object_oid() and bsearch_midx()
sha1_name: use midx for abbreviations
packfile: use midx for object loads
.gitignore | 1 +
Documentation/config.txt | 3 +
Documentation/git-midx.txt | 106 ++++
Documentation/technical/multi-pack-index.txt | 149 +++++
Documentation/technical/pack-format.txt | 85 +++
Makefile | 2 +
builtin.h | 1 +
builtin/midx.c | 352 +++++++++++
cache.h | 1 +
command-list.txt | 1 +
config.c | 5 +
environment.c | 2 +
git.c | 1 +
midx.c | 850 +++++++++++++++++++++++++++
midx.h | 136 +++++
packfile.c | 79 ++-
packfile.h | 2 +
sha1_name.c | 70 ++-
t/t5318-midx.sh | 189 ++++++
19 files changed, 2020 insertions(+), 15 deletions(-)
create mode 100644 Documentation/git-midx.txt
create mode 100644 Documentation/technical/multi-pack-index.txt
create mode 100644 builtin/midx.c
create mode 100644 midx.c
create mode 100644 midx.h
create mode 100755 t/t5318-midx.sh
--
2.15.0
