Hi all,
I want to flag the growing size of our security-tracker git
repositories and see how y'all feel about tackling it. I brought this
up at the last LTS meeting and was glad to hear I'm not alone. :)
I spent some time actually profiling both repositories, and the result
surprised me - so the diagnosis below is different (and better!) than
what I'd guessed.
The size, and where it comes from:
| repo | .git size | working tree (actual data)
| elts security-tracker | ~45G | ~350M
| lts security-tracker | ~27G | ~58M
The data we actually work with is tiny; everything else is history.
It's essentially just one single file: data/CVE/list (a ~53 MB text
file) accounts for 44.6 GB of the 44.7 GB in ELTS, and 25.9 GB of the
26.0 GB in LTS. That's ~116k committed versions of that one file, heh.
And it turns out git is storing them really inefficiently. Two
adjacent versions of data/CVE/list differ by only ~100 bytes (we
change ~33 lines per commit on average), yet git has stored thousands
of those versions as near-full 11–13 MB copies instead of tiny diffs.
Delta compression basically gave up on this file, which is a known
failure mode for a single 50 MB+ file under git's default pack
settings.
But this is (still) good news: it means almost all of that 45G/27G is
reclaimable by a plain repack - the commits are all fine, git just
packed them badly.
I tested this: repacking 500 recent versions of the file (5,397 MB on
disk) with a proper delta window brought them down to 87 MB - that's a
62x shrink, with zero data loss and no history rewrite.
8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<
What broke for me this week:
======================
My E/LTS clone was ~9 days behind, and a plain git pull failed
repeatedly. The server kept canceling the negotiation before it
finished counting objects:
remote: Enumerating objects: 55596, done.
remote: rpc error: code = DeadlineExceeded desc = running upload-pack:
waiting for negotiation: context canceled
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
A normal pull tries to compute the whole pack in one shot, and on a
repo this bloated that exceeds Salsa's upload-pack deadline, so it
dies with an early EOF every time. This is a direct symptom of the
same root cause - the server is being forced to stream tens of GB, it
shouldn't have to.
The only way I got current was to fetch shallowly and then deepen in
small increments so each request stayed under the timeout:
- git fetch --depth=1 origin master => succeeded, but left a
disconnected history that refused to fast-forward ("refusing to merge
unrelated histories").
- then, git fetch --deepen=<N> repeatedly in small chunks (15–20
commits at a time; a deepen of 100 hit the same timeout). After each,
I checked git merge-base --is-ancestor <old-HEAD> origin/master to see
whether the shallow boundary had reached back far enough to reconnect
with my existing history.
- once my old commit became an ancestor of the new origin/master, a
plain git merge --ff-only origin/master finally fast-forwarded
cleanly. phew!
It worked, but it took many rounds, left my clone shallow, and each
aborted attempt dumped a multi-GB tmp_pack_* into .git (that's what
briefly inflated my local .git to ~68 GB - junk from failed fetches,
cleaned up with git gc). No contributor should have to
reverse-engineer this just to pull. :)
Proposed solutions (in order):
=======================
1. Server-side repack on Salsa - the actual fix. Ask the Salsa admins
to run, on both repositories:
git repack -a -d -f --window=250 --depth=50
Based on my 62x measurement, this should take ELTS from ~45G to ~1.5G
and LTS from ~27G to ~0.75G - note that it is just an estimation and
not the exact numbers (which i'm happy to provide, too, if needed!).
It fixes both the disk size and the fetch timeout, and importantly, it
requires no re-clone and no commit-hash changes. Everyone's existing
clones keep working. This is by far the highest-leverage action IMO.
2. Make it stick with the committed pack config. A -f repack is
one-time; new commits can slowly re-bloat unless git keeps choosing
good deltas. So we should set and commit sane defaults:
git config pack.window 250
git config pack.depth 50
(Our .gitattributes already correctly marks data/CVE/list as text diff
- the problem was window size, not attributes, so we should not touch
that.)
3. Stopgap until the repack lands (hopefully!) - shallow clones, so
nobody else hits the timeout in the meantime:
git prune-packed && git gc # to clean the tmp_pack_* files
git clone --depth=1 <url> # ~400 MB instead of 45 GB
git fetch --deepen=50 # to update later, in steps, if needed
This won't fix anything, obviously, but it's just a stopgap - I
noticed Lee was running into this, too.
8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<
Let me know what y'all think. Happy to run more experiments on my side
and give you exact numbers or whatever y'all might need to be
convinced. :)
- u
