I have often reported problems with git fetch when there are
many refs in a repo, and I have been pleasantly surprised
how many problems I reported were so quickly fixed. :) With
time, others have created various synthetic test cases to
ensure that git can handle many many refs. A simple
synthetic test case with 1M refs all pointing to the same
sha1 seems to be easily handled by git these days. However,
in our experience with our internal git repo, we still have
performance issues related to having too many refs, in our
kernel/msm instance we have around 400K.
When I tried the simple synthetic test case and could not
reproduce bad results, so I tried something just a little
more complex and was able to get atrocious results!!!
Basically, I generate a packed-refs files with many refs
which each point to a different sha1. To get a list of
valid but unique sha1s for the repo, I simply used rev-list.
The result, a copy of linus' repo with a million unique
valid refs and a git fetch of a single updated ref taking a
very long time (55mins and it did not complete yet). Note,
with 100K refs it completes in about 2m40s. It is likely
not linear since 2m40s * 10 would be ~26m (but the
difference could also just be how the data in the sha1s are
Here is my small reproducible test case for this issue:
cp -rp linux linux.1Mrefs-revlist
echo "Hello" > hello ; git add hello ; git ci -a -m 'hello'
git rev-list HEAD | for nn in $(seq 0 100) ; do for c in
$(seq 0 10000) ; do read sha ; echo $sha refs/c/$nn/$c$nn ;
done ; done > .git/packed-refs
time git fetch file:///$(dirname $PWD)/linux
Any insights as to why it is so slow, and how we could
possibly speed it up?
PS: My tests were performed with git version 126.96.36.199 on
linux 2.6.32-37-generic #81-Ubuntu SMP
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html