Re: How to still kill git fetch with too many refs

2013-07-02 Thread Brandon Casey
On Mon, Jul 1, 2013 at 10:01 PM, Jeff King wrote: > On Tue, Jul 02, 2013 at 12:41:51AM -0400, Jeff King wrote: > >> I replicated your test setup, and the problem is that we have many >> common objects on both sides during the ref negotiation. So we end up in >> rev_list_push for each one, which ha

Re: How to still kill git fetch with too many refs

2013-07-02 Thread Martin Fick
On Tuesday, July 02, 2013 03:24:14 am Michael Haggerty wrote: > > git rev-list HEAD | for nn in $(seq 0 100) ; do for c > > in $(seq 0 1) ; do read sha ; echo $sha > > refs/c/$nn/$c$nn ; done ; done > .git/packed-refs > > I believe this generates a packed-refs file that is not > sorted lexic

Re: How to still kill git fetch with too many refs

2013-07-02 Thread Michael Haggerty
On 07/02/2013 05:02 AM, Martin Fick wrote: > I have often reported problems with git fetch when there are > many refs in a repo, and I have been pleasantly surprised > how many problems I reported were so quickly fixed. :) With > time, others have created various synthetic test cases to > ensur

Re: How to still kill git fetch with too many refs

2013-07-01 Thread Jeff King
On Mon, Jul 01, 2013 at 10:19:51PM -0700, Junio C Hamano wrote: > > Like the patch below, which is built on top of next (which has Junio's > > prio_queue implementation), and has both the priority queue fix for > > rev_list_push and the mark_complete sort-at-the-end fix. > > Wow, I saw "160 lines

Re: How to still kill git fetch with too many refs

2013-07-01 Thread Junio C Hamano
Jeff King writes: > On Tue, Jul 02, 2013 at 12:41:51AM -0400, Jeff King wrote: > >> I replicated your test setup, and the problem is that we have many >> common objects on both sides during the ref negotiation. So we end up in >> rev_list_push for each one, which has the same O(n^2) behavior. >>

Re: How to still kill git fetch with too many refs

2013-07-01 Thread Jeff King
On Tue, Jul 02, 2013 at 12:41:51AM -0400, Jeff King wrote: > I replicated your test setup, and the problem is that we have many > common objects on both sides during the ref negotiation. So we end up in > rev_list_push for each one, which has the same O(n^2) behavior. > Switching it to just sort a

Re: How to still kill git fetch with too many refs

2013-07-01 Thread Jeff King
On Tue, Jul 02, 2013 at 12:07:58AM -0400, Jeff King wrote: > And yet another alternative would be to keep the list unsorted during > the mark_complete calls, and then sort it at the end. Like this: Amusingly, I seem to have posted the exact same patch last year: http://article.gmane.org/gmane.

Re: How to still kill git fetch with too many refs

2013-07-01 Thread Jeff King
On Mon, Jul 01, 2013 at 09:02:31PM -0600, Martin Fick wrote: > A simple synthetic test case with 1M refs all pointing to the same > sha1 seems to be easily handled by git these days. However, in our > experience with our internal git repo, we still have performance > issues related to having too

How to still kill git fetch with too many refs

2013-07-01 Thread Martin Fick
I have often reported problems with git fetch when there are many refs in a repo, and I have been pleasantly surprised how many problems I reported were so quickly fixed. :) With time, others have created various synthetic test cases to ensure that git can handle many many refs. A simple synt