Re: Poor performance of git describe in big repos

2013-06-03 Thread Junio C Hamano
Junio C Hamano writes: > Alex Bennée writes: > >> Why is git attempting to parse a commit not on the DAG for the branch >> I'm attempting to describe? > > I think that is because you need to parse the objects at the tip of > refs to see if they are on the DAG in the first place. > > If there wer

Re: Poor performance of git describe in big repos

2013-06-03 Thread Junio C Hamano
Alex Bennée writes: > Why is git attempting to parse a commit not on the DAG for the branch > I'm attempting to describe? I think that is because you need to parse the objects at the tip of refs to see if they are on the DAG in the first place. If there weren't any annotated tag, conceivably yo

Re: Poor performance of git describe in big repos

2013-06-03 Thread Jeff King
On Mon, Jun 03, 2013 at 09:39:21AM +0100, Alex Bennée wrote: > > in the target repository. You can check that it's working because "git > > rev-list --all --count" should be an order of magnitude faster. You may > > need to add "save_commit_buffer = 0" in any commands you are checking, > > though,

Re: Poor performance of git describe in big repos

2013-06-03 Thread Alex Bennée
On 31 May 2013 17:17, Jeff King wrote: > On Fri, May 31, 2013 at 12:27:11PM +0200, Thomas Rast wrote: > >> Thomas Rast writes: >> >> > However, if that turns out to be the culprit, it's not fixable >> > currently[1]. Having commits with insanely long messages is just, well, >> > insane. >> > >>

Re: Poor performance of git describe in big repos

2013-06-03 Thread Alex Bennée
On 31 May 2013 10:57, Alex Bennée wrote: > On 31 May 2013 09:46, Thomas Rast wrote: >> >> So that deleted all unannotated tags pointing at commits, and then it >> was fast. Curious. >> >> However, if that turns out to be the culprit, it's not fixable >> currently[1]. Having commits with insanel

Re: Poor performance of git describe in big repos

2013-05-31 Thread Jeff King
On Fri, May 31, 2013 at 12:27:11PM +0200, Thomas Rast wrote: > Thomas Rast writes: > > > However, if that turns out to be the culprit, it's not fixable > > currently[1]. Having commits with insanely long messages is just, well, > > insane. > > > > [1] unless we do a major rework of the loading

Re: Poor performance of git describe in big repos

2013-05-31 Thread Thomas Rast
Thomas Rast writes: > However, if that turns out to be the culprit, it's not fixable > currently[1]. Having commits with insanely long messages is just, well, > insane. > > [1] unless we do a major rework of the loading infrastructure, so that > we can teach it to load only the beginning of a c

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 31 May 2013 09:46, Thomas Rast wrote: > Alex Bennée writes: > >> I think you are right. I was brave (well I assumed the tags would come >> back from the upstream repo) and ran: >> >> git for-each-ref | grep "refs/tags" | grep "commit" | cut -d '/' -f 3 >> | xargs git tag -d > > So that deleted

Re: Poor performance of git describe in big repos

2013-05-31 Thread John Keeping
On Fri, May 31, 2013 at 09:49:57AM +0100, Alex Bennée wrote: > On 31 May 2013 09:32, John Keeping wrote: > > Thomas is suggesting that you might have a tag that does not point at a > > commit but instead points to a blob object. > > It's looking like I just have some very heavy commits. One data

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 31 May 2013 09:32, John Keeping wrote: > On Fri, May 31, 2013 at 09:14:49AM +0100, Alex Bennée wrote: >> On 30 May 2013 20:30, John Keeping wrote: >> > On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: >> >> Alex Bennée writes: >> >> >> >> > On 30 May 2013 16:33, Thomas Rast wrote

Re: Poor performance of git describe in big repos

2013-05-31 Thread Thomas Rast
Alex Bennée writes: > I think you are right. I was brave (well I assumed the tags would come > back from the upstream repo) and ran: > > git for-each-ref | grep "refs/tags" | grep "commit" | cut -d '/' -f 3 > | xargs git tag -d So that deleted all unannotated tags pointing at commits, and then i

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 31 May 2013 09:24, Thomas Rast wrote: > Alex Bennée writes: >> On 30 May 2013 20:30, John Keeping wrote: >>> On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée writes: > On 30 May 2013 16:33, Thomas Rast wrote: > No, my theory is that you tagged *the bl

Re: Poor performance of git describe in big repos

2013-05-31 Thread John Keeping
On Fri, May 31, 2013 at 09:14:49AM +0100, Alex Bennée wrote: > On 30 May 2013 20:30, John Keeping wrote: > > On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: > >> Alex Bennée writes: > >> > >> > On 30 May 2013 16:33, Thomas Rast wrote: > >> >> Alex Bennée writes: > > > >> > Will it

Re: Poor performance of git describe in big repos

2013-05-31 Thread Thomas Rast
Alex Bennée writes: > On 30 May 2013 20:30, John Keeping wrote: >> On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: >>> Alex Bennée writes: >>> >>> > On 30 May 2013 16:33, Thomas Rast wrote: >>> >> Alex Bennée writes: >> >>> > Will it be loading the blob for every commit it trave

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 30 May 2013 20:30, John Keeping wrote: > On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: >> Alex Bennée writes: >> >> > On 30 May 2013 16:33, Thomas Rast wrote: >> >> Alex Bennée writes: > >> > Will it be loading the blob for every commit it traverses or just ones >> > that hi

Re: Poor performance of git describe in big repos

2013-05-30 Thread John Keeping
On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: > Alex Bennée writes: > > > On 30 May 2013 16:33, Thomas Rast wrote: > >> Alex Bennée writes: > >> > >>> 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 > >>> 33.62% git libz.so.1.2.3.4 [.] inflate_fast > >>

Re: Poor performance of git describe in big repos

2013-05-30 Thread Antoine Pelisse
> The culprit, according to some callgrind investigation, is > lookup_commit_reference_gently() [for the unannotated case] or > deref_tag() [annotated case] calling parse_object(). Using the scenario you described earlier, I think it ends-up spending most of its time in check_sha1_signature (both

Re: Poor performance of git describe in big repos

2013-05-30 Thread Thomas Rast
Thomas Rast writes: > I had a brief look around sha1_file.c, in particular sha1_object_info, > and it turns out we lack the "deflate only early part" logic as I > suspected. So that'll have to be fixed first. After that I *think* it > should automatically carry over into the tag readers. Strik

Re: Poor performance of git describe in big repos

2013-05-30 Thread Thomas Rast
Alex Bennée writes: > On 30 May 2013 16:33, Thomas Rast wrote: >> Alex Bennée writes: >> >>> 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 >>> 33.62% git libz.so.1.2.3.4 [.] inflate_fast >>> 10.39% git libz.so.1.2.3.4 [.] adler32 >>> 2.03% git [kernel.k

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
On 30 May 2013 16:33, Thomas Rast wrote: > Alex Bennée writes: > >> 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 >> 33.62% git libz.so.1.2.3.4 [.] inflate_fast >> 10.39% git libz.so.1.2.3.4 [.] adler32 >> 2.03% git [kernel.kallsyms] [k] clear_page_c >

Re: Poor performance of git describe in big repos

2013-05-30 Thread Thomas Rast
Alex Bennée writes: > 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 > 33.62% git libz.so.1.2.3.4 [.] inflate_fast > 10.39% git libz.so.1.2.3.4 [.] adler32 > 2.03% git [kernel.kallsyms] [k] clear_page_c Do you have any large blobs in the repo that are r

Re: Poor performance of git describe in big repos

2013-05-30 Thread Ramkumar Ramachandra
Alex Bennée wrote: > 15:50 ajb@sloy/x86_64 [work.git] >time git log --pretty=oneline | wc -l > 24648 > > real0m0.434s > user0m0.388s > sys 0m0.112s > > Although it doesn't take too long to walk the whole mainline history > (obviously ignoring all the other branches). Damn, non-starter.

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
On 30 May 2013 15:32, Ramkumar Ramachandra wrote: > Alex Bennée wrote: >> And through my "special" repo: >> >> 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 >> 33.62% git libz.so.1.2.3.4 [.] inflate_fast >> 10.39% git libz.so.1.2.3.4 [.] adler32 >> 2.03% gi

Re: Poor performance of git describe in big repos

2013-05-30 Thread Ramkumar Ramachandra
Alex Bennée wrote: > And through my "special" repo: > > 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 > 33.62% git libz.so.1.2.3.4 [.] inflate_fast > 10.39% git libz.so.1.2.3.4 [.] adler32 > 2.03% git [kernel.kallsyms] [k] clear_page_c > > I'm not sure w

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
On 30 May 2013 14:45, Duy Nguyen wrote: > On Thu, May 30, 2013 at 8:34 PM, Alex Bennée wrote: > > Thanks. Can you share "verify-pack -v" output of > pack-a9ba133a6f25ffa74c3c407e09ab030f8745b201.pack? I think you need > to put it somewhere on Internet temporarily as it's likely to exceed > git@v

Re: Poor performance of git describe in big repos

2013-05-30 Thread Duy Nguyen
On Thu, May 30, 2013 at 8:34 PM, Alex Bennée wrote: > From the following run: > > > 14:31 ajb@sloy/x86_64 [work.git] >time /usr/bin/git --no-pager > describe --long --tags > ajb-build-test-5224-11-g9660048 > > real0m14.720s > user0m12.985s > sys 0m1.700s > 14:31 ajb@sloy/x86_64 [work.g

Re: Poor performance of git describe in big repos

2013-05-30 Thread Duy Nguyen
On Thu, May 30, 2013 at 7:29 PM, Alex Bennée wrote: > I ran perf on it and the top items in the report where: > > 41.58% git libcrypto.so.1.0.0 [.] 0x6ae73 > 33.96% git libz.so.1.2.3.4 [.] 0xe0ec > 10.39% git libz.so.1.2.3.4 [.] adler32 > 2.03% git [kernel.kallsyms] [k]

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
> You may find that performance improves if you repack with "git gc --aggressive". It seems that increases the time to get to where it wants to: 14:12 ajb@sloy/x86_64 [work.git] >time /usr/bin/git --no-pager describe --long --tags --debug searching to describe HEAD lightweight 10 ajb-build

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
It looks like it's a file caching effect combined with my repo being more pathalogical in size and contents. Note run 1 (cold) vs run 2 on the linux file tree: 13:52 ajb@sloy/x86_64 [linux.git] >time git describe --debug --long --tags HEAD~1 searching to describe HEAD~1 annotated

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
The repo is a fairly hairy one as it includes two historically un-related but content related repos which I'm the process of cherry-picking stuff across. 11:58 ajb@sloy/x86_64 [work.git] >git count-objects -v count: 493 size: 4572 in-pack: 399307 packs: 1 size-pack: 1930755 prune-packable: 0 garba

Re: Poor performance of git describe in big repos

2013-05-30 Thread John Keeping
On Thu, May 30, 2013 at 11:38:32AM +0100, Alex Bennée wrote: > One factor might be the size of my repo (.git is around 2.4G). Could > this just be due to computational cost of searching through large > packs to walk the commit chain? Is there any way to make this easier > for git to do? What does

Re: Poor performance of git describe in big repos

2013-05-30 Thread Ramkumar Ramachandra
Alex Bennée wrote: >>time /usr/bin/git --no-pager > traversed 223 commits > > real0m4.817s > user0m4.320s > sys 0m0.464s I'm quite clueless about why it is taking this long: I think it's IO because there's nothing to compute? I really can't trace anything unless you can reproduce it o

Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
Hi, I'm a fairly heavy user of the magit Emacs extension for interacting with my git repos. However I've noticed there are some cases where lag is very high. By analysing strace output of emacs calling git I found two commands that where particularly problematic when interrogating the repo: 11:00