Re: Poor performance of git describe in big repos

2013-06-03 Thread Alex Bennée
On 31 May 2013 10:57, Alex Bennée kernel-hac...@bennee.com wrote: On 31 May 2013 09:46, Thomas Rast tr...@inf.ethz.ch wrote: So that deleted all unannotated tags pointing at commits, and then it was fast. Curious. However, if that turns out to be the culprit, it's not fixable currently[1].

Re: Poor performance of git describe in big repos

2013-06-03 Thread Alex Bennée
On 31 May 2013 17:17, Jeff King p...@peff.net wrote: On Fri, May 31, 2013 at 12:27:11PM +0200, Thomas Rast wrote: Thomas Rast tr...@inf.ethz.ch writes: However, if that turns out to be the culprit, it's not fixable currently[1]. Having commits with insanely long messages is just, well,

Re: Poor performance of git describe in big repos

2013-06-03 Thread Junio C Hamano
Alex Bennée kernel-hac...@bennee.com writes: Why is git attempting to parse a commit not on the DAG for the branch I'm attempting to describe? I think that is because you need to parse the objects at the tip of refs to see if they are on the DAG in the first place. If there weren't any

Re: Poor performance of git describe in big repos

2013-06-03 Thread Junio C Hamano
Junio C Hamano gits...@pobox.com writes: Alex Bennée kernel-hac...@bennee.com writes: Why is git attempting to parse a commit not on the DAG for the branch I'm attempting to describe? I think that is because you need to parse the objects at the tip of refs to see if they are on the DAG in

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 30 May 2013 20:30, John Keeping j...@keeping.me.uk wrote: On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 16:33, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée kernel-hac...@bennee.com writes: snip Will it be

Re: Poor performance of git describe in big repos

2013-05-31 Thread Thomas Rast
Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 20:30, John Keeping j...@keeping.me.uk wrote: On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 16:33, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée

Re: Poor performance of git describe in big repos

2013-05-31 Thread John Keeping
On Fri, May 31, 2013 at 09:14:49AM +0100, Alex Bennée wrote: On 30 May 2013 20:30, John Keeping j...@keeping.me.uk wrote: On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 16:33, Thomas Rast tr...@inf.ethz.ch wrote:

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 31 May 2013 09:24, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 20:30, John Keeping j...@keeping.me.uk wrote: On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 16:33,

Re: Poor performance of git describe in big repos

2013-05-31 Thread Thomas Rast
Alex Bennée kernel-hac...@bennee.com writes: I think you are right. I was brave (well I assumed the tags would come back from the upstream repo) and ran: git for-each-ref | grep refs/tags | grep commit | cut -d '/' -f 3 | xargs git tag -d So that deleted all unannotated tags pointing at

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 31 May 2013 09:32, John Keeping j...@keeping.me.uk wrote: On Fri, May 31, 2013 at 09:14:49AM +0100, Alex Bennée wrote: On 30 May 2013 20:30, John Keeping j...@keeping.me.uk wrote: On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée kernel-hac...@bennee.com writes:

Re: Poor performance of git describe in big repos

2013-05-31 Thread John Keeping
On Fri, May 31, 2013 at 09:49:57AM +0100, Alex Bennée wrote: On 31 May 2013 09:32, John Keeping j...@keeping.me.uk wrote: Thomas is suggesting that you might have a tag that does not point at a commit but instead points to a blob object. It's looking like I just have some very heavy

Re: Poor performance of git describe in big repos

2013-05-31 Thread Alex Bennée
On 31 May 2013 09:46, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée kernel-hac...@bennee.com writes: I think you are right. I was brave (well I assumed the tags would come back from the upstream repo) and ran: git for-each-ref | grep refs/tags | grep commit | cut -d '/' -f 3 | xargs git

Re: Poor performance of git describe in big repos

2013-05-31 Thread Thomas Rast
Thomas Rast tr...@inf.ethz.ch writes: However, if that turns out to be the culprit, it's not fixable currently[1]. Having commits with insanely long messages is just, well, insane. [1] unless we do a major rework of the loading infrastructure, so that we can teach it to load only the

Re: Poor performance of git describe in big repos

2013-05-31 Thread Jeff King
On Fri, May 31, 2013 at 12:27:11PM +0200, Thomas Rast wrote: Thomas Rast tr...@inf.ethz.ch writes: However, if that turns out to be the culprit, it's not fixable currently[1]. Having commits with insanely long messages is just, well, insane. [1] unless we do a major rework of the

Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
Hi, I'm a fairly heavy user of the magit Emacs extension for interacting with my git repos. However I've noticed there are some cases where lag is very high. By analysing strace output of emacs calling git I found two commands that where particularly problematic when interrogating the repo:

Re: Poor performance of git describe in big repos

2013-05-30 Thread Ramkumar Ramachandra
Alex Bennée wrote: time /usr/bin/git --no-pager traversed 223 commits real0m4.817s user0m4.320s sys 0m0.464s I'm quite clueless about why it is taking this long: I think it's IO because there's nothing to compute? I really can't trace anything unless you can reproduce it on a

Re: Poor performance of git describe in big repos

2013-05-30 Thread John Keeping
On Thu, May 30, 2013 at 11:38:32AM +0100, Alex Bennée wrote: One factor might be the size of my repo (.git is around 2.4G). Could this just be due to computational cost of searching through large packs to walk the commit chain? Is there any way to make this easier for git to do? What does git

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
The repo is a fairly hairy one as it includes two historically un-related but content related repos which I'm the process of cherry-picking stuff across. 11:58 ajb@sloy/x86_64 [work.git] git count-objects -v count: 493 size: 4572 in-pack: 399307 packs: 1 size-pack: 1930755 prune-packable: 0

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
It looks like it's a file caching effect combined with my repo being more pathalogical in size and contents. Note run 1 (cold) vs run 2 on the linux file tree: 13:52 ajb@sloy/x86_64 [linux.git] time git describe --debug --long --tags HEAD~1 searching to describe HEAD~1 annotated

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
You may find that performance improves if you repack with git gc --aggressive. It seems that increases the time to get to where it wants to: 14:12 ajb@sloy/x86_64 [work.git] time /usr/bin/git --no-pager describe --long --tags --debug searching to describe HEAD lightweight 10

Re: Poor performance of git describe in big repos

2013-05-30 Thread Duy Nguyen
On Thu, May 30, 2013 at 7:29 PM, Alex Bennée kernel-hac...@bennee.com wrote: I ran perf on it and the top items in the report where: 41.58% git libcrypto.so.1.0.0 [.] 0x6ae73 33.96% git libz.so.1.2.3.4 [.] 0xe0ec 10.39% git libz.so.1.2.3.4 [.] adler32 2.03% git

Re: Poor performance of git describe in big repos

2013-05-30 Thread Duy Nguyen
On Thu, May 30, 2013 at 8:34 PM, Alex Bennée kernel-hac...@bennee.com wrote: From the following run: 14:31 ajb@sloy/x86_64 [work.git] time /usr/bin/git --no-pager describe --long --tags ajb-build-test-5224-11-g9660048 real0m14.720s user0m12.985s sys 0m1.700s 14:31

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
On 30 May 2013 14:45, Duy Nguyen pclo...@gmail.com wrote: On Thu, May 30, 2013 at 8:34 PM, Alex Bennée kernel-hac...@bennee.com wrote: snip Thanks. Can you share verify-pack -v output of pack-a9ba133a6f25ffa74c3c407e09ab030f8745b201.pack? I think you need to put it somewhere on Internet

Re: Poor performance of git describe in big repos

2013-05-30 Thread Ramkumar Ramachandra
Alex Bennée wrote: And through my special repo: 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 33.62% git libz.so.1.2.3.4 [.] inflate_fast 10.39% git libz.so.1.2.3.4 [.] adler32 2.03% git [kernel.kallsyms] [k] clear_page_c I'm not sure why

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
On 30 May 2013 15:32, Ramkumar Ramachandra artag...@gmail.com wrote: Alex Bennée wrote: And through my special repo: 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 33.62% git libz.so.1.2.3.4 [.] inflate_fast 10.39% git libz.so.1.2.3.4 [.] adler32 2.03%

Re: Poor performance of git describe in big repos

2013-05-30 Thread Ramkumar Ramachandra
Alex Bennée wrote: 15:50 ajb@sloy/x86_64 [work.git] time git log --pretty=oneline | wc -l 24648 real0m0.434s user0m0.388s sys 0m0.112s Although it doesn't take too long to walk the whole mainline history (obviously ignoring all the other branches). Damn, non-starter.

Re: Poor performance of git describe in big repos

2013-05-30 Thread Thomas Rast
Alex Bennée kernel-hac...@bennee.com writes: 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 33.62% git libz.so.1.2.3.4 [.] inflate_fast 10.39% git libz.so.1.2.3.4 [.] adler32 2.03% git [kernel.kallsyms] [k] clear_page_c Do you have any large blobs

Re: Poor performance of git describe in big repos

2013-05-30 Thread Alex Bennée
On 30 May 2013 16:33, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée kernel-hac...@bennee.com writes: 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 33.62% git libz.so.1.2.3.4 [.] inflate_fast 10.39% git libz.so.1.2.3.4 [.] adler32 2.03% git

Re: Poor performance of git describe in big repos

2013-05-30 Thread Thomas Rast
Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 16:33, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée kernel-hac...@bennee.com writes: 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 33.62% git libz.so.1.2.3.4 [.] inflate_fast 10.39% git

Re: Poor performance of git describe in big repos

2013-05-30 Thread Thomas Rast
Thomas Rast tr...@inf.ethz.ch writes: I had a brief look around sha1_file.c, in particular sha1_object_info, and it turns out we lack the deflate only early part logic as I suspected. So that'll have to be fixed first. After that I *think* it should automatically carry over into the tag

Re: Poor performance of git describe in big repos

2013-05-30 Thread Antoine Pelisse
The culprit, according to some callgrind investigation, is lookup_commit_reference_gently() [for the unannotated case] or deref_tag() [annotated case] calling parse_object(). Using the scenario you described earlier, I think it ends-up spending most of its time in check_sha1_signature (both

Re: Poor performance of git describe in big repos

2013-05-30 Thread John Keeping
On Thu, May 30, 2013 at 06:21:55PM +0200, Thomas Rast wrote: Alex Bennée kernel-hac...@bennee.com writes: On 30 May 2013 16:33, Thomas Rast tr...@inf.ethz.ch wrote: Alex Bennée kernel-hac...@bennee.com writes: 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 33.62%