Re: Huge performance bottleneck reading packs

2016-10-14 Thread Jeff King
On Fri, Oct 14, 2016 at 08:55:29AM +0200, Vegard Nossum wrote: > On 10/13/2016 10:43 PM, Jeff King wrote: > > No problem. I do think you'll benefit a lot from packing everything into > > a single pack, but it's also clear that Git was doing more work than it > > needed to be. This was a known issu

Re: Huge performance bottleneck reading packs

2016-10-14 Thread Jakub Narębski
W dniu 13.10.2016 o 11:04, Vegard Nossum pisze: > On 10/13/2016 01:47 AM, Jeff King wrote: >> On Wed, Oct 12, 2016 at 07:18:07PM -0400, Jeff King wrote: >> >>> Also, is it possible to make the repository in question available? I >>> might be able to reproduce based on your description, but it would

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Vegard Nossum
On 10/13/2016 10:43 PM, Jeff King wrote: No problem. I do think you'll benefit a lot from packing everything into a single pack, but it's also clear that Git was doing more work than it needed to be. This was a known issue when we added the racy-check to has_sha1_file(), and knew that we might ha

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Jeff King
On Thu, Oct 13, 2016 at 08:18:11PM +0200, Vegard Nossum wrote: > > My guess is that the number is relatively high. And that would explain > > why nobody else has really complained much; such a pattern is probably > > uncommon. > > I get ~3,700 objects "they are advertising that we don't have". >

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Vegard Nossum
On 10/13/2016 05:26 PM, Jeff King wrote: On Thu, Oct 13, 2016 at 09:20:07AM +0200, Vegard Nossum wrote: Does the patch below help? Yes, ~2m10s -> ~1m25s when I test a git fetch this morning (the other variation in time may be due to different CPU usage by other programs, but I ran with/withou

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Jeff King
On Thu, Oct 13, 2016 at 09:20:07AM +0200, Vegard Nossum wrote: > > Does the patch below help? > > Yes, ~2m10s -> ~1m25s when I test a git fetch this morning (the other > variation in time may be due to different CPU usage by other programs, > but I ran with/without the patch multiple times and th

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Jeff King
On Thu, Oct 13, 2016 at 09:17:34AM +0200, Vegard Nossum wrote: > Oops. I disabled gc a while ago; one reason I did that is that it takes > a long time to run and it has a tendency to kick in at the worst time. I > guess I should really put it in cron then. > > I'm not sure if this is related, but

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Vegard Nossum
On 10/13/2016 12:45 AM, Junio C Hamano wrote: > Vegard Nossum writes: > >> A closer inspection reveals the problem to really be that this is an >> extremely hot path with more than -- holy cow -- 4,106,756,451 >> iterations on the 'packed_git' list for a single 'git fetch' on my >> repository. I'

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Vegard Nossum
On 10/13/2016 01:47 AM, Jeff King wrote: On Wed, Oct 12, 2016 at 07:18:07PM -0400, Jeff King wrote: Also, is it possible to make the repository in question available? I might be able to reproduce based on your description, but it would save time if I could directly run gdb on your example. I

Re: Huge performance bottleneck reading packs

2016-10-13 Thread Vegard Nossum
On 10/13/2016 01:01 AM, Jeff King wrote: On Thu, Oct 13, 2016 at 12:30:52AM +0200, Vegard Nossum wrote: However, the commit found by 'git blame' above appears just fine to me, I haven't been able to spot a bug in it. A closer inspection reveals the problem to really be that this is an extremel

Re: Huge performance bottleneck reading packs

2016-10-12 Thread Jeff King
On Wed, Oct 12, 2016 at 07:18:07PM -0400, Jeff King wrote: > Also, is it possible to make the repository in question available? I > might be able to reproduce based on your description, but it would save > time if I could directly run gdb on your example. I tried this by making a bunch of packs i

Re: Huge performance bottleneck reading packs

2016-10-12 Thread Jeff King
On Wed, Oct 12, 2016 at 07:01:43PM -0400, Jeff King wrote: > On Thu, Oct 13, 2016 at 12:30:52AM +0200, Vegard Nossum wrote: > > > However, the commit found by 'git blame' above appears just fine to me, > > I haven't been able to spot a bug in it. > > > > A closer inspection reveals the problem t

Re: Huge performance bottleneck reading packs

2016-10-12 Thread Jeff King
On Thu, Oct 13, 2016 at 12:30:52AM +0200, Vegard Nossum wrote: > However, the commit found by 'git blame' above appears just fine to me, > I haven't been able to spot a bug in it. > > A closer inspection reveals the problem to really be that this is an > extremely hot path with more than -- holy

Re: Huge performance bottleneck reading packs

2016-10-12 Thread Junio C Hamano
Vegard Nossum writes: > A closer inspection reveals the problem to really be that this is an > extremely hot path with more than -- holy cow -- 4,106,756,451 > iterations on the 'packed_git' list for a single 'git fetch' on my > repository. I'm guessing the patch above just made the inner loop >

Huge performance bottleneck reading packs

2016-10-12 Thread Vegard Nossum
Hi all, I've bisected a performance regression (noticed by Quentin and myself) which caused a 'git fetch' to go from ~1m30s to ~2m40s: commit 47bf4b0fc52f3ad5823185a85f5f82325787c84b Author: Jeff King Date: Mon Jun 30 13:04:03 2014 -0400 prepare_packed_git_one: refactor duplicate-pack ch