Re: Using bitmaps to accelerate fetch and clone

2012-10-02 Thread Shawn Pearce
On Mon, Oct 1, 2012 at 5:48 AM, Nguyen Thai Ngoc Duy wrote: > Probably off topic. Does saving a list of missing bases in the pack > index help storing thin packs directly? I may be missing some points > because I don't see why thin packs cannot be stored on disk in the > first place. Packs are su

Re: Using bitmaps to accelerate fetch and clone

2012-09-30 Thread Shawn Pearce
On Sun, Sep 30, 2012 at 6:59 PM, Nguyen Thai Ngoc Duy wrote: > On Mon, Oct 1, 2012 at 8:07 AM, Shawn Pearce wrote: >>> You mentioned this before in your idea mail a while back. I wonder if >>> it's worth storing bitmaps for all packs, not just the self contained >>> ones. >> >> Colby and I starte

Re: Using bitmaps to accelerate fetch and clone

2012-09-30 Thread Shawn Pearce
On Fri, Sep 28, 2012 at 5:00 AM, Nguyen Thai Ngoc Duy wrote: > On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce wrote: >> * https://git.eclipse.org/r/7939 >> >> Defines the new E003 index format and the bit set >> implementation logic. > > Quote from the patch's message: > > "Currently, the new

Re: Using bitmaps to accelerate fetch and clone

2012-09-28 Thread Nguyen Thai Ngoc Duy
On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce wrote: > * https://git.eclipse.org/r/7939 > > Defines the new E003 index format and the bit set > implementation logic. Quote from the patch's message: "Currently, the new index format can only be used with pack files that contain a complete clos

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Nguyen Thai Ngoc Duy
On Fri, Sep 28, 2012 at 12:20 AM, Jeff King wrote: >> Definitely :-). I have shown my interest in this topic before. So I >> should probably say that I'm going to work on this on C Git, but >> slllwwwly. As this benefits the server side greatly, perhaps a >> GitHubber ;-) might want to work on

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Nguyen Thai Ngoc Duy
On Thu, Sep 27, 2012 at 9:33 PM, Shawn Pearce wrote: >> I'd like to see some sort of extension mechanism like in >> $GIT_DIR/index, so that we don't have to increase pack index version >> often. > > This might be worthwhile. I dislike the way $GIT_DIR/index encodes > extensions. Forcing an extensi

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 02:33:01PM -0700, Junio C Hamano wrote: > Jeff King writes: > > > So yeah, we would want to put the pack trailer sha1 into the > > supplementary index file, and check that it matches when we open it. > > It's a slight annoyance, but it's O(1). > > Yes. If I am not mista

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Junio C Hamano
Jeff King writes: > So yeah, we would want to put the pack trailer sha1 into the > supplementary index file, and check that it matches when we open it. > It's a slight annoyance, but it's O(1). Yes. If I am not mistaken, that is exactly how an .idx file makes sure that it describes the matching

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 02:52:29PM -0400, Jeff King wrote: > > No. The pack file name is composed from the SHA-1 of the sorted SHA-1s > > in the pack. Any change in compression settings or delta windows or > > even just random scheduling variations when repacking can cause > > offsets to slide, ev

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread David Michael Barr
Hi all, On Fri, Sep 28, 2012 at 3:20 AM, Jeff King wrote: > On Thu, Sep 27, 2012 at 07:17:42PM +0700, Nguyen Thai Ngoc Duy wrote: > >> > Operation Index V2 Index VE003 >> > Clone 37530ms (524.06 MiB) 82ms (524.06 MiB) >> > Fetch (1 commit

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 11:36:19AM -0700, Shawn O. Pearce wrote: > > Interesting. I would have assumed it depended on order in the index. > > No. We tried that. Assigning bits by order in index (aka order of > SHA-1s sorted) results in horrible compression of the bitmap itself > because of the un

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Shawn Pearce
On Thu, Sep 27, 2012 at 11:22 AM, Jeff King wrote: > > I think clients will also want it. If we can make "git rev-list > --objects --all" faster (which this should be able to do), we can speed > up "git prune", which in turn is by far the slowest part of "git gc > --auto", since in the typical cas

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 10:35:47AM -0700, Shawn O. Pearce wrote: > > If that is the case, do we need to bump the index > > version at all? Why not store a plain v2 index, and then store an > > additional file "pack-XXX.reachable" that contains the bitmaps and an > > independent version number. >

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Shawn Pearce
On Thu, Sep 27, 2012 at 10:20 AM, Jeff King wrote: > On Thu, Sep 27, 2012 at 07:17:42PM +0700, Nguyen Thai Ngoc Duy wrote: > >> > Operation Index V2 Index VE003 >> > Clone 37530ms (524.06 MiB) 82ms (524.06 MiB) >> > Fetch (1 commit back)

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 07:17:42PM +0700, Nguyen Thai Ngoc Duy wrote: > > Operation Index V2 Index VE003 > > Clone 37530ms (524.06 MiB) 82ms (524.06 MiB) > > Fetch (1 commit back) 75ms 107ms > > Fetch (10 commits ba

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Shawn Pearce
On Thu, Sep 27, 2012 at 5:17 AM, Nguyen Thai Ngoc Duy wrote: > On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce wrote: >> Google has published a series of patches (see links below) to JGit to > > Should discussions about this series happen in here, jgit mailing or > gerrit? I just want to make sure

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Nguyen Thai Ngoc Duy
On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce wrote: > Google has published a series of patches (see links below) to JGit to Should discussions about this series happen in here, jgit mailing or gerrit? I just want to make sure I'll discuss it at the right place. > improve fetch and clone perform

Using bitmaps to accelerate fetch and clone

2012-09-26 Thread Shawn Pearce
Google has published a series of patches (see links below) to JGit to improve fetch and clone performance by adding compressed bitmaps to the pack-*.idx structure. Operation Index V2 Index VE003 Clone 37530ms (524.06 MiB) 82ms (524.06 MiB)