Re: Using bitmaps to accelerate fetch and clone

2012-10-02 Thread Shawn Pearce
On Mon, Oct 1, 2012 at 5:48 AM, Nguyen Thai Ngoc Duy pclo...@gmail.com wrote: Probably off topic. Does saving a list of missing bases in the pack index help storing thin packs directly? I may be missing some points because I don't see why thin packs cannot be stored on disk in the first place.

Re: Using bitmaps to accelerate fetch and clone

2012-09-30 Thread Shawn Pearce
On Sun, Sep 30, 2012 at 6:59 PM, Nguyen Thai Ngoc Duy pclo...@gmail.com wrote: On Mon, Oct 1, 2012 at 8:07 AM, Shawn Pearce spea...@spearce.org wrote: You mentioned this before in your idea mail a while back. I wonder if it's worth storing bitmaps for all packs, not just the self contained

Re: Using bitmaps to accelerate fetch and clone

2012-09-28 Thread Nguyen Thai Ngoc Duy
On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce spea...@spearce.org wrote: * https://git.eclipse.org/r/7939 Defines the new E003 index format and the bit set implementation logic. Quote from the patch's message: Currently, the new index format can only be used with pack files that contain

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Nguyen Thai Ngoc Duy
On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce spea...@spearce.org wrote: Google has published a series of patches (see links below) to JGit to Should discussions about this series happen in here, jgit mailing or gerrit? I just want to make sure I'll discuss it at the right place. improve fetch

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Shawn Pearce
On Thu, Sep 27, 2012 at 5:17 AM, Nguyen Thai Ngoc Duy pclo...@gmail.com wrote: On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce spea...@spearce.org wrote: Google has published a series of patches (see links below) to JGit to Should discussions about this series happen in here, jgit mailing or

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 07:17:42PM +0700, Nguyen Thai Ngoc Duy wrote: Operation Index V2 Index VE003 Clone 37530ms (524.06 MiB) 82ms (524.06 MiB) Fetch (1 commit back) 75ms 107ms Fetch (10 commits back)

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Shawn Pearce
On Thu, Sep 27, 2012 at 10:20 AM, Jeff King p...@peff.net wrote: On Thu, Sep 27, 2012 at 07:17:42PM +0700, Nguyen Thai Ngoc Duy wrote: Operation Index V2 Index VE003 Clone 37530ms (524.06 MiB) 82ms (524.06 MiB) Fetch (1 commit back)

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 10:35:47AM -0700, Shawn O. Pearce wrote: If that is the case, do we need to bump the index version at all? Why not store a plain v2 index, and then store an additional file pack-XXX.reachable that contains the bitmaps and an independent version number. This is

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Shawn Pearce
On Thu, Sep 27, 2012 at 11:22 AM, Jeff King p...@peff.net wrote: I think clients will also want it. If we can make git rev-list --objects --all faster (which this should be able to do), we can speed up git prune, which in turn is by far the slowest part of git gc --auto, since in the typical

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 11:36:19AM -0700, Shawn O. Pearce wrote: Interesting. I would have assumed it depended on order in the index. No. We tried that. Assigning bits by order in index (aka order of SHA-1s sorted) results in horrible compression of the bitmap itself because of the uniform

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread David Michael Barr
Hi all, On Fri, Sep 28, 2012 at 3:20 AM, Jeff King p...@peff.net wrote: On Thu, Sep 27, 2012 at 07:17:42PM +0700, Nguyen Thai Ngoc Duy wrote: Operation Index V2 Index VE003 Clone 37530ms (524.06 MiB) 82ms (524.06 MiB) Fetch (1

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 02:52:29PM -0400, Jeff King wrote: No. The pack file name is composed from the SHA-1 of the sorted SHA-1s in the pack. Any change in compression settings or delta windows or even just random scheduling variations when repacking can cause offsets to slide, even if

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Junio C Hamano
Jeff King p...@peff.net writes: So yeah, we would want to put the pack trailer sha1 into the supplementary index file, and check that it matches when we open it. It's a slight annoyance, but it's O(1). Yes. If I am not mistaken, that is exactly how an .idx file makes sure that it describes

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Jeff King
On Thu, Sep 27, 2012 at 02:33:01PM -0700, Junio C Hamano wrote: Jeff King p...@peff.net writes: So yeah, we would want to put the pack trailer sha1 into the supplementary index file, and check that it matches when we open it. It's a slight annoyance, but it's O(1). Yes. If I am not

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Nguyen Thai Ngoc Duy
On Thu, Sep 27, 2012 at 9:33 PM, Shawn Pearce spea...@spearce.org wrote: I'd like to see some sort of extension mechanism like in $GIT_DIR/index, so that we don't have to increase pack index version often. This might be worthwhile. I dislike the way $GIT_DIR/index encodes extensions. Forcing

Re: Using bitmaps to accelerate fetch and clone

2012-09-27 Thread Nguyen Thai Ngoc Duy
On Fri, Sep 28, 2012 at 12:20 AM, Jeff King p...@peff.net wrote: Definitely :-). I have shown my interest in this topic before. So I should probably say that I'm going to work on this on C Git, but slllwwwly. As this benefits the server side greatly, perhaps a GitHubber ;-) might want to