Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-16 Thread Junio C Hamano
Shawn Pearce writes: > On Wed, Aug 15, 2012 at 10:42 PM, Junio C Hamano wrote: > ... >> An obvious way to record the "delta chain" is to simply keep the >> name_hash of each object in the pack, which would need 2 bytes per >> object in the pack, that would bloat pack_idx_entry size from 32 >> by

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-16 Thread Shawn Pearce
On Wed, Aug 15, 2012 at 10:42 PM, Junio C Hamano wrote: > Shawn Pearce writes: > >> ... But I think its worth giving >> him a few weeks to finish getting the code ready, vs. rushing >> something in that someone else thinks might help. We have waited more >> than 6 years or whatever to improve pac

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-15 Thread Junio C Hamano
Shawn Pearce writes: > ... But I think its worth giving > him a few weeks to finish getting the code ready, vs. rushing > something in that someone else thinks might help. We have waited more > than 6 years or whatever to improve packing. Colby's experiments are > showing massive improvements (40

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-13 Thread Nguyen Thai Ngoc Duy
On Tue, Aug 14, 2012 at 7:46 AM, Shawn Pearce wrote: > Colby is nearly done prototyping the bitmap reachability > implementation in JGit and will release the code under the BSD license > there soon. I can't promise when yet because Colby will soon be > heading out for some (much deserved) vacation

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-13 Thread Shawn Pearce
Let me start by echoing Junio's remark... "lacks sufficient justification". You don't give enough evidence to support even why it is worth looking at this commit, let alone why it should be included and cause a format change in the idx file format. At some point you start to hand-wave about how it

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-13 Thread Junio C Hamano
Nguyen Thai Ngoc Duy writes: > On Mon, Aug 13, 2012 at 2:49 AM, Junio C Hamano wrote: >> For example, the reachability bitmap would want to say something >> like "Traversing from commit A, these objects in this pack are >> reachable." The bitmap for one commit A would logically consist of >> N

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-12 Thread Nguyen Thai Ngoc Duy
On Mon, Aug 13, 2012 at 2:49 AM, Junio C Hamano wrote: > For example, the reachability bitmap would want to say something > like "Traversing from commit A, these objects in this pack are > reachable." The bitmap for one commit A would logically consist of > N bits for a packfile that stores N obj

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-12 Thread Junio C Hamano
Junio C Hamano writes: > Nguyễn Thái Ngọc Duy writes: > >> The main reason to group objects by type is to make it possible to >> create another sha1->something mapping for a particular object type, >> without wasting space for storing sha-1 keys again. For example, we >> can store commit caches

Re: [PATCH/RFC] index-pack: produce pack index version 3

2012-08-12 Thread Junio C Hamano
Nguyễn Thái Ngọc Duy writes: > The main reason to group objects by type is to make it possible to > create another sha1->something mapping for a particular object type, > without wasting space for storing sha-1 keys again. For example, we > can store commit caches, tree caches... at the end of t

[PATCH/RFC] index-pack: produce pack index version 3

2012-08-12 Thread Nguyễn Thái Ngọc Duy
The sha-1->offset table in v3 is sorted by object type first, then sha-1 (as opposed to sorted by sha-1 only in v2). There are four fan-out tables, one for each object type. So to look for the offset of an object, we first locate the fan-out table based on object type, then do binary search in that