Re: [PATCH 00/32] Split index mode for very large indexes

2014-05-09 Thread Junio C Hamano
Duy Nguyen writes: > On Mon, Apr 28, 2014 at 02:18:44PM -0700, Shawn Pearce wrote: >> > The read penalty is not addressed here, so I still pay 14MB hashing >> > cost. But that's an easy problem. We could cache the validated index >> > in a daemon. Whenever git needs to load an index, it pokes the

Re: [PATCH 00/32] Split index mode for very large indexes

2014-05-09 Thread Duy Nguyen
On Mon, Apr 28, 2014 at 02:18:44PM -0700, Shawn Pearce wrote: > > The read penalty is not addressed here, so I still pay 14MB hashing > > cost. But that's an easy problem. We could cache the validated index > > in a daemon. Whenever git needs to load an index, it pokes the daemon. > > The daemon ve

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-30 Thread Duy Nguyen
On Wed, Apr 30, 2014 at 04:48:05PM -0400, Richard Hansen wrote: > I played around with these changes a bit and have some questions: > > * These changes should only affect performance when the index is > updated, right? In other words, if I do "git status; git status" > the second "git s

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-30 Thread Richard Hansen
On 2014-04-28 06:55, Nguyễn Thái Ngọc Duy wrote: > From the user point of view, this reduces the writable size of index > down to the number of updated files. For example my webkit index v4 is > 14MB. With a fresh split, I only have to update an index of 200KB. > Every file I touch will add about 8

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Duy Nguyen
On Tue, Apr 29, 2014 at 4:18 AM, Shawn Pearce wrote: > On Mon, Apr 28, 2014 at 3:55 AM, Nguyễn Thái Ngọc Duy > wrote: >> I hinted about it earlier [1]. It now passes the test suite and with a >> design that I'm happy with (thanks to Junio for a suggestion about the >> rename problem). >> >> From

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Junio C Hamano
Nguyễn Thái Ngọc Duy writes: > The read penalty is not addressed here, so I still pay 14MB hashing > cost. Hmm, yeah, the cost for verify_hdr() would still matter, and presumably you would be hashing the additional 200kB to validate the smaller "changes since the base" file to give users the sa

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Shawn Pearce
On Mon, Apr 28, 2014 at 3:55 AM, Nguyễn Thái Ngọc Duy wrote: > I hinted about it earlier [1]. It now passes the test suite and with a > design that I'm happy with (thanks to Junio for a suggestion about the > rename problem). > > From the user point of view, this reduces the writable size of index