Re: [PATCH 00/32] Split index mode for very large indexes

2014-05-09 Thread Duy Nguyen
On Mon, Apr 28, 2014 at 02:18:44PM -0700, Shawn Pearce wrote: The read penalty is not addressed here, so I still pay 14MB hashing cost. But that's an easy problem. We could cache the validated index in a daemon. Whenever git needs to load an index, it pokes the daemon. The daemon verifies

Re: [PATCH 00/32] Split index mode for very large indexes

2014-05-09 Thread Junio C Hamano
Duy Nguyen pclo...@gmail.com writes: On Mon, Apr 28, 2014 at 02:18:44PM -0700, Shawn Pearce wrote: The read penalty is not addressed here, so I still pay 14MB hashing cost. But that's an easy problem. We could cache the validated index in a daemon. Whenever git needs to load an index, it

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-30 Thread Richard Hansen
On 2014-04-28 06:55, Nguyễn Thái Ngọc Duy wrote: From the user point of view, this reduces the writable size of index down to the number of updated files. For example my webkit index v4 is 14MB. With a fresh split, I only have to update an index of 200KB. Every file I touch will add about 80

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-30 Thread Duy Nguyen
On Wed, Apr 30, 2014 at 04:48:05PM -0400, Richard Hansen wrote: I played around with these changes a bit and have some questions: * These changes should only affect performance when the index is updated, right? In other words, if I do git status; git status the second git status

[PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Nguyễn Thái Ngọc Duy
I hinted about it earlier [1]. It now passes the test suite and with a design that I'm happy with (thanks to Junio for a suggestion about the rename problem). From the user point of view, this reduces the writable size of index down to the number of updated files. For example my webkit index v4

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Shawn Pearce
On Mon, Apr 28, 2014 at 3:55 AM, Nguyễn Thái Ngọc Duy pclo...@gmail.com wrote: I hinted about it earlier [1]. It now passes the test suite and with a design that I'm happy with (thanks to Junio for a suggestion about the rename problem). From the user point of view, this reduces the writable

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Junio C Hamano
Nguyễn Thái Ngọc Duy pclo...@gmail.com writes: The read penalty is not addressed here, so I still pay 14MB hashing cost. Hmm, yeah, the cost for verify_hdr() would still matter, and presumably you would be hashing the additional 200kB to validate the smaller changes since the base file to

Re: [PATCH 00/32] Split index mode for very large indexes

2014-04-28 Thread Duy Nguyen
On Tue, Apr 29, 2014 at 4:18 AM, Shawn Pearce spea...@spearce.org wrote: On Mon, Apr 28, 2014 at 3:55 AM, Nguyễn Thái Ngọc Duy pclo...@gmail.com wrote: I hinted about it earlier [1]. It now passes the test suite and with a design that I'm happy with (thanks to Junio for a suggestion about the