Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-20 Thread Ben Peart
On 11/20/2017 6:51 PM, Ramsay Jones wrote: On 20/11/17 14:01, Ben Peart wrote: Further testing has revealed that switching from the regular heap to a refactored version of the mem_pool in fast-import.c produces similar gains as parallelizing do_index_load().  This appears to be a much

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-20 Thread Ramsay Jones
On 20/11/17 14:01, Ben Peart wrote: > Further testing has revealed that switching from the regular heap to a > refactored version of the mem_pool in fast-import.c produces similar gains as > parallelizing do_index_load().  This appears to be a much simpler patch for > similar gains so we will

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-20 Thread Jeff King
On Mon, Nov 20, 2017 at 09:20:35AM -0500, Jeff King wrote: > Out of curiosity, have you tried experimenting with any high-performance > 3rd-party allocator libraries? I've often wondered if we could get a > performance improvement from dropping in a new allocator, but was never > able to measure

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-20 Thread Jeff King
On Mon, Nov 20, 2017 at 09:01:45AM -0500, Ben Peart wrote: > Further testing has revealed that switching from the regular heap to a > refactored version of the mem_pool in fast-import.c produces similar gains > as parallelizing do_index_load(). This appears to be a much simpler patch > for

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-20 Thread Ben Peart
Further testing has revealed that switching from the regular heap to a refactored version of the mem_pool in fast-import.c produces similar gains as parallelizing do_index_load(). This appears to be a much simpler patch for similar gains so we will be pursuing that path. Combining the two

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Junio C Hamano
Ben Peart writes: > OK. I'll call this new extension "EOIE" ("end of index > entries"). Other than the standard header/footer, it will only contain > a 32 bit offset to the beginning of the extension entries. I'll > always write out that extension unless you would prefer it

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Ben Peart
On 11/14/2017 8:12 PM, Junio C Hamano wrote: Ben Peart writes: I have no thoughts or plans for changes in the future of IEOT (which I plan to rename ITOC). At this point in time, I can't even imagine what else we'd want as the index only contains cache entries, ...

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Ben Peart
On 11/14/2017 10:04 AM, Junio C Hamano wrote: Ben Peart writes: How about I add the logic to write out a special extension right before the SHA1 that contains an offset to the beginning of the extensions section. I will also add the logic in do_read_index() to search

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Junio C Hamano
Ben Peart writes: > How about I add the logic to write out a special extension right > before the SHA1 that contains an offset to the beginning of the > extensions section. I will also add the logic in do_read_index() to > search for and load this special extension if it

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Ben Peart
On 11/13/2017 8:10 PM, Junio C Hamano wrote: Ben Peart writes: The proposed format for extensions (ie having both a header and a footer with name and size) in the current patch already enables having multiple extensions that can be parsed forward or backward. Any

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-13 Thread Junio C Hamano
Ben Peart writes: > The proposed format for extensions (ie having both a header and a > footer with name and size) in the current patch already enables having > multiple extensions that can be parsed forward or backward. Any > extensions that would want to be parse-able in

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-13 Thread Ben Peart
On 11/9/2017 11:46 PM, Junio C Hamano wrote: Ben Peart writes: To make this work for V4 indexes, when writing the index, it periodically "resets" the prefix-compression by encoding the current entry as if the path name for the previous entry is completely different

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-09 Thread Junio C Hamano
Ben Peart writes: > To make this work for V4 indexes, when writing the index, it periodically > "resets" the prefix-compression by encoding the current entry as if the > path name for the previous entry is completely different and saves the > offset of that entry in the