Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-28 Thread Ben Peart
On 8/24/2018 2:40 PM, Duy Nguyen wrote: On Thu, Aug 23, 2018 at 7:33 PM Stefan Beller wrote: +core.fastIndex:: + Enable parallel index loading ++ +This can speed up operations like 'git diff' and 'git status' especially +when the index is very large. When enabled, Git will do the

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Ben Peart
On 8/24/2018 3:00 PM, Duy Nguyen wrote: On Fri, Aug 24, 2018 at 8:40 PM Ben Peart wrote: On 8/24/2018 2:20 PM, Duy Nguyen wrote: On Fri, Aug 24, 2018 at 5:37 PM Duy Nguyen wrote: On Thu, Aug 23, 2018 at 10:36 PM Ben Peart wrote: Nice to see this done without a new index extension

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Duy Nguyen
On Fri, Aug 24, 2018 at 8:40 PM Ben Peart wrote: > > > > On 8/24/2018 2:20 PM, Duy Nguyen wrote: > > On Fri, Aug 24, 2018 at 5:37 PM Duy Nguyen wrote: > >> On Thu, Aug 23, 2018 at 10:36 PM Ben Peart wrote: > Nice to see this done without a new index extension that records > offsets,

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Ben Peart
On 8/24/2018 2:20 PM, Duy Nguyen wrote: On Fri, Aug 24, 2018 at 5:37 PM Duy Nguyen wrote: On Thu, Aug 23, 2018 at 10:36 PM Ben Peart wrote: Nice to see this done without a new index extension that records offsets, so that we can load existing index files in parallel. Yes, I prefer this

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Duy Nguyen
On Thu, Aug 23, 2018 at 7:33 PM Stefan Beller wrote: > > +core.fastIndex:: > > + Enable parallel index loading > > ++ > > +This can speed up operations like 'git diff' and 'git status' especially > > +when the index is very large. When enabled, Git will do the index > > +loading from the

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Duy Nguyen
On Fri, Aug 24, 2018 at 5:37 PM Duy Nguyen wrote: > On Thu, Aug 23, 2018 at 10:36 PM Ben Peart wrote: > > > Nice to see this done without a new index extension that records > > > offsets, so that we can load existing index files in parallel. > > > > > > > Yes, I prefer this simpler model as

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Ben Peart
On 8/24/2018 11:57 AM, Duy Nguyen wrote: On Fri, Aug 24, 2018 at 05:37:20PM +0200, Duy Nguyen wrote: Since we're cutting corners to speed things up, could you try something like this? I notice that reading v4 is significantly slower than v2 and apparently strlen() (at least from glibc) is

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Duy Nguyen
On Fri, Aug 24, 2018 at 05:37:20PM +0200, Duy Nguyen wrote: > Since we're cutting corners to speed things up, could you try > something like this? > > I notice that reading v4 is significantly slower than v2 and > apparently strlen() (at least from glibc) is much cleverer and at > least gives me

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-24 Thread Duy Nguyen
Since we're cutting corners to speed things up, could you try something like this? I notice that reading v4 is significantly slower than v2 and apparently strlen() (at least from glibc) is much cleverer and at least gives me a few percentage time saving. diff --git a/read-cache.c b/read-cache.c

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-23 Thread Ben Peart
On 8/23/2018 2:06 PM, Junio C Hamano wrote: Ben Peart writes: This patch helps address the CPU cost of loading the index by creating multiple threads to divide the work of loading and converting the cache entries across all available CPU cores. Nice. +int

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-23 Thread Ben Peart
On 8/23/2018 1:31 PM, Stefan Beller wrote: On Thu, Aug 23, 2018 at 8:45 AM Ben Peart wrote: This patch helps address the CPU cost of loading the index by creating multiple threads to divide the work of loading and converting the cache entries across all available CPU cores. It

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-23 Thread Junio C Hamano
Ben Peart writes: > This patch helps address the CPU cost of loading the index by creating > multiple threads to divide the work of loading and converting the cache > entries across all available CPU cores. Nice. > +int git_config_get_fast_index(void) > +{ > + int val; > + > + if

Re: [PATCH v1] read-cache: speed up index load through parallelization

2018-08-23 Thread Stefan Beller
On Thu, Aug 23, 2018 at 8:45 AM Ben Peart wrote: > > This patch helps address the CPU cost of loading the index by creating > multiple threads to divide the work of loading and converting the cache > entries across all available CPU cores. > > It accomplishes this by having the primary thread

[PATCH v1] read-cache: speed up index load through parallelization

2018-08-23 Thread Ben Peart
This patch helps address the CPU cost of loading the index by creating multiple threads to divide the work of loading and converting the cache entries across all available CPU cores. It accomplishes this by having the primary thread loop across the index file tracking the offset and (for V4