On Wed, Mar 27, 2013 at 1:00 PM, Junio C Hamano <gits...@pobox.com> wrote:
>
> Given that we haven't tweaked the parallelism or thread-cost
> parameters since the inception of the mechanism in Nov 2008, I
> suspect that we would see praises from some and grievances from
> other corners of the user base for a while until we find acceptable
> values for them
Looking at the parameters again, I really think they are pretty sane,
and I don't think the numbers are all that likely to have shifted from
2008. The maximum thread value is quite reasonable: twenty threads is
sufficient to cover quite a bit of latency, and brings "several
seconds" down to "under half a second" for any truly IO-limited load,
while not being disastrous for the case where everything is in cache
and we only have a limited number of CPU cores.
And the "at least 500 files per thread" limit is eminently reasonable
too - smaller projects like git won't have more than five or so
threads.
So I'd be very surprised if the values need much tweaking. Sure, there
might be some extreme cases that might tune for some particular
patterns, and maybe we should make the values be tunable rather than
totally hardcoded, but I suspect there's limited up-side.
It might be interesting for the people who really like tuning, though.
So in addition to "index.preload=true", maybe an extended config
format like "index_preload=50,200" to say "maximum of fifty threads,
for every 200 files" could be done just so people could play around
with the numbers and see how much (if at all) they actually matter.
But I really don't think the original 20/500 rule is likely to be all
that bad for anybody. Unless there is some *really* sucky thread
library out there (ie fully user-space threads, so filename lookup
isn't actually parallelised at all), but at least for that case the
fix is to just say "ok, your threads aren't real threads, so just
disable index preloading entirely).
Linus
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
