Re: git status takes 30 seconds on Windows 7. Why?
On Wed, Mar 27, 2013 at 11:39 PM, Jim Kinsman wrote: > git status takes 30 seconds on Windows 7. Here are some stats: > git ls-files | wc -l > 27330 > > git ls-files -o | wc -l > 4 > > $ git diff --name-only | xargs du -chs > 68K update_import_contacts.php > 68K total > > What can I do??? This is so slow it is unbearable. > By the way i've done git gc several times and nothing changed. You can try "status -uno" to skip showing untracked files (and may be do without -uno before commit so you don't miss files). You may also try core.ignoreStat (but I think it's not very convenient to use) -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On Wed, Mar 27, 2013 at 1:00 PM, Junio C Hamano wrote: > > Given that we haven't tweaked the parallelism or thread-cost > parameters since the inception of the mechanism in Nov 2008, I > suspect that we would see praises from some and grievances from > other corners of the user base for a while until we find acceptable > values for them Looking at the parameters again, I really think they are pretty sane, and I don't think the numbers are all that likely to have shifted from 2008. The maximum thread value is quite reasonable: twenty threads is sufficient to cover quite a bit of latency, and brings "several seconds" down to "under half a second" for any truly IO-limited load, while not being disastrous for the case where everything is in cache and we only have a limited number of CPU cores. And the "at least 500 files per thread" limit is eminently reasonable too - smaller projects like git won't have more than five or so threads. So I'd be very surprised if the values need much tweaking. Sure, there might be some extreme cases that might tune for some particular patterns, and maybe we should make the values be tunable rather than totally hardcoded, but I suspect there's limited up-side. It might be interesting for the people who really like tuning, though. So in addition to "index.preload=true", maybe an extended config format like "index_preload=50,200" to say "maximum of fifty threads, for every 200 files" could be done just so people could play around with the numbers and see how much (if at all) they actually matter. But I really don't think the original 20/500 rule is likely to be all that bad for anybody. Unless there is some *really* sucky thread library out there (ie fully user-space threads, so filename lookup isn't actually parallelised at all), but at least for that case the fix is to just say "ok, your threads aren't real threads, so just disable index preloading entirely). Linus -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
Linus Torvalds writes: > On Wed, Mar 27, 2013 at 12:04 PM, Jeff King wrote: >> >> Yes, I think that's pretty much the case (though most of my >> Git-on-Windows experience is from cygwin long ago, where the stat >> performance was truly horrendous). Have you tried setting >> core.preloadindex, which should run the stats in parallel? > > I wonder if preloadindex shouldn't be enabled by default. I am surprised that we haven't done so. Given that we haven't tweaked the parallelism or thread-cost parameters since the inception of the mechanism in Nov 2008, I suspect that we would see praises from some and grievances from other corners of the user base for a while until we find acceptable values for them, but I agree the feature has been in use sufficiently by some people (heh, I just discovered that I don't have it in my config), it can be the default. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On Wed, Mar 27, 2013 at 12:04 PM, Jeff King wrote: > > Yes, I think that's pretty much the case (though most of my > Git-on-Windows experience is from cygwin long ago, where the stat > performance was truly horrendous). Have you tried setting > core.preloadindex, which should run the stats in parallel? I wonder if preloadindex shouldn't be enabled by default.. It's a huge deal on NFS, and the only real downside is that it expects threading to work. It potentially slows things down a tiny bit for single-CPU cases with everything cached, but that isn't likely to be a relevant case. Of course, it can trigger filesystem scalability issues, and as a result it will often not help very much if you have the bulk of your files in one (or a few) directories. But anybody who has so many files that performance is an issue is not likely to have them all in one place. And apparently the Windows FS metadata caching sucks, and things fall out of the cache for large trees. Color me not-very-surprised. It's probably some size limit on the metadata that you can tweak. So I';m sure there's some registry setting or other that would make windows able to cache more than a few thousand filenames, and it would probably improve performance a lot, but I do think preloadindex has been around long enough that it could just be the default. Of course, Jim should verify that preloadindex actually does solve his problem. With 20k+ files, it should max out the 20 IO threads for preloading, and assuming the filesystem IO scales reasonably well, it should fix the problem. But we do do a number of metadata ops synchronously even with preloadindex, so things won't scale perfectly. (In particular: do open each directory and do the readdir stuff and try to open .gitignore whether it exists or not. So you'll get synchronous IO for each directory, but at least the per-file IO to check all the file stat data should scale). Linus -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On Wed, Mar 27, 2013 at 06:46:57PM +, John Keeping wrote: > I think the simple reality is that Git was written with the assumption > that stat is cheap and that isn't really the case on Windows, where the > filesystem cache doesn't seem to do that well with this. Yes, I think that's pretty much the case (though most of my Git-on-Windows experience is from cygwin long ago, where the stat performance was truly horrendous). Have you tried setting core.preloadindex, which should run the stats in parallel? -Peff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On Wed, Mar 27, 2013 at 01:15:43PM -0500, Jim Kinsman wrote: > The only anti-virus I have installed is Microsoft Security Essentials > I turned off and it was still the same: > $ cat /usr/bin/gitstatus > start_time=`date +%s` > git status && echo run time is $(expr `date +%s` - $start_time) s > > > $ gitstatus > # On branch test > # Changes not staged for commit: > # (use "git add ..." to update what will be committed) > # (use "git checkout -- ..." to discard changes in working directory) > # > # modified: orgoptions.php > # modified: update_import_contacts.php > # > no changes added to commit (use "git add" and/or "git commit -a") > run time is 10 s That doesn't seem hugely surprising to me. I have a moderately sized repository (3047 files, although it's Java so there are some deep trees) and I get the following (Vista on a reasonably old laptop, best of 3, Git version 1.8.1.msysgit.1): $ time git ls-files >/dev/null real0m0.047s user0m0.015s sys 0m0.015s $ time git status >/dev/null real0m2.715s user0m0.000s sys 0m0.031s I'm not sure the "user" and "sys" times are correct, but the "real" times feel right. By comparison, on Linux on a much newer machine (so not much of a comparison) on the same repository: $ time git status >/dev/null real0m0.347s user0m0.171s sys 0m0.167s I think the simple reality is that Git was written with the assumption that stat is cheap and that isn't really the case on Windows, where the filesystem cache doesn't seem to do that well with this. It may be that Git's Windows compatibility code could do be made more efficient but I know nothing about that, although a quick look in compat/mingw.c indicates that Git does already use its own stat implementations in place of the MSys ones in search of speed. John -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
The only anti-virus I have installed is Microsoft Security Essentials I turned off and it was still the same: $ cat /usr/bin/gitstatus start_time=`date +%s` git status && echo run time is $(expr `date +%s` - $start_time) s $ gitstatus # On branch test # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: orgoptions.php # modified: update_import_contacts.php # no changes added to commit (use "git add" and/or "git commit -a") run time is 10 s On Wed, Mar 27, 2013 at 12:17 PM, Matthieu Moy wrote: > Jim Kinsman writes: > >> git status takes 30 seconds on Windows 7. > > Any anti-virus installed? They can interfer badly with disk-intensive > tasks ... > > -- > Matthieu Moy > http://www-verimag.imag.fr/~moy/ -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On Wed, Mar 27, 2013 at 11:39:31AM -0500, Jim Kinsman wrote: > git status takes 30 seconds on Windows 7. Here are some stats: > git ls-files | wc -l > 27330 > > git ls-files -o | wc -l > 4 > > $ git diff --name-only | xargs du -chs > 68K update_import_contacts.php > 68K total > > What can I do??? This is so slow it is unbearable. > By the way i've done git gc several times and nothing changed. Can you run these commands under "time" so that we can see that it's definitely the "git ls-files" taking 30 seconds and not something in $PS1? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
Jim Kinsman writes: > git status takes 30 seconds on Windows 7. Any anti-virus installed? They can interfer badly with disk-intensive tasks ... -- Matthieu Moy http://www-verimag.imag.fr/~moy/ -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On Wed, 27 Mar 2013 11:39:31 -0500 Jim Kinsman wrote: > git status takes 30 seconds on Windows 7. Here are some stats: [...] > What can I do??? This is so slow it is unbearable. > By the way i've done git gc several times and nothing changed. You could try some voodoo [1] or experimental caching features [2]. 1. http://groups.google.com/group/msysgit/browse_thread/thread/02e3c0e046f07215 2. http://groups.google.com/group/msysgit/browse_thread/thread/7cbfe3ca452650d1/93ce48e3875f7416 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git status takes 30 seconds on Windows 7. Why?
On 03/27/2013 05:39 PM, Jim Kinsman wrote: > git status takes 30 seconds on Windows 7. Here are some stats: > git ls-files | wc -l > 27330 > > git ls-files -o | wc -l > 4 > > $ git diff --name-only | xargs du -chs > 68K update_import_contacts.php > 68K total > > What can I do??? This is so slow it is unbearable. > By the way i've done git gc several times and nothing changed. I'm guessing it's the disk that's so slow. I accidentally put a git repo on a network-mounted drive once. With 20ms round-trip time to the server, git operations took forever. Could you try it on a disk you know is local? Preferrably a solid state drive. If it's still slow there, we know for sure something's broken inside git. If switching media causes git to become fast, you'll know it's a hardware problem. -- Andreas Ericsson andreas.erics...@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 Considering the successes of the wars on alcohol, poverty, drugs and terror, I think we should give some serious thought to declaring war on peace. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
git status takes 30 seconds on Windows 7. Why?
git status takes 30 seconds on Windows 7. Here are some stats: git ls-files | wc -l 27330 git ls-files -o | wc -l 4 $ git diff --name-only | xargs du -chs 68K update_import_contacts.php 68K total What can I do??? This is so slow it is unbearable. By the way i've done git gc several times and nothing changed. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
