Re: inotify to minimize stat() calls

2013-04-29 Thread Duy Nguyen
On Tue, Apr 30, 2013 at 1:05 AM, Robert Zeh robert.allan@gmail.com wrote: The call to lstat is only there for testing and should not be in there for the final version. Is there an easy way to only enable it for tests? The usual trick is invent a new GIT_ environment variable. Then check it

Re: inotify to minimize stat() calls

2013-03-10 Thread Ramkumar Ramachandra
Duy Nguyen wrote: On Fri, Mar 8, 2013 at 3:15 PM, Junio C Hamano gits...@pobox.com wrote: The possible options are: + - - 'no' - Show no untracked files + - 'no' - Show no untracked files (this is fastest) There is a trade-off around the use of -uno between safety and

Re: inotify to minimize stat() calls

2013-03-08 Thread Junio C Hamano
Torsten Bögershausen tbo...@web.de writes: Doesn't this make one wonder why a separate bit and implementation is necessary to say I am not interested in untracked files when -uno option is already there? ... I need to admit that I wasn't aware about git status -uno. Not so fast. I did not

Re: inotify to minimize stat() calls

2013-03-08 Thread Torsten Bögershausen
On 08.03.13 09:15, Junio C Hamano wrote: Torsten Bögershausen tbo...@web.de writes: Doesn't this make one wonder why a separate bit and implementation is necessary to say I am not interested in untracked files when -uno option is already there? ... I need to admit that I wasn't aware about

Re: inotify to minimize stat() calls

2013-03-08 Thread Duy Nguyen
On Fri, Mar 8, 2013 at 3:15 PM, Junio C Hamano gits...@pobox.com wrote: The possible options are: + - - 'no' - Show no untracked files + - 'no' - Show no untracked files (this is fastest) There is a trade-off around the use of -uno between safety and performance. The

Re: inotify to minimize stat() calls

2013-03-07 Thread Torsten Bögershausen
On 11.02.13 03:56, Duy Nguyen wrote: On Mon, Feb 11, 2013 at 3:16 AM, Junio C Hamano gits...@pobox.com wrote: The other lstat() experiment was a very interesting one, but this is not yet an interesting experiment to see where in the ignore codepath we are spending times. We know that we can

Re: inotify to minimize stat() calls

2013-03-07 Thread Junio C Hamano
Torsten Bögershausen tbo...@web.de writes: diff --git a/builtin/commit.c b/builtin/commit.c index d6dd3df..6a5ba11 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1158,6 +1158,8 @@ int cmd_status(int argc, const char **argv, const char *prefix) unsigned char sha1[20];

Re: inotify to minimize stat() calls

2013-03-07 Thread Torsten Bögershausen
On 08.03.13 01:04, Junio C Hamano wrote: Torsten Bögershausen tbo...@web.de writes: diff --git a/builtin/commit.c b/builtin/commit.c index d6dd3df..6a5ba11 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1158,6 +1158,8 @@ int cmd_status(int argc, const char **argv, const char

Re: inotify to minimize stat() calls

2013-02-19 Thread Ramkumar Ramachandra
Karsten Blees wrote: Am 11.02.2013 04:53, schrieb Duy Nguyen: On Sun, Feb 10, 2013 at 11:58 PM, Erik Faye-Lund kusmab...@gmail.com wrote: Karsten Blees has done something similar-ish on Windows, and he posted the results here:

Re: inotify to minimize stat() calls

2013-02-19 Thread Ramkumar Ramachandra
Robert Zeh wrote: On Sun, Feb 10, 2013 at 9:21 PM, Duy Nguyen pclo...@gmail.com wrote: On Mon, Feb 11, 2013 at 2:03 AM, Robert Zeh robert.allan@gmail.com wrote: On Sat, Feb 9, 2013 at 1:35 PM, Junio C Hamano gits...@pobox.com wrote: Ramkumar Ramachandra artag...@gmail.com writes: This

Re: inotify to minimize stat() calls

2013-02-19 Thread Drew Northup
On Sun, Feb 10, 2013 at 12:24 AM, Duy Nguyen pclo...@gmail.com wrote: On Sun, Feb 10, 2013 at 12:10 AM, Ramkumar Ramachandra artag...@gmail.com wrote: Finn notes in the commit message that it offers no speedup, because .gitignore files in every directory still have to be read. I think this

Re: inotify to minimize stat() calls

2013-02-19 Thread Duy Nguyen
On Tue, Feb 19, 2013 at 8:16 PM, Drew Northup n1xim.em...@gmail.com wrote: Did your testing turn up anything about the amount of time spent parsing the .gitignore/.gitattributes files? Not the syscall count, but the actual time spent running the parser (which I presume is largely CPU-bound).

Re: inotify to minimize stat() calls

2013-02-14 Thread Magnus Bäck
On Sunday, February 10, 2013 at 08:26 EST, demerphq demer...@gmail.com wrote: Is windows stat really so slow? Well, the problem is that there is no such thing as Windows stat :-) I encountered this perception in windows Perl in the past, and I know that on windows Perl stat *appears*

Re: inotify to minimize stat() calls

2013-02-14 Thread Ævar Arnfjörð Bjarmason
On Fri, Feb 8, 2013 at 10:10 PM, Ramkumar Ramachandra artag...@gmail.com wrote: For large repositories, many simple git commands like `git status` take a while to respond. I understand that this is because of large number of stat() calls to figure out which files were changed. I overheard

Re: inotify to minimize stat() calls

2013-02-14 Thread Junio C Hamano
Ævar Arnfjörð Bjarmason ava...@gmail.com writes: I looked into this a long while ago and remembered that rebase was doing something like a git status for every commit that it made to check the dirtyness. This could be vastly improved by having an unsafe option to git-rebase where it just

Re: inotify to minimize stat() calls

2013-02-13 Thread Duy Nguyen
On Tue, Feb 12, 2013 at 09:48:18PM +0100, Karsten Blees wrote: However, the difference between git status -uall and -uno was always about 1.3 s in all fscache versions, even though opendir/readdir/closedir was served entirely from the cache. I added a bit of performance tracing to find the

Re: inotify to minimize stat() calls

2013-02-13 Thread Jeff King
On Wed, Feb 13, 2013 at 07:15:47PM +0700, Nguyen Thai Ngoc Duy wrote: On Wed, Feb 13, 2013 at 3:48 AM, Karsten Blees karsten.bl...@gmail.com wrote: 2.) 0.135 s is spent in name-hash.c/hash_index_entry_directories, reindexing the same directories over and over again. In the end, the

Re: inotify to minimize stat() calls

2013-02-13 Thread Jeff King
On Wed, Feb 13, 2013 at 01:18:51PM -0500, Jeff King wrote: I think the best way forward is to actually create a separate hash table for the directory lookups. I note that we only care about these entries in directory_exists_in_index_icase, which is really about whether something is there,

Re: inotify to minimize stat() calls

2013-02-13 Thread Karsten Blees
Am 13.02.2013 19:18, schrieb Jeff King: Moreover, looking at it again, I don't think my patch produces the right behavior: we have a single dir_next pointer, even though the same ce_entry may appear under many directory hashes. So the cache_entries that has to dir/foo/ and those that hash to

Re: inotify to minimize stat() calls

2013-02-13 Thread Jeff King
On Wed, Feb 13, 2013 at 09:25:59PM +0100, Karsten Blees wrote: Am 13.02.2013 19:18, schrieb Jeff King: Moreover, looking at it again, I don't think my patch produces the right behavior: we have a single dir_next pointer, even though the same ce_entry may appear under many directory

Re: inotify to minimize stat() calls

2013-02-13 Thread Karsten Blees
Am 13.02.2013 23:55, schrieb Jeff King: On Wed, Feb 13, 2013 at 09:25:59PM +0100, Karsten Blees wrote: Alternatively, we could simply create normal cache_entries for the directories that are linked via ce-next, but have a trailing '/' in their name? Reference counting sounds good to me, at

Re: inotify to minimize stat() calls

2013-02-12 Thread Karsten Blees
Am 11.02.2013 04:53, schrieb Duy Nguyen: On Sun, Feb 10, 2013 at 11:58 PM, Erik Faye-Lund kusmab...@gmail.com wrote: Karsten Blees has done something similar-ish on Windows, and he posted the results here: https://groups.google.com/forum/#!topic/msysgit/fL_jykUmUNE/discussion The new

Re: inotify to minimize stat() calls

2013-02-11 Thread Duy Nguyen
On Mon, Feb 11, 2013 at 9:56 AM, Duy Nguyen pclo...@gmail.com wrote: Yeah, it did not cut out syscall cost, I also cut a lot of user-space processing (plus .gitignore content access). From the timings I posted earlier, unmodified dir.c real0m0.550s0m0.287s user0m0.305s

Re: inotify to minimize stat() calls

2013-02-11 Thread Robert Zeh
On Sun, Feb 10, 2013 at 9:21 PM, Duy Nguyen pclo...@gmail.com wrote: On Mon, Feb 11, 2013 at 2:03 AM, Robert Zeh robert.allan@gmail.com wrote: On Sat, Feb 9, 2013 at 1:35 PM, Junio C Hamano gits...@pobox.com wrote: Ramkumar Ramachandra artag...@gmail.com writes: This is much better than

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Sun, Feb 10, 2013 at 12:24:58PM +0700, Duy Nguyen wrote: On Sun, Feb 10, 2013 at 12:10 AM, Ramkumar Ramachandra artag...@gmail.com wrote: Finn notes in the commit message that it offers no speedup, because .gitignore files in every directory still have to be read. I think this is

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Sun, Feb 10, 2013 at 06:17:32PM +0700, Duy Nguyen wrote: The following patch eliminates untracked search code. As we can see, open+getdents also disappears with this patch: 0.462909 40950 lstat 0.462909 40950 lstat 0.003417 129 brk 0.003417 129 brk 0.000762 53 read 0.000762

Re: inotify to minimize stat() calls

2013-02-10 Thread demerphq
On 10 February 2013 12:17, Duy Nguyen pclo...@gmail.com wrote: Bear in mind though this is Linux, where lstat is fast. On systems with slow lstat, these timings could look very different due to the large number of lstat calls compared to open+getdents. I really like to see similar numbers on

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Sun, Feb 10, 2013 at 8:26 PM, demerphq demer...@gmail.com wrote: On 10 February 2013 12:17, Duy Nguyen pclo...@gmail.com wrote: Bear in mind though this is Linux, where lstat is fast. On systems with slow lstat, these timings could look very different due to the large number of lstat calls

Re: inotify to minimize stat() calls

2013-02-10 Thread Ramkumar Ramachandra
On Sun, Feb 10, 2013 at 4:47 PM, Duy Nguyen pclo...@gmail.com wrote: On Sun, Feb 10, 2013 at 12:24:58PM +0700, Duy Nguyen wrote: On Sun, Feb 10, 2013 at 12:10 AM, Ramkumar Ramachandra artag...@gmail.com wrote: Finn notes in the commit message that it offers no speedup, because .gitignore

Re: inotify to minimize stat() calls

2013-02-10 Thread Erik Faye-Lund
On Sun, Feb 10, 2013 at 12:17 PM, Duy Nguyen pclo...@gmail.com wrote: On Sun, Feb 10, 2013 at 12:24:58PM +0700, Duy Nguyen wrote: On Sun, Feb 10, 2013 at 12:10 AM, Ramkumar Ramachandra artag...@gmail.com wrote: Finn notes in the commit message that it offers no speedup, because .gitignore

Re: inotify to minimize stat() calls

2013-02-10 Thread Robert Zeh
On Sat, Feb 9, 2013 at 1:35 PM, Junio C Hamano gits...@pobox.com wrote: Ramkumar Ramachandra artag...@gmail.com writes: This is much better than Junio's suggestion to study possible implementations on all platforms and designing a generic daemon/ communication channel. That's no weekend

Re: inotify to minimize stat() calls

2013-02-10 Thread Martin Fick
On Sunday, February 10, 2013 12:03:00 pm Robert Zeh wrote: On Sat, Feb 9, 2013 at 1:35 PM, Junio C Hamano gits...@pobox.com wrote: Ramkumar Ramachandra artag...@gmail.com writes: This is much better than Junio's suggestion to study possible implementations on all platforms and designing

Re: inotify to minimize stat() calls

2013-02-10 Thread Junio C Hamano
Duy Nguyen pclo...@gmail.com writes: On Sun, Feb 10, 2013 at 06:17:32PM +0700, Duy Nguyen wrote: The following patch eliminates untracked search code. As we can see, open+getdents also disappears with this patch: 0.462909 40950 lstat 0.462909 40950 lstat 0.003417 129 brk 0.003417

Re: inotify to minimize stat() calls

2013-02-10 Thread Robert Zeh
On Feb 10, 2013, at 1:26 PM, Martin Fick mf...@codeaurora.org wrote: On Sunday, February 10, 2013 12:03:00 pm Robert Zeh wrote: On Sat, Feb 9, 2013 at 1:35 PM, Junio C Hamano gits...@pobox.com wrote: Ramkumar Ramachandra artag...@gmail.com writes: This is much better than Junio's

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Mon, Feb 11, 2013 at 3:16 AM, Junio C Hamano gits...@pobox.com wrote: The other lstat() experiment was a very interesting one, but this is not yet an interesting experiment to see where in the ignore codepath we are spending times. We know that we can tell wt_status_collect_untracked() not

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Sun, Feb 10, 2013 at 11:45 PM, Ramkumar Ramachandra artag...@gmail.com wrote: So you're skipping the rest of refresh_cache_ent(), which contains our lstat() and returning immediately. Instead of marking paths with the assume unchanged bit, as core.ignoreStat does, you're directly attacking

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Mon, Feb 11, 2013 at 2:03 AM, Robert Zeh robert.allan@gmail.com wrote: On Sat, Feb 9, 2013 at 1:35 PM, Junio C Hamano gits...@pobox.com wrote: Ramkumar Ramachandra artag...@gmail.com writes: This is much better than Junio's suggestion to study possible implementations on all platforms

Re: inotify to minimize stat() calls

2013-02-10 Thread Duy Nguyen
On Sun, Feb 10, 2013 at 11:58 PM, Erik Faye-Lund kusmab...@gmail.com wrote: Karsten Blees has done something similar-ish on Windows, and he posted the results here: https://groups.google.com/forum/#!topic/msysgit/fL_jykUmUNE/discussion I also seem to remember he doing a ReadDirectoryChangesW

Re: inotify to minimize stat() calls

2013-02-09 Thread Duy Nguyen
On Sat, Feb 9, 2013 at 7:53 PM, Ramkumar Ramachandra artag...@gmail.com wrote: Ramkumar Ramachandra wrote: What about getting systemd to watch everything for us? systemd is the perfect candidate! How about this as a start? I did not really check what it does, but it does not look complicate

Re: inotify to minimize stat() calls

2013-02-09 Thread Ramkumar Ramachandra
Duy Nguyen wrote: How about this as a start? I did not really check what it does, but it does not look complicate enough to pull systemd in. http://article.gmane.org/gmane.comp.version-control.git/151934 Clever hack. I didn't know that there was a switch called core.ignoreStat which will

Re: inotify to minimize stat() calls

2013-02-09 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes: This is much better than Junio's suggestion to study possible implementations on all platforms and designing a generic daemon/ communication channel. That's no weekend project. It appears that you misunderstood what I wrote. That was not here

Re: inotify to minimize stat() calls

2013-02-09 Thread Duy Nguyen
On Sun, Feb 10, 2013 at 12:10 AM, Ramkumar Ramachandra artag...@gmail.com wrote: Finn notes in the commit message that it offers no speedup, because .gitignore files in every directory still have to be read. I think this is silly: we really should be caching .gitignore, and touching it only

Re: inotify to minimize stat() calls

2013-02-08 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes: ... Will Git ever consider using inotify on Linux? What is the downside? I think this has come up from time to time, but my understanding is that nobody thought things through to find a good layer in the codebase to interface to an external

Re: inotify to minimize stat() calls

2013-02-08 Thread Junio C Hamano
Junio C Hamano gits...@pobox.com writes: Ramkumar Ramachandra artag...@gmail.com writes: ... Will Git ever consider using inotify on Linux? What is the downside? I think this has come up from time to time, but my understanding is that nobody thought things through to find a good layer in

Re: inotify to minimize stat() calls

2013-02-08 Thread Duy Nguyen
On Sat, Feb 9, 2013 at 5:45 AM, Junio C Hamano gits...@pobox.com wrote: Junio C Hamano gits...@pobox.com writes: Ramkumar Ramachandra artag...@gmail.com writes: ... Will Git ever consider using inotify on Linux? What is the downside? I think this has come up from time to time, but my

Re: inotify to minimize stat() calls

2013-02-08 Thread Junio C Hamano
Duy Nguyen pclo...@gmail.com writes: Can we replace open a socket to our daemon with open a special file in .git to get stat data written by our daemon? TCP/IP socket means system-wide daemon, not attractive. UNIX socket is not available on Windows (although there may be named pipe, I don't

Re: inotify to minimize stat() calls

2013-02-08 Thread Junio C Hamano
Junio C Hamano gits...@pobox.com writes: I checked read-cache.c and preload-index.c code. To get the discussion rolling, I think something like the outline below may be a good starting point and a feasible weekend hack for somebody competent: * At the beginning of preload_index(), instead

Re: inotify to minimize stat() calls

2013-02-08 Thread Robert Zeh
The delay for commands like git status is much worse on Windows than Linux; for my workflow I would be happy with a Windows only implementation. From the description so far, I have some question: how does the daemon get started and stopped? Is there one per repository --- this seems to be