Re: Watchman support for git

2014-05-19 Thread Duy Nguyen
On Fri, May 16, 2014 at 2:42 AM, David Turner dtur...@twopensource.com wrote: I assume you won't change your mind about this. Which is fine to me. I will still try out my approach with your libwatchman though. Just curious about its performance and complexity, compared to your approach. I am

Re: Watchman support for git

2014-05-15 Thread David Turner
On Wed, 2014-05-14 at 17:36 +0700, Duy Nguyen wrote: With that in mind, I think you don't need to keep a fs cache on disk at all. All you need to store (in the index) is the clock value from watchman. After we parse the index, we perform a since query to get path names (and perhaps

Re: Watchman support for git

2014-05-14 Thread Duy Nguyen
On Wed, May 14, 2014 at 6:44 AM, David Turner dtur...@twopensource.com wrote: I'm not so happy that git now needs to link to libjansson.so and libwatchman.so. I would load libwatchman.so at runtime only when core.usewatchman is on, but this is more of personal taste. I assume you mean

Re: Watchman support for git

2014-05-14 Thread Duy Nguyen
One more thing I think we can solve with watchman is the racy timestamp issue (see 29e4d36 and 407c8eb). Basically if a file is updated within a second (and the system only supports timestamp granuarity down to a second) then git can't know if a file is changed by comparing stat data, so it

Re: Watchman support for git

2014-05-13 Thread David Turner
On Mon, 2014-05-12 at 17:45 +0700, Duy Nguyen wrote: This is your quote from above, moved down a bit: update_fs_cache should only have to update based on what it has learned from watchman. So if no .gitignore has been changed, it should not have to do very much work. I could take the

Re: Watchman support for git

2014-05-13 Thread Duy Nguyen
On Wed, May 14, 2014 at 5:38 AM, David Turner dtur...@twopensource.com wrote: On Mon, 2014-05-12 at 17:45 +0700, Duy Nguyen wrote: This is your quote from above, moved down a bit: update_fs_cache should only have to update based on what it has learned from watchman. So if no .gitignore has

Re: Watchman support for git

2014-05-13 Thread David Turner
On Wed, 2014-05-14 at 05:54 +0700, Duy Nguyen wrote: On Wed, May 14, 2014 at 5:38 AM, David Turner dtur...@twopensource.com wrote: On Mon, 2014-05-12 at 17:45 +0700, Duy Nguyen wrote: This is your quote from above, moved down a bit: update_fs_cache should only have to update based on

Re: Watchman support for git

2014-05-13 Thread David Turner
On Sat, 2014-05-10 at 15:16 +0700, Duy Nguyen wrote: On Sat, May 3, 2014 at 6:14 AM, dtur...@twopensource.com wrote: The most sigificant patch uses Facebook's watchman daemon[1] to monitor the repository work tree for changes. This makes allows git status to avoid traversing the entire

Re: Watchman support for git

2014-05-12 Thread Duy Nguyen
On Mon, May 12, 2014 at 5:56 AM, David Turner dtur...@twopensource.com wrote: So without watchman I got 299.871ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) 0) go 498.205ms cmd_status:1300 refresh_index(the_index, REFRESH_QUIE 796.050ms wt_status_collect:622

Re: Watchman support for git

2014-05-11 Thread David Turner
On Sun, 2014-05-11 at 07:21 +0700, Duy Nguyen wrote: On Sun, May 11, 2014 at 1:38 AM, David Turner dtur...@twopensource.com wrote: I got warning: Watchman watch error: Got bad JSON from watchman get-sockname: '[' or '{' expected near end of file. Any ideas what I did wrong? I'm using

Re: Watchman support for git

2014-05-10 Thread Duy Nguyen
On Sat, May 3, 2014 at 6:14 AM, dtur...@twopensource.com wrote: The most sigificant patch uses Facebook's watchman daemon[1] to monitor the repository work tree for changes. This makes allows git status to avoid traversing the entire work tree to find changes. Some comments on this series. I

Re: Watchman support for git

2014-05-10 Thread David Turner
On Sat, 2014-05-10 at 12:26 +0700, Duy Nguyen wrote: On Sat, May 3, 2014 at 6:14 AM, dtur...@twopensource.com wrote: The most sigificant patch uses Facebook's watchman daemon[1] to monitor the repository work tree for changes. This makes allows git status to avoid traversing the entire

Re: Watchman support for git

2014-05-10 Thread Duy Nguyen
On Sun, May 11, 2014 at 1:38 AM, David Turner dtur...@twopensource.com wrote: I got warning: Watchman watch error: Got bad JSON from watchman get-sockname: '[' or '{' expected near end of file. Any ideas what I did wrong? I'm using watchman.git and libwatchman.git. check-0.9.11 and jansson-2.4

Re: Watchman support for git

2014-05-09 Thread David Lang
On Thu, 8 May 2014, Sebastian Schuberth wrote: On 03.05.2014 05:40, Felipe Contreras wrote: That's very interesting. Do you get similar improvements when doing something similar in Merurial (watchman vs . no watchman). I have not tried it. My understanding is that this is why Facebook

Re: Watchman support for git

2014-05-09 Thread David Turner
On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote: On Thu, 8 May 2014, Sebastian Schuberth wrote: On 03.05.2014 05:40, Felipe Contreras wrote: That's very interesting. Do you get similar improvements when doing something similar in Merurial (watchman vs . no watchman). I have not

Re: Watchman support for git

2014-05-09 Thread David Lang
On Fri, 9 May 2014, David Turner wrote: On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote: On Thu, 8 May 2014, Sebastian Schuberth wrote: On 03.05.2014 05:40, Felipe Contreras wrote: That's very interesting. Do you get similar improvements when doing something similar in Merurial

Re: Watchman support for git

2014-05-09 Thread David Turner
On Fri, 2014-05-09 at 11:08 -0700, David Lang wrote: On Fri, 9 May 2014, David Turner wrote: On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote: On Thu, 8 May 2014, Sebastian Schuberth wrote: On 03.05.2014 05:40, Felipe Contreras wrote: That's very interesting. Do you get similar

Re: Watchman support for git

2014-05-09 Thread David Lang
On Fri, 9 May 2014, David Turner wrote: On Fri, 2014-05-09 at 11:08 -0700, David Lang wrote: On Fri, 9 May 2014, David Turner wrote: On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote: On Thu, 8 May 2014, Sebastian Schuberth wrote: On 03.05.2014 05:40, Felipe Contreras wrote: That's

Re: Watchman support for git

2014-05-09 Thread David Turner
On Fri, 2014-05-09 at 11:27 -0700, David Lang wrote: That's not my understanding from Durham Goode's talk in January. Yes, operations involving history go to the server. But the client also maintains a copy of the working tree, and it is for this that watchman is used. Otherwise, why

Re: Watchman support for git

2014-05-09 Thread Duy Nguyen
On Sat, May 3, 2014 at 6:14 AM, dtur...@twopensource.com wrote: The most sigificant patch uses Facebook's watchman daemon[1] to monitor the repository work tree for changes. This makes allows git status to avoid traversing the entire work tree to find changes. I got warning: Watchman watch

Re: Watchman support for git

2014-05-08 Thread Sebastian Schuberth
On 03.05.2014 05:40, Felipe Contreras wrote: That's very interesting. Do you get similar improvements when doing something similar in Merurial (watchman vs . no watchman). I have not tried it. My understanding is that this is why Facebook wrote Watchman and added support for it to

RE: Watchman support for git

2014-05-05 Thread David Turner
On Fri, 2014-05-02 at 22:40 -0500, Felipe Contreras wrote: David Turner wrote: On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote: dturner@ wrote: Test repository 1: Linux Linux is about 45k files in 3k directories. The average length of a filename is about 32 bytes.

RE: Watchman support for git

2014-05-05 Thread Felipe Contreras
David Turner wrote: On Fri, 2014-05-02 at 22:40 -0500, Felipe Contreras wrote: David Turner wrote: On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote: dturner@ wrote: Test repository 1: Linux Linux is about 45k files in 3k directories. The average length of a

Re: Watchman support for git

2014-05-05 Thread Duy Nguyen
On Sat, May 3, 2014 at 7:52 AM, Duy Nguyen pclo...@gmail.com wrote: wt_status_collect_changes_index() depends on how damaged cache-tree is (in this case, totally scraped). watchman does not help this either. We need to try to heal cache-tree as much as possible to reduce the number. On the

Re: Watchman support for git

2014-05-05 Thread Duy Nguyen
On Tue, May 6, 2014 at 7:26 AM, Duy Nguyen pclo...@gmail.com wrote: This discard_index() code was added recently in e28f764 (unpack-trees: plug a memory leak - 2013-08-13). As a workaround, perhaps we only do so when the sequencer is running. Hmm.. as o-result does not carry cache-tree anyway,

Re: Watchman support for git

2014-05-05 Thread David Turner
On Sun, 2014-05-04 at 07:15 +0700, Duy Nguyen wrote: I would like to merge the feature into master. It works well for me, and some of my colleagues who have tried it out. Have you tried to turn watchman on by default, then run it with git test suite? That usually helps. I have. The

Re: Watchman support for git

2014-05-03 Thread Duy Nguyen
On Sat, May 3, 2014 at 11:39 AM, David Turner dtur...@twopensource.com wrote: Index v4 and split index (and the following read-cache daemon, hopefully) Looking at some of the archives for read-cache daemon, it seems to be somewhat similar to watchman, right? But I only saw inotify code; what

Re: Watchman support for git

2014-05-03 Thread David Turner
On Sat, 2014-05-03 at 15:49 +0700, Duy Nguyen wrote: On Sat, May 3, 2014 at 11:39 AM, David Turner dtur...@twopensource.com wrote: Index v4 and split index (and the following read-cache daemon, hopefully) Looking at some of the archives for read-cache daemon, it seems to be somewhat

Re: Watchman support for git

2014-05-03 Thread Duy Nguyen
On Sun, May 4, 2014 at 3:49 AM, David Turner dtur...@twopensource.com wrote: On Sat, 2014-05-03 at 15:49 +0700, Duy Nguyen wrote: On Sat, May 3, 2014 at 11:39 AM, David Turner dtur...@twopensource.com wrote: Index v4 and split index (and the following read-cache daemon, hopefully)

Re: Watchman support for git

2014-05-02 Thread Duy Nguyen
On Sat, May 3, 2014 at 6:14 AM, dtur...@twopensource.com wrote: The index format change might be less important with the split index; I haven't investigated that since at the time I wrote these patches, it didn't exist. This is the worst case scenario of git status on webkit.git (182k files,

RE: Watchman support for git

2014-05-02 Thread David Turner
On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote: dturner@ wrote: Test repository 1: Linux Linux is about 45k files in 3k directories. The average length of a filename is about 32 bytes. Git status timing: no watchman: 125ms watchman: 90ms That's very interesting.

RE: Watchman support for git

2014-05-02 Thread Felipe Contreras
David Turner wrote: On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote: dturner@ wrote: Test repository 1: Linux Linux is about 45k files in 3k directories. The average length of a filename is about 32 bytes. Git status timing: no watchman: 125ms watchman: 90ms

Re: Watchman support for git

2014-05-02 Thread David Turner
On Sat, 2014-05-03 at 07:52 +0700, Duy Nguyen wrote: On Sat, May 3, 2014 at 6:14 AM, dtur...@twopensource.com wrote: The index format change might be less important with the split index; I haven't investigated that since at the time I wrote these patches, it didn't exist. This is the