Re: EVFILT_VNODE doesn't scale to large directory trees?
That doesn't answer my question. I'm not even using make. I could write a few thousand words describing exactly what I'm trying to do and why it does, in fact, make sense, but it's really beside the point. I just want to know if there is any scalable way to monitor a very large directory tree for changes. Is there? On Sun, Oct 24, 2010 at 9:46 PM, Robert Bonomi bon...@mail.r-bonomi.comwrote: From owner-freebsd-questi...@freebsd.org Sun Oct 24 22:17:42 2010 Date: Sun, 24 Oct 2010 18:05:34 -0700 From: Kenton Varda tempo...@gmail.com To: freebsd-questions@freebsd.org Subject: EVFILT_VNODE doesn't scale to large directory trees? Hi all, I am trying to write some code which monitors a possibly-large directory tree for changes. Specifically, it's a build system, and I want it to automatically start rebuilding whenever I modify a source file. So far the approach I've taken is to use EVFILT_VNODE to watch every file and directory in the tree. This seems to work OK so far, but it worries me that I have to open() every single file. When I ran the same code on Darwin, it promptly hit the open file descriptor limit, and I'm worried that FreeBSD will do the same on larger code trees. Is there any better way to accomplish this? Hate to say it, but Linux's inotify() seems more scalable here. From what I can tell from the docs, it doesn't require opening the watched files and it will even watch all files in a directory with one call. You're re-inventing the wheel. 1) Set up a 'makefile' for the entire tree. 2) set up a daemon task that a) cd's to the root direcory of the build tree, b) executes a loop, consisting of 1) the 'make all' command, 2) a reasonably short 'sleep' If 'efficiency' is a concern, then establish a procedure for checking-out/ checking-in files from the repository. When a file is checked in, check for (a) it being a new file, *OR* (b) having changes from the prior version. If either condition is true, fire off 'make' to do the necessary re-build. NOTE: 'cvs' has the above feature as a built-in option. simply specify 'make' as a program to be run when you do a 'cvs commit' to store changes back into the repository. Did I say soemthing about re-inventing the wheel?? grin ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: EVFILT_VNODE doesn't scale to large directory trees?
On Mon, Oct 25, 2010 at 1:54 AM, Igor V. Ruzanov ig...@canmos.ru wrote: I thought so too but was not sure about his problem. With kqueue()/kevent() we can monitor every open file descriptor. So how many files are laid down in the directory tree, 100, 1000, 15000? In every such way its possible to write or find already written daemon that monitors every file in selected folder doing several jobs in separate threads. There is might be little problem with receiving of *lots* events, so we could create several pipes (for example) for setting up of communication between deamon and system. Note that we must remember to set proper meaning of kern.maxfiles sysctl variable. I worry that simply increasing the FD limits to meet my needs would have some negative effects, otherwise the limits would be much higher in the first place. How much kernel memory does each open FD consume? Probably most of that is wasted space, since I'm opening these FDs for no other purpose than to pass them to kqueue -- I never read or write them. But it sounds like you're saying that there is no alternative (other than polling, which would obviously be a lot worse), so I guess I'll live with it. Well, one other idea: Is there a way to simply monitor *all* I/O by all processes owned by the current user? I could then filter the events down to the directory I'm interested in. Not the ideal solution, but it would scale to a source tree of infinite size (since the machine can only be accessing a finite number of those files at once). It seems likely that this has been implemented somewhere due to the obvious system monitoring applications, but I'm not quite sure where to start looking. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: EVFILT_VNODE doesn't scale to large directory trees?
Ivan Voras wrote: Short answer: no. Long answer: There should be. There were past discussions on writing such a facility to e.g. receive events for all files on per-mountpoint basis (which you could filter...), but we're not there yet. Thanks! That answers my question. I'll find some sort of hack for now. (Sorry, I didn't see this answer at first since I wasn't CC'd.) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
EVFILT_VNODE doesn't scale to large directory trees?
Hi all, I am trying to write some code which monitors a possibly-large directory tree for changes. Specifically, it's a build system, and I want it to automatically start rebuilding whenever I modify a source file. So far the approach I've taken is to use EVFILT_VNODE to watch every file and directory in the tree. This seems to work OK so far, but it worries me that I have to open() every single file. When I ran the same code on Darwin, it promptly hit the open file descriptor limit, and I'm worried that FreeBSD will do the same on larger code trees. Is there any better way to accomplish this? Hate to say it, but Linux's inotify() seems more scalable here. From what I can tell from the docs, it doesn't require opening the watched files and it will even watch all files in a directory with one call. -Kenton ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
