Re: EVFILT_VNODE doesn't scale to large directory trees?
On Mon, 25 Oct 2010, Kenton Varda wrote: |On Mon, Oct 25, 2010 at 1:54 AM, Igor V. Ruzanov wrote: | |> I thought so too but was not sure about his problem. With |> kqueue()/kevent() we can monitor every open file descriptor. So how many |> files are laid down in the directory tree, 100, 1000, 15000? In every |> such way its possible to write or find already written daemon that |> monitors every file in selected folder doing several jobs in separate |> threads. There is might be little problem with receiving of *lots* events, |> so we could create several pipes (for example) for setting up of |> communication between deamon and system. |> Note that we must remember to set proper meaning of kern.maxfiles sysctl |> variable. |> | |I worry that simply increasing the FD limits to meet my needs would have |some negative effects, otherwise the limits would be much higher in the |first place. How much kernel memory does each open FD consume? Probably |most of that is wasted space, since I'm opening these FDs for no other |purpose than to pass them to kqueue -- I never read or write them. But it |sounds like you're saying that there is no alternative (other than polling, |which would obviously be a lot worse), so I guess I'll live with it. | |Well, one other idea: Is there a way to simply monitor *all* I/O by all |processes owned by the current user? I could then filter the events down to |the directory I'm interested in. Not the ideal solution, but it would scale |to a source tree of infinite size (since the machine can only be accessing a |finite number of those files at once). It seems likely that this has been |implemented somewhere due to the obvious system monitoring applications, but |I'm not quite sure where to start looking. | As a weak solution - is to install `lsof' on your system and do grepping of the command output periodically to see what the files are opened by process(es) with a certain their owner. But this method not very good since we must collect any system event in real time way that is implemented in kqueue. Another way is to poll events with FAM mechanism that comes from SGI IRIX. Try to research if any solutions to use FAM under FreeBSD and what the methods of events monitoring could be used in FAM together with your project. +---+ ! CANMOS ISP Network! +---+ ! Best regards ! ! Igor V. Ruzanov, network operational staff! ! e-Mail: ig...@canmos.ru ! +---+ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
Ivan Voras wrote: > Short answer: no. > > Long answer: There should be. There were past discussions on writing > such a facility to e.g. receive events for all files on per-mountpoint > basis (which you could filter...), but we're not there yet. Thanks! That answers my question. I'll find some sort of hack for now. (Sorry, I didn't see this answer at first since I wasn't CC'd.) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
On Mon, Oct 25, 2010 at 1:54 AM, Igor V. Ruzanov wrote: > I thought so too but was not sure about his problem. With > kqueue()/kevent() we can monitor every open file descriptor. So how many > files are laid down in the directory tree, 100, 1000, 15000? In every > such way its possible to write or find already written daemon that > monitors every file in selected folder doing several jobs in separate > threads. There is might be little problem with receiving of *lots* events, > so we could create several pipes (for example) for setting up of > communication between deamon and system. > Note that we must remember to set proper meaning of kern.maxfiles sysctl > variable. > I worry that simply increasing the FD limits to meet my needs would have some negative effects, otherwise the limits would be much higher in the first place. How much kernel memory does each open FD consume? Probably most of that is wasted space, since I'm opening these FDs for no other purpose than to pass them to kqueue -- I never read or write them. But it sounds like you're saying that there is no alternative (other than polling, which would obviously be a lot worse), so I guess I'll live with it. Well, one other idea: Is there a way to simply monitor *all* I/O by all processes owned by the current user? I could then filter the events down to the directory I'm interested in. Not the ideal solution, but it would scale to a source tree of infinite size (since the machine can only be accessing a finite number of those files at once). It seems likely that this has been implemented somewhere due to the obvious system monitoring applications, but I'm not quite sure where to start looking. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
On 10/25/10 03:05, Kenton Varda wrote: > Hi all, > > I am trying to write some code which monitors a possibly-large directory > tree for changes. Specifically, it's a build system, and I want it to > automatically start rebuilding whenever I modify a source file. > > So far the approach I've taken is to use EVFILT_VNODE to watch every file > and directory in the tree. This seems to work OK so far, but it worries me > that I have to open() every single file. When I ran the same code on > Darwin, it promptly hit the open file descriptor limit, and I'm worried that > FreeBSD will do the same on larger code trees. > > Is there any better way to accomplish this? Hate to say it, but Linux's > inotify() seems more scalable here. From what I can tell from the docs, it > doesn't require opening the watched files and it will even watch all files > in a directory with one call. Short answer: no. Long answer: There should be. There were past discussions on writing such a facility to e.g. receive events for all files on per-mountpoint basis (which you could filter...), but we're not there yet. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
On Mon, 25 Oct 2010, Erik Trulsson wrote: |On Mon, Oct 25, 2010 at 10:48:58AM +0400, Igor V. Ruzanov wrote: |> On Sun, 24 Oct 2010, Kenton Varda wrote: |> |> |That doesn't answer my question. I'm not even using make. I could write a |> |few thousand words describing exactly what I'm trying to do and why it does, |> |in fact, make sense, but it's really beside the point. I just want to know |> |if there is any scalable way to monitor a very large directory tree for |> |changes. Is there? |> | |> Dig `kqueue' - its the native FreeBSD's events polling/notification |> mechanism. | |Since the OP mentioned using EVFILT_VNODE I would assume he is already |using kqueue but is not satisfied with it. | I thought so too but was not sure about his problem. With kqueue()/kevent() we can monitor every open file descriptor. So how many files are laid down in the directory tree, 100, 1000, 15000? In every such way its possible to write or find already written daemon that monitors every file in selected folder doing several jobs in separate threads. There is might be little problem with receiving of *lots* events, so we could create several pipes (for example) for setting up of communication between deamon and system. Note that we must remember to set proper meaning of kern.maxfiles sysctl variable. +---+ ! CANMOS ISP Network! +---+ ! Best regards ! ! Igor V. Ruzanov, network operational staff! ! e-Mail: ig...@canmos.ru ! +---+ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
On Mon, Oct 25, 2010 at 10:48:58AM +0400, Igor V. Ruzanov wrote: > On Sun, 24 Oct 2010, Kenton Varda wrote: > > |That doesn't answer my question. I'm not even using make. I could write a > |few thousand words describing exactly what I'm trying to do and why it does, > |in fact, make sense, but it's really beside the point. I just want to know > |if there is any scalable way to monitor a very large directory tree for > |changes. Is there? > | > Dig `kqueue' - its the native FreeBSD's events polling/notification > mechanism. Since the OP mentioned using EVFILT_VNODE I would assume he is already using kqueue but is not satisfied with it. -- Erik Trulsson ertr1...@student.uu.se ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
On Sun, 24 Oct 2010, Kenton Varda wrote: |That doesn't answer my question. I'm not even using make. I could write a |few thousand words describing exactly what I'm trying to do and why it does, |in fact, make sense, but it's really beside the point. I just want to know |if there is any scalable way to monitor a very large directory tree for |changes. Is there? | Dig `kqueue' - its the native FreeBSD's events polling/notification mechanism. +---+ ! CANMOS ISP Network! +---+ ! Best regards ! ! Igor V. Ruzanov, network operational staff! ! e-Mail: ig...@canmos.ru ! +---+ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
That doesn't answer my question. I'm not even using make. I could write a few thousand words describing exactly what I'm trying to do and why it does, in fact, make sense, but it's really beside the point. I just want to know if there is any scalable way to monitor a very large directory tree for changes. Is there? On Sun, Oct 24, 2010 at 9:46 PM, Robert Bonomi wrote: > > From owner-freebsd-questi...@freebsd.org Sun Oct 24 22:17:42 2010 > > Date: Sun, 24 Oct 2010 18:05:34 -0700 > > From: Kenton Varda > > To: freebsd-questions@freebsd.org > > Subject: EVFILT_VNODE doesn't scale to large directory trees? > > > > Hi all, > > > > I am trying to write some code which monitors a possibly-large directory > > tree for changes. Specifically, it's a build system, and I want it to > > automatically start rebuilding whenever I modify a source file. > > > > So far the approach I've taken is to use EVFILT_VNODE to watch every file > > and directory in the tree. This seems to work OK so far, but it worries > me > > that I have to open() every single file. When I ran the same code on > > Darwin, it promptly hit the open file descriptor limit, and I'm worried > that > > FreeBSD will do the same on larger code trees. > > > > Is there any better way to accomplish this? Hate to say it, but Linux's > > inotify() seems more scalable here. From what I can tell from the docs, > it > > doesn't require opening the watched files and it will even watch all > files > > in a directory with one call. > > > You're re-inventing the wheel. > > 1) Set up a 'makefile' for the entire tree. > > 2) set up a daemon task that > a) cd's to the root direcory of the build tree, > b) executes a loop, consisting of > 1) the 'make all' command, > 2) a reasonably short 'sleep' > > > If 'efficiency' is a concern, then establish a procedure for checking-out/ > checking-in files from the repository. When a file is checked in, check > for (a) it being a new file, *OR* (b) having changes from the prior > version. > If either condition is true, fire off 'make' to do the necessary re-build. > > NOTE: 'cvs' has the above feature as a built-in option. simply specify > 'make' as a program to be run when you do a 'cvs commit' to store changes > back into the repository. > > Did I say soemthing about re-inventing the wheel?? > > > > ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: EVFILT_VNODE doesn't scale to large directory trees?
> From owner-freebsd-questi...@freebsd.org Sun Oct 24 22:17:42 2010 > Date: Sun, 24 Oct 2010 18:05:34 -0700 > From: Kenton Varda > To: freebsd-questions@freebsd.org > Subject: EVFILT_VNODE doesn't scale to large directory trees? > > Hi all, > > I am trying to write some code which monitors a possibly-large directory > tree for changes. Specifically, it's a build system, and I want it to > automatically start rebuilding whenever I modify a source file. > > So far the approach I've taken is to use EVFILT_VNODE to watch every file > and directory in the tree. This seems to work OK so far, but it worries me > that I have to open() every single file. When I ran the same code on > Darwin, it promptly hit the open file descriptor limit, and I'm worried that > FreeBSD will do the same on larger code trees. > > Is there any better way to accomplish this? Hate to say it, but Linux's > inotify() seems more scalable here. From what I can tell from the docs, it > doesn't require opening the watched files and it will even watch all files > in a directory with one call. You're re-inventing the wheel. 1) Set up a 'makefile' for the entire tree. 2) set up a daemon task that a) cd's to the root direcory of the build tree, b) executes a loop, consisting of 1) the 'make all' command, 2) a reasonably short 'sleep' If 'efficiency' is a concern, then establish a procedure for checking-out/ checking-in files from the repository. When a file is checked in, check for (a) it being a new file, *OR* (b) having changes from the prior version. If either condition is true, fire off 'make' to do the necessary re-build. NOTE: 'cvs' has the above feature as a built-in option. simply specify 'make' as a program to be run when you do a 'cvs commit' to store changes back into the repository. Did I say soemthing about re-inventing the wheel?? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
EVFILT_VNODE doesn't scale to large directory trees?
Hi all, I am trying to write some code which monitors a possibly-large directory tree for changes. Specifically, it's a build system, and I want it to automatically start rebuilding whenever I modify a source file. So far the approach I've taken is to use EVFILT_VNODE to watch every file and directory in the tree. This seems to work OK so far, but it worries me that I have to open() every single file. When I ran the same code on Darwin, it promptly hit the open file descriptor limit, and I'm worried that FreeBSD will do the same on larger code trees. Is there any better way to accomplish this? Hate to say it, but Linux's inotify() seems more scalable here. From what I can tell from the docs, it doesn't require opening the watched files and it will even watch all files in a directory with one call. -Kenton ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
