On Tue, Aug 11, 2015 at 02:44:10PM +0000, Emmanuel Dreyfus wrote: > Do we have a smart way to monitor a whole filesystem hierarchy for > changes?
Replying to myself: the short answer is no. kqueue will not scale to thousands of nodes because it needs to open a filedescriptor for each. lsyncd has been mentionned in the thread. On NetBSD, it uses kqueue and therefore suffers the sclaing limitation. An ongoing project to fix kqueue was mentionned: https://wiki.netbsd.org/projects/project/kqueue-dirnotify/ That situation begs for an interm solution. Since my problem is to mirror a file hierarchy from an host to another, I crafter an utility called fsssyncd as an attempt to address the problem without lenghlty kernel hacking (that will be necessary, but not this week!): http://ftp.espci.fr/pub/syncffsd/syncffsd-0.1.tgz Basic usage: syncfssd /usr/local/source host:/usr/local/destination >From the man page: > syncfssd uses rsync(1) to replicate a directory tree to another host. > This involves detecting changes, something NetBSD is not well prepared to > handle. The kqueue(2) interface can detect changes, but it needs to use a > file descriptor for every node monitored, and this does not scales very > well for huge directories trees. > > syncfssd attempts to improve the situation until kqueue(2) gets better, > by using FFS superblock's fs_time field, which is updated on every > filesystem change. This lets syncfssd detects a change immediatly if the > filesystem is mounted synchronous, or after a few seconds if mounted with > WAPBL(4) enabled. > > Once a change is detected, the relevant nodes are found by walking source > looking for files that changed since the previous filesystem modifica- > tion. The resulting nodes are deduped (i.e.: we do not retain a node in > a directory if the directory itself changed), and are fed to rsync(1). This logic could be embedded into lsyncd but I am not sure such work makes sense. Improving kqueue seems a better, more general goal. Feedbacks on this version 0.1 are welcome. -- Emmanuel Dreyfus [email protected]
