On Thu, Aug 13, 2015 at 5:51 AM, Emmanuel Dreyfus <[email protected]> wrote: > On Tue, Aug 11, 2015 at 02:44:10PM +0000, Emmanuel Dreyfus wrote: >> Do we have a smart way to monitor a whole filesystem hierarchy for >> changes? > > Replying to myself: the short answer is no. kqueue will not scale to > thousands of nodes because it needs to open a filedescriptor for each. > > lsyncd has been mentionned in the thread. On NetBSD, it uses kqueue > and therefore suffers the sclaing limitation. > > An ongoing project to fix kqueue was mentionned: > https://wiki.netbsd.org/projects/project/kqueue-dirnotify/ > > That situation begs for an interm solution. Since my problem is to mirror > a file hierarchy from an host to another, I crafter an utility called > fsssyncd as an attempt to address the problem without lenghlty kernel > hacking (that will be necessary, but not this week!): > http://ftp.espci.fr/pub/syncffsd/syncffsd-0.1.tgz > > Basic usage: > syncfssd /usr/local/source host:/usr/local/destination > > From the man page: >> syncfssd uses rsync(1) to replicate a directory tree to another host. >> This involves detecting changes, something NetBSD is not well prepared to >> handle. The kqueue(2) interface can detect changes, but it needs to use a >> file descriptor for every node monitored, and this does not scales very >> well for huge directories trees. >> >> syncfssd attempts to improve the situation until kqueue(2) gets better, >> by using FFS superblock's fs_time field, which is updated on every >> filesystem change. This lets syncfssd detects a change immediatly if the >> filesystem is mounted synchronous, or after a few seconds if mounted with >> WAPBL(4) enabled. >> >> Once a change is detected, the relevant nodes are found by walking source >> looking for files that changed since the previous filesystem modifica- >> tion. The resulting nodes are deduped (i.e.: we do not retain a node in >> a directory if the directory itself changed), and are fed to rsync(1). > > This logic could be embedded into lsyncd but I am not sure such work makes > sense. Improving kqueue seems a better, more general goal. > > Feedbacks on this version 0.1 are welcome. > > -- > Emmanuel Dreyfus > [email protected]
Syncing dirs rsync-style is served well by unison http://www.cis.upenn.edu/~bcpierce/unison/
