On Dec 13, 2011, at 6:30 PM, Edward Ned Harvey wrote: > > In my situation, I have millions of files which are mostly static. So > simply walking the filesystem to find which files changed is the big time > consumer. We were formerly backing this up via rsync, and it ran 10-12 > hours per night. Then we switched to ZFS and now the incremental takes > typically 7 minutes. Mark's LVM snapshots should be able to achieve similar > results on this particular dataset.
Probably faster still, as there is no file system walking to be done. See also the DRBD. It and Mark's backup method have some similar ideas. I see this as an example of a limitation of traditional UFS-derived file systems and why you may want to consider a modern, high-performance file system rather than ext2/3/4 with Mark's backup system. The bottleneck is the stat() function. It's slow on traditional UFS-style file systems. We've known it since the heyday of Usenet. Replacing a traditional directory structure with a database offers massive performance improvements as you've seen with your conversion to ZFS. These improvements carry over to almost all file operations, not just backups. --Rich P. _______________________________________________ Discuss mailing list [email protected] http://lists.blu.org/mailman/listinfo/discuss
