+------------------------------------------------------------------------------ | On 2009-10-03 18:50:58, Jeff Haferman wrote: | | I did an rsync of this directory structure to another filesystem | [lustre-based, FWIW] and it took about 24 hours to complete. We have | done rsyncs on other directories that are much larger in terms of | file-sizes, but have thousands of files rather than tens, hundreds, and | millions of files. | | Is there someway to speed up "simple" things like determining the | contents of these directories?
Use zfs snapshots. See zfs(1M) and review the incremental send syntax. | And why does an rsync take so much | longer on these directories when directories that contain hundreds of rsync has to build its file list (stat is slow) on both sides of the sync, then compare them, and then send each one. (d)truss it sometime. It's a lot of syscalls. The initial zfs send may be slow, depending on the total size, but the incrementals will be pretty fast. Certainly faster than rsync (by orders of magnitude), as ZFS already knows which blocks it seends to send, and is only sending blocks. If the target host doesn't support ZFS in some form, you could dump the snapshots to disk and use those for backups. Or restructure your storage hierarchy (which uh, you might want to do anyway). -- bda cyberpunk is dead. long live cyberpunk. _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
