On Δευτέρα, 3 Δεκέμβριος 2012 6:49:05 μμ, Tim Gustafson wrote:
Lsyncd would be good for such a case.
Have you checked that you're not running out of memory?
I have not seen any errors or warnings to that effect.
You probably want --delete-during instead of --delete.
Will that speed up the file comparison stage, or is that just a good idea?
If you're checking for hard links, rsync has to track links for the entire
session. That can bog things down for zillions of files. If you know you
won't have links outside particular boundaries, you might benefit from
chunking up your rsync invocations to match those boundaries.
You might benefit from doing that regardless, since multiple rsyncs in
parallel will do a better job of saturating your I/O bandwidth in this
particular case.
We're already doing one rsync for each file system. We have about
2,000 file systems that we're rsyncing between these servers. There
are just two file systems in particular that are troublesome. In
fact, the two file systems in question were formerly one file system,
and we split it in two to accomplish exactly what you describe. Maybe
more splitting is in order, but because of the nature of the data,
we're always going to have at least one tree of folders with a zillion
files in it.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
