On 10/09/13 03:38, ame...@gmail.com wrote: > It's great you brought up bbcp. I didn't factor this into my initial > email, but if we could further split up large files into N chunks and > transport them concurrently, that would provide massive benefits as > well. > > It's clear there are multiple areas for improvement that would let us > do away with having to use other tools for various scenarios. If we > could put some planning/development time in even the > easiest-to-implement cases, it would be highly-beneficial to a large > number of users. >
Indeed. I actually hacked together a shell script wrapper around rsync which would take the directory you were wanting to copy, tar it up and then split the resultant large file into 'N' chunks - then transfer the chunks in parallel. Using tricks with "post xfer" at the other end, I would automagically unpack back to the original directory structure. Major improvement in throughput - but only good for "new" file transfers - not for replicating deltas/etc. And before you ask, no you can't have it :-) It's part of a rather complex file replication environment we have and isn't a module that could easily be detached for redistribution To finish with, even though we can make rsync run much "faster" over WANs, we don't use the feature I just mentioned! Just because you can saturate your WAN pipe doesn't mean you should - typically there's other traffic that also needs to co-exist and such a hammering would have consequences - things have to be thought through. It would be nice to have the ability, but that doesn't mean everyone would use it all the time -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html