A block level tool won't do a differential back up like rsync does will it?
On Tue, Sep 13, 2011 at 11:04, Kevin Korb <[email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > The rsync is doing a whole lot more than transferring 600MB of data. It > is transferring 600MB of differences. It is going through 27.4 million > files on one end and possibly that many on the other. That is a 402MB > listing of files plus as many as 54.8 million calls to stat(). > > Rsync is not about minimizing transfer time. It is about minimizing > network IO at the expense of CPU and disk IO. Rsync reduced your > network IO from 185.39GB to 600MB only 200 of which was actually > differences in your data. > > IOW rsync did its job. > > The unfortunate truth is that no file based tool will perform anywhere > near wire speeds when it has millions of files to deal with. If you > want something wire speed then you should look for a block level tool. > > On 09/12/11 21:54, David Somers-Harris wrote: > > No, I don't have checksum based checking enabled. > > > > Sorry for neglecting to mention the options I'm using, here they are: > > > > *rsync -vah --exclude=".*" --delete --stats* > > > > Would anything in there cause it to be unnecessarily slow? It's taking 3 > > hours but shouldn't take more than 5 minutes... > > > > > > On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness <[email protected] > > <mailto:[email protected]>> wrote: > > > > It could definitely be normal if say you have checksum based > > checking enabled... > > > > > > On 09/12/2011 01:56 AM, David Somers-Harris wrote: > > > > > > Is it normal for rsync to take 3 hours on this transfer? > > > > Number of files: 27419348 > > Number of files transferred: 19501 > > Total file size: 185.39G bytes > > Total transferred file size: 195.92M bytes > > Literal data: 195.68M bytes > > Matched data: 241.09K bytes > > File list size: 402.01M > > File list generation time: 0.561 seconds > > File list transfer time: 0.000 seconds > > Total bytes sent: 600.61M > > Total bytes received: 2.44M > > > > sent 600.61M bytes received 2.44M bytes 54.42K bytes/sec > > total size is 185.39G speedup is 307.42 > > > > > > A normal transfer of 600 MB for me between these two servers > > only takes a few seconds.... Is there some way I can speed this > up? > > > > > > > > - -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. [email protected] (work) > Orlando, Florida [email protected] (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.17 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk5uuhYACgkQVKC1jlbQAQcV1wCeLLN+f45ydJre5UHmrdIj77FK > OakAoOMdRTPLvdnSFSEX9407LiWuEMH3 > =O2vB > -----END PGP SIGNATURE----- > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
