-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The rsync is doing a whole lot more than transferring 600MB of data. It is transferring 600MB of differences. It is going through 27.4 million files on one end and possibly that many on the other. That is a 402MB listing of files plus as many as 54.8 million calls to stat().
Rsync is not about minimizing transfer time. It is about minimizing network IO at the expense of CPU and disk IO. Rsync reduced your network IO from 185.39GB to 600MB only 200 of which was actually differences in your data. IOW rsync did its job. The unfortunate truth is that no file based tool will perform anywhere near wire speeds when it has millions of files to deal with. If you want something wire speed then you should look for a block level tool. On 09/12/11 21:54, David Somers-Harris wrote: > No, I don't have checksum based checking enabled. > > Sorry for neglecting to mention the options I'm using, here they are: > > *rsync -vah --exclude=".*" --delete --stats* > > Would anything in there cause it to be unnecessarily slow? It's taking 3 > hours but shouldn't take more than 5 minutes... > > > On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness <[email protected] > <mailto:[email protected]>> wrote: > > It could definitely be normal if say you have checksum based > checking enabled... > > > On 09/12/2011 01:56 AM, David Somers-Harris wrote: > > > Is it normal for rsync to take 3 hours on this transfer? > > Number of files: 27419348 > Number of files transferred: 19501 > Total file size: 185.39G bytes > Total transferred file size: 195.92M bytes > Literal data: 195.68M bytes > Matched data: 241.09K bytes > File list size: 402.01M > File list generation time: 0.561 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 600.61M > Total bytes received: 2.44M > > sent 600.61M bytes received 2.44M bytes 54.42K bytes/sec > total size is 185.39G speedup is 307.42 > > > A normal transfer of 600 MB for me between these two servers > only takes a few seconds.... Is there some way I can speed this up? > > > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. [email protected] (work) Orlando, Florida [email protected] (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5uuhYACgkQVKC1jlbQAQcV1wCeLLN+f45ydJre5UHmrdIj77FK OakAoOMdRTPLvdnSFSEX9407LiWuEMH3 =O2vB -----END PGP SIGNATURE----- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
