Re: Multi-Threading?
Ok, you guys' suggestions helped alot. I changed --compress to --whole-file, and in fact that increased my speed from 4.5Mb/s to 18.5! Much better. Still core-bound though. (not disk, or network) Now that I've finished my restore, I've run iperf. Droog -- Hex 937 Mbits/sec! Merlin -- Hex 40.6 Mbits/sec (wifi) So I'm getting amazing performance from the Gb connexion; I should because I used expensive Ubiquiti ToughCable and ToughConnectors for the 70' run to the garage where my backup server is. BTW, Ubiquity has some amazing products, especially the NanoBridge M2. My wifi performance is not too good, considering that I run 5GHz only, 802.11n. It's a Netgear WNDR3700 router and the client card is the Intel 4595 built into my HP 8710w laptop. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can see is on the backup server one core of the CPU is running 100% rsync. Clearly rsync is not multi-threaded. That's probably the delta-transfer algorithm. Turning it off with --whole-file should help somewhat. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
Thanks, but the whole function of my backup server pivots on rsync features. Need something rsync-like, but multi-threaded, or with a whole lot less overhead. On Tuesday 17 May, 2011 09:18:01 Chris Hawkins wrote: I have no idea about potential rsync modifications, but you might try FDT to solve this problem: http://monalisa.cern.ch/FDT/ I have found it easy to use and it absolutely maxes all available bandwidth for the fastest possible data transfer. It doesn't have rsync goodies like update and only changed data, but it's multithreaded, multi-stream, and all around a really fast data mover. Chris - Original Message - I have a backup server now restoring 6TB of data to a client machine. This has been going on for four days now, and no sign of getting close to completion. The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can see is on the backup server one core of the CPU is running 100% rsync. Clearly rsync is not multi-threaded. In researching this I find that a change to multi-threaded goodness would require a massive rewrite, and would only be considered for an rsync replacement. Is there such a replacement in the offing? Is there any sort of workaround for the time being? ATM the large amount of data is the /home directory on the client machine. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Also, make sure you aren't running with compression as it will just waste CPU cycles. If you are running rsync over ssh it might also be worth while to setup a one time use rsyncd or even NFS mount to eliminate the encryption overhead. On 05/17/11 12:50, Matt McCutchen wrote: On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can see is on the backup server one core of the CPU is running 100% rsync. Clearly rsync is not multi-threaded. That's probably the delta-transfer algorithm. Turning it off with --whole-file should help somewhat. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3Sp+AACgkQVKC1jlbQAQeSbwCdGmV1zMtCk9Phx12M/RvuCW93 MfgAoJ7qJmMKTzftRvpnFn/XScijDsbw =WJp3 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
Matt McCutchen (m...@mattmccutchen.net) wrote on 17 May 2011 12:50: On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can see is on the backup server one core of the CPU is running 100% rsync. Clearly rsync is not multi-threaded. That's probably the delta-transfer algorithm. Yes. And what's important is not the net flux, it's the transfer rate, which can be a lot higher if there are few changes. You might want to check the read rate at the server, it should be in the hundreds of MB/s. If not rsync got stuck somewhere. Multi-threading rsync is of little use because the the bottleneck will be the disk IO. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 17 May 2011, cac...@quantum-sci.com wrote: In researching this I find that a change to multi-threaded goodness would require a massive rewrite, and would only be considered for an rsync replacement. Abstracting the core functionality into a librsync.so would be really spiffy too... ..Ch:W.. -- An idea does not gain truth as it gains followers. Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tuesday 17 May, 2011 10:43:21 you wrote: Wow! That's a lot of data you're transferring into one server, even with a GB pipe you're using, which appears to be using a fraction of it. Yes, it's my home theater computer with all the movies. I've just set up a RAID array and need to move my data there. At any rate, I was wondering can you break up that /home directory into several parts in terms of your rsync command line. Example: rsync... /home/a... rsync... /home/b... rsync... /home/c... I thought of this, but so many subdirectories are tiny, and it would be a fussy operation to transfer them individually. If I needed this as a matter of course I might go to the trouble, but I am just waiiting for this to complete so I can watch some decent shows. (MythTV) I've removed the -compress flag and added --whole-file, and am waiting for it to recalculate the transfer. It has already done some small files, and one large file which transferred at 18.65MB/s, a big improvement over 4.5. Still running CPU at 100% on one core. On Tuesday 17 May, 2011 10:46:12 Carlos Carvalho wrote: You might want to check the read rate at the server, it should be in the hundreds of MB/s. If not rsync got stuck somewhere. Carlos as I am CPU-bound on the server, I don't really take the disk read rate very seriously, but iotop says rsync is doing around 85M/s on these three WD Green 2TB drives. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: On Tue, 17 May 2011, cac...@quantum-sci.com wrote: In researching this I find that a change to multi-threaded goodness would require a massive rewrite, and would only be considered for an rsync replacement. Abstracting the core functionality into a librsync.so would be really spiffy too... All easy to say, harder to do (and maintain). I'm thankful that rsync meets my needs right now, and that Wayne has gotten as many bugs out of it as he has over the last few years. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 17 May 2011, Matt McCutchen wrote: On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: Abstracting the core functionality into a librsync.so would be really spiffy too... All easy to say, harder to do (and maintain). I'm thankful that rsync meets my needs right now, and that Wayne has gotten as many bugs out of it as he has over the last few years. Agreed. I mostly meant that as tongue-in-cheek. Creating librsync.so seems about as hard (or harder) as multi-threading. ..Ch:W.. -- An idea does not gain truth as it gains followers. Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
