poor rsync performance with specific Linux kernel version?
Hi all I am new to this list but a happy rsync user for quite some time. Thanks for this great tool. We are experiencing very slow rsync performance when using it as a backup tool for our server virtualisation "Proxmox VE" (http://pve.proxmox.com/wiki/Main_Page). I have searched for our issue and posted at the Proxmox forums ( http://forum.proxmox.com/threads/6175-rsync-%28host%29-performance-issues-with-kernel-2.6.32-on-Supermicro-platform) but no results so I hope to ask the question here properly: Trying to rsync an 8 GB testfile (dd if=/dev/zero of=/var/lib/vz/testfile_MDO bs=1024k count=8192 conv=fdatasync) on quite a high-end system (Supermicro "X8DTI-F" board, 12 GB RAM, SAS disks on Adaptec RAID controller) running Proxmox with a (Debian) Linux kernel at that moment: rsync --progress /var/lib/vz/testfile_MDO /var/lib/vz/testfile_MDO2 testfile_MDO 592379904 6% 949.34kB/s 2:20:24 ^C I had to cancel due to the low speed at that time. The issue seems related to the specific combination of hardware and that particular kernel (Proxmox' "Debian" kernel 2.6.32). If I replace that kernel with their alternative "Redhat" 2.6.18 kernel version, the same rsync process is fast and will sync/copy the 8GB file in less than 2 minutes. We can replicate the slow speed problem on two other servers which use almost identical hardware. Can I get any help or idea here on this list what I could try to further isolate this issue? Regards, Michael -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 17 May 2011, Matt McCutchen wrote: > On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: > > > > Abstracting the core functionality into a librsync.so would be really > > spiffy too... > > All easy to say, harder to do (and maintain). I'm thankful that rsync > meets my needs right now, and that Wayne has gotten as many bugs out of > it as he has over the last few years. Agreed. I mostly meant that as tongue-in-cheek. Creating librsync.so seems about as hard (or harder) as multi-threading. ..Ch:W.. -- "An idea does not gain truth as it gains followers." Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: > On Tue, 17 May 2011, cac...@quantum-sci.com wrote: > > > In researching this I find that a change to multi-threaded goodness > > would require a massive rewrite, and would only be considered for an > > rsync replacement. > > Abstracting the core functionality into a librsync.so would be really > spiffy too... All easy to say, harder to do (and maintain). I'm thankful that rsync meets my needs right now, and that Wayne has gotten as many bugs out of it as he has over the last few years. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tuesday 17 May, 2011 10:43:21 you wrote: > Wow! That's a lot of data you're transferring into one server, even with a GB > pipe you're using, which appears to be using a fraction of it. Yes, it's my home theater computer with all the movies. I've just set up a RAID array and need to move my data there. > At any rate, I was wondering can you break up that /home directory into > several parts in terms of your "rsync" command line. > > Example: rsync... /home/a... > rsync... /home/b... > rsync... /home/c... I thought of this, but so many subdirectories are tiny, and it would be a fussy operation to transfer them individually. If I needed this as a matter of course I might go to the trouble, but I am just waiiting for this to complete so I can watch some decent shows. (MythTV) I've removed the -compress flag and added --whole-file, and am waiting for it to recalculate the transfer. It has already done some small files, and one large file which transferred at 18.65MB/s, a big improvement over 4.5. Still running CPU at 100% on one core. On Tuesday 17 May, 2011 10:46:12 Carlos Carvalho wrote: > You might want to check the read rate at the server, it should be > in the hundreds of MB/s. If not rsync got stuck somewhere. Carlos as I am CPU-bound on the server, I don't really take the disk read rate very seriously, but iotop says rsync is doing around 85M/s on these three WD Green 2TB drives. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 17 May 2011, cac...@quantum-sci.com wrote: > In researching this I find that a change to multi-threaded goodness > would require a massive rewrite, and would only be considered for an > rsync replacement. Abstracting the core functionality into a librsync.so would be really spiffy too... ..Ch:W.. -- "An idea does not gain truth as it gains followers." Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
Matt McCutchen (m...@mattmccutchen.net) wrote on 17 May 2011 12:50: >On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: >> The connexion is Gb enet end-to-end, and is running at only 40Mb/s. >> It has far more capacity than that. The only limiting factor I can >> see is on the backup server one core of the CPU is running 100% rsync. >> Clearly rsync is not multi-threaded. > >That's probably the delta-transfer algorithm. Yes. And what's important is not the net flux, it's the transfer rate, which can be a lot higher if there are few changes. You might want to check the read rate at the server, it should be in the hundreds of MB/s. If not rsync got stuck somewhere. Multi-threading rsync is of little use because the the bottleneck will be the disk IO. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Also, make sure you aren't running with compression as it will just waste CPU cycles. If you are running rsync over ssh it might also be worth while to setup a one time use rsyncd or even NFS mount to eliminate the encryption overhead. On 05/17/11 12:50, Matt McCutchen wrote: > On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: >> The connexion is Gb enet end-to-end, and is running at only 40Mb/s. >> It has far more capacity than that. The only limiting factor I can >> see is on the backup server one core of the CPU is running 100% rsync. >> Clearly rsync is not multi-threaded. > > That's probably the delta-transfer algorithm. Turning it off with > --whole-file should help somewhat. > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3Sp+AACgkQVKC1jlbQAQeSbwCdGmV1zMtCk9Phx12M/RvuCW93 MfgAoJ7qJmMKTzftRvpnFn/XScijDsbw =WJp3 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
Thanks, but the whole function of my backup server pivots on rsync features. Need something rsync-like, but multi-threaded, or with a whole lot less overhead. On Tuesday 17 May, 2011 09:18:01 Chris Hawkins wrote: > I have no idea about potential rsync modifications, but you might try FDT to > solve this problem: > > http://monalisa.cern.ch/FDT/ > > I have found it easy to use and it absolutely maxes all available bandwidth > for the fastest possible data transfer. It doesn't have rsync goodies like > update and only changed data, but it's multithreaded, multi-stream, and all > around a really fast data mover. > > Chris > > - Original Message - > > I have a backup server now restoring 6TB of data to a client machine. > > This has been going on for four days now, and no sign of getting close > > to completion. > > > > The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It > > has far more capacity than that. The only limiting factor I can see is > > on the backup server one core of the CPU is running 100% rsync. > > Clearly rsync is not multi-threaded. > > > > In researching this I find that a change to multi-threaded goodness > > would require a massive rewrite, and would only be considered for an > > rsync replacement. > > > > Is there such a replacement in the offing? Is there any sort of > > workaround for the time being? ATM the large amount of data is the > > /home directory on the client machine. > > > > -- > > Please use reply-all for most replies to avoid omitting the mailing > > list. > > To unsubscribe or change options: > > https://lists.samba.org/mailman/listinfo/rsync > > Before posting, read: > > http://www.catb.org/~esr/faqs/smart-questions.html > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: > The connexion is Gb enet end-to-end, and is running at only 40Mb/s. > It has far more capacity than that. The only limiting factor I can > see is on the backup server one core of the CPU is running 100% rsync. > Clearly rsync is not multi-threaded. That's probably the delta-transfer algorithm. Turning it off with --whole-file should help somewhat. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
I have no idea about potential rsync modifications, but you might try FDT to solve this problem: http://monalisa.cern.ch/FDT/ I have found it easy to use and it absolutely maxes all available bandwidth for the fastest possible data transfer. It doesn't have rsync goodies like update and only changed data, but it's multithreaded, multi-stream, and all around a really fast data mover. Chris - Original Message - > I have a backup server now restoring 6TB of data to a client machine. > This has been going on for four days now, and no sign of getting close > to completion. > > The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It > has far more capacity than that. The only limiting factor I can see is > on the backup server one core of the CPU is running 100% rsync. > Clearly rsync is not multi-threaded. > > In researching this I find that a change to multi-threaded goodness > would require a massive rewrite, and would only be considered for an > rsync replacement. > > Is there such a replacement in the offing? Is there any sort of > workaround for the time being? ATM the large amount of data is the > /home directory on the client machine. > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Multi-Threading?
I have a backup server now restoring 6TB of data to a client machine. This has been going on for four days now, and no sign of getting close to completion. The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can see is on the backup server one core of the CPU is running 100% rsync. Clearly rsync is not multi-threaded. In researching this I find that a change to multi-threaded goodness would require a massive rewrite, and would only be considered for an rsync replacement. Is there such a replacement in the offing? Is there any sort of workaround for the time being? ATM the large amount of data is the /home directory on the client machine. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html