threaded rsync to improve performance over long distance ?

2013-01-20 Thread Markus Moeller
I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus -- Please use reply-all for most replies to

Re: threaded rsync to improve performance over long distance ?

2013-01-20 Thread Kevin Korb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rsync does not use scp in any way. On 01/20/13 14:47, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible

Re: threaded rsync to improve performance over long distance ?

2013-01-20 Thread Joe
At the risk of stating the obvious, is there any way you can break your datasets into smaller pieces? Then, you could start several rsync processes in parallel - one for each piece, possibly using separate Internet connections. As Kevin says, I'm sure there is no direct relationship between

Re: threaded rsync to improve performance over long distance ?

2013-01-20 Thread Markus Moeller
Hi Kevin, but rsync uses ssh doesn't it ? Are there any tuning options for buffers to deal with the tcp SYN SYN-ACK delay ? Thank you Markus Kevin Korb k...@sanitarium.net wrote in message news:50fc5e6f.1000...@sanitarium.net... -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rsync does

Re: threaded rsync to improve performance over long distance ?

2013-01-20 Thread Kevin Korb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You can use the HPN enhancement patches for ssh. They do make a big difference: http://www.psc.edu/index.php/hpn-ssh On 01/20/13 18:22, Markus Moeller wrote: Hi Kevin, but rsync uses ssh doesn't it ? Are there any tuning options for buffers

Re: threaded rsync to improve performance over long distance ?

2013-01-20 Thread Greg Siekas
With the SSH-HPN you can achieve over 200MB/sec. Depends on your latency and packet drops but you should certainly try a modified ssh/scp with rsync. Also are you using compression with rsync -z? That's an area that could be multi threaded. On Jan 20, 2013, at 3:25 PM, Kevin Korb

[SCM] The rsync repository. - branch master updated

2013-01-20 Thread Rsync CVS commit messages
The branch, master has been updated via eec2608 Improve description of --max-delete. from 76340ea Fix weird error in test programs on SunOS. ;a=shortlog;h=master - Log - commit