Re: Much improved speeds of rsync via SSH - something to consider

[Karl O. Pinc via rsync]

On Mon, 2 Apr 2018 15:05:14 +0200
Thanassis Tsiodras via rsync  wrote:

> I recently concluded a bug hunt to trace why my rsync-ing to an SBC
> was much slower than the corresponding iperf3-reported speeds. To
> give a concise summary of the situation, in slow wifi links using SSH
> with ProxyCommand tremendously speeds up things:

> $ rsync -avz --progress -e 'ssh -o "ProxyCommand nc %h %p"'
> ./sample.data root@192.168.1.150:/dev/shm/

> If it isn't clear - the speed of the upload went from 543 kbytes/sec
> to 2690 kbytes/sec.
> 
> If you want to see why - and how I traced this down - you can read my
> complete report in the UNIX StackExchange forum. To avoid being
> classified as a spammer, I won't include a link - search for question
> 434825 in the search bar. The executive summary is that SSH disables
> Nagle's algorithm by default - and in slow links this can cause
> tremendous impact (as you see above).
> 
> Hope this helps people that backup over slow links (wi-fi or
> otherwise).

In the interest of collecting ssh-related issues in a single
thread I add this note:

The HPN patches to the portable OpenSSH implementation can
improve speed over high speed links with long round trip times.

  https://www.psc.edu/hpn-ssh

It also helps in those cases where CPU is a bottleneck and
encryption of transmitted data unnecessary.

The FAQ is also interesting.

The executive summary is:

  SCP and the underlying SSH2 protocol implementation in OpenSSH is
  network performance limited by statically defined internal flow
  control buffers.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Much improved speeds of rsync via SSH - something to consider

[Thanassis Tsiodras via rsync]

Dear rsync devs,

I recently concluded a bug hunt to trace why my rsync-ing to an SBC was
much slower than the corresponding iperf3-reported speeds. To give a
concise summary of the situation, in slow wifi links using SSH with
ProxyCommand tremendously speeds up things:

$ dd if=/dev/urandom bs=1M count=50 of=sample.data
50+0 records in
50+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.24135 s, 217 MB/s

$ rsync -avz --progress ./sample.data root@192.168.1.150:/dev/shm/
sending incremental file list
sample.data
 52,428,800 100%  575.23kB/s0:01:28 (xfr#1, to-chk=0/1)

sent 52,458,757 bytes  received 35 bytes  543,614.42 bytes/sec
total size is 52,428,800  speedup is 1.00

$ ssh root@192.168.1.150 'rm /dev/shm/sample.data'

$ rsync -avz --progress -e 'ssh -o "ProxyCommand nc %h %p"'
./sample.data root@192.168.1.150:/dev/shm/
sending incremental file list
sample.data
 52,428,800 100%3.40MB/s0:00:14 (xfr#1, to-chk=0/1)

sent 52,458,757 bytes  received 35 bytes  2,690,194.46 bytes/sec
total size is 52,428,800  speedup is 1.00

If it isn't clear - the speed of the upload went from 543 kbytes/sec to
2690 kbytes/sec.

If you want to see why - and how I traced this down - you can read my
complete report in the UNIX StackExchange forum. To avoid being classified
as a spammer, I won't include a link - search for question 434825 in the
search bar. The executive summary is that SSH disables Nagle's algorithm by
default - and in slow links this can cause tremendous impact (as you see
above).

Hope this helps people that backup over slow links (wi-fi or otherwise).

Cheers,
Thanassis Tsiodras, Dr.-Ing.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Rsync via ssh hangs on same file repeatedly

[Gawain]

At 9:26 PM -0500 on 5/9/06, Tony Abernethy wrote:



Flames invitied if I'm wrong, but I think you're looking at
the last file successfully transferred as opposed to the
first file unsuccessfully transferred.

I think I saw a /var in there
You get something ungodly long trying to rsync the file that is logging the
rsync.
something like
rsync -av --exclude log/ --exclude work/ $name
/sdb1/2006-04-01-hostname/
may be in order  (the above is just a local copy to a USB drive.

You probably have some timing issues as to whether or not
rsyncing the log file actually works or is an infinite loop.
I'd put watching the grass grow at a higher priority than trying to fix
the problem.


Thanks for your reply. I needed some time to continue testing. It 
turned out that the LaCie Ethernet disk was the culprit. Something 
was causing a timeout during the data write and the process was 
hanging up. I've since installed another internal drive in the 
machine and we've been happily backing up servers ever since.

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync via ssh hangs on same file repeatedly

[Wayne Davison]

On Tue, May 09, 2006 at 08:52:57PM -0500, Gawain wrote:
 I'm having problems with an rsync via ssh process hanging during
 transfer.

The issues and debugging page on the rsync website gives some advice on
how to investigate a hang.  It mentions running netstat on each system
to check on the state of the Send-Q and Recv-Q for the socket, and also
using a tool such as strace or truss to see what system calls each
process gets stuck on.  If you can provide this information, it will
help us to diagnose the problem.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync via ssh hangs on same file repeatedly

[Gawain]

Hi all,

I'm having problems with an rsync via ssh process hanging during transfer.

The basic problem, (gory details to follow):

 I've set up a machine on my LAN to act as the receiver and have two 
(soon to be more) remote servers transferring files via a script.


It seems as if certain files are causing the transfer to hang. These 
are different files on each of the sending machines, and there 
doesn't seem to be any correlation between them. The plan is to back 
up several directories (/usr /var /home /etc) so I thought at first 
there could be some sort of memory issue as the transfers stalled 
partway through. Then I realized that even if I only transferred one 
directory the process would still hang on this particular file. In 
one case, the file was a 57kb executable and in another it was a 
238kb JPEG. Everything works fine up to that point and then bam! It 
just sits there on that file and never finishes. I've turned on some 
of the extended reporting options, and here's some output from one of 
the transfers from a /usr/bin directory:


-

1205 files to consider
appletproxy
4867 100%0.00kB/s0:00:00 (xfer#1, to-check=1185/1205)
apropos
2378 100%2.27MB/s0:00:00 (xfer#2, to-check=1184/1205)
ar
   35480 100%2.82MB/s0:00:00 (xfer#3, to-check=1183/1205)
ark
4831 100%  314.52kB/s0:00:00 (xfer#4, to-check=1182/1205)
artsbuilder
  344201 100%  517.13kB/s0:00:00 (xfer#5, to-check=1181/1205)
artscat
Killed by signal 2. 48.71kB/s0:00:00

-

In the case above the 'artscat' file hangs at precisely 55% each 
time, at exactly 32768 bytes. H. A clue, perhaps?


One of the sending systems is a RedHat box, the other is FC2. The 
Fedora system is running rsync 2.6.2, the RH box is now running 2.6.8 
after hoping that an upgrade from 2.5.5 would cure the problem. No 
change.


That leaves the receiving system as the common link, so I suspect the 
problem may lie there or with the HD. It's a G4 Mac running OS X 
10.4.6 with the standard install of rsync 2.6.3. It's a fresh install 
of the OS and the machine will be used solely as a backup server. I'm 
backing up to a LaCie 250GB Ethernet disk, formatted with the Ext3 
filesystem. I don't have enough free space on the receiver's internal 
HD to test the entire backup, but the backup of just the /usr/bin 
directory completes successfully if I back up to the internal HD 
instead of the Ethernet disk. As you can see above, the transfer 
hangs only six files into a 1205 file run when backing up to the 
LaCie. However it works fine on other large directories.


I'm out of ideas on this one, except for possibly installing a larger 
internal HD and temporarily forgetting about the LaCie.


Any ideas? I'd be grateful for any insights you can provide.

Gawain
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Rsync via ssh hangs on same file repeatedly

[Tony Abernethy]

Flames invitied if I'm wrong, but I think you're looking at
the last file successfully transferred as opposed to the
first file unsuccessfully transferred.

I think I saw a /var in there
You get something ungodly long trying to rsync the file that is logging the
rsync.
something like
rsync -av --exclude log/ --exclude work/ $name
/sdb1/2006-04-01-hostname/
may be in order  (the above is just a local copy to a USB drive.

You probably have some timing issues as to whether or not
rsyncing the log file actually works or is an infinite loop.
I'd put watching the grass grow at a higher priority than trying to fix
the problem.



 -Original Message-
 From: [EMAIL PROTECTED]
 [mailto:[EMAIL PROTECTED] Behalf Of
 Gawain
 Sent: Tuesday, May 09, 2006 8:53 PM
 To: rsync@lists.samba.org
 Subject: Rsync via ssh hangs on same file repeatedly


 Hi all,

 I'm having problems with an rsync via ssh process hanging during transfer.

 The basic problem, (gory details to follow):

   I've set up a machine on my LAN to act as the receiver and have two
 (soon to be more) remote servers transferring files via a script.

 It seems as if certain files are causing the transfer to hang. These
 are different files on each of the sending machines, and there
 doesn't seem to be any correlation between them. The plan is to back
 up several directories (/usr /var /home /etc) so I thought at first
 there could be some sort of memory issue as the transfers stalled
 partway through. Then I realized that even if I only transferred one
 directory the process would still hang on this particular file. In
 one case, the file was a 57kb executable and in another it was a
 238kb JPEG. Everything works fine up to that point and then bam! It
 just sits there on that file and never finishes. I've turned on some
 of the extended reporting options, and here's some output from one of
 the transfers from a /usr/bin directory:

 -

 1205 files to consider
 appletproxy
  4867 100%0.00kB/s0:00:00 (xfer#1, to-check=1185/1205)
 apropos
  2378 100%2.27MB/s0:00:00 (xfer#2, to-check=1184/1205)
 ar
 35480 100%2.82MB/s0:00:00 (xfer#3, to-check=1183/1205)
 ark
  4831 100%  314.52kB/s0:00:00 (xfer#4, to-check=1182/1205)
 artsbuilder
344201 100%  517.13kB/s0:00:00 (xfer#5, to-check=1181/1205)
 artscat
 Killed by signal 2. 48.71kB/s0:00:00

 -

 In the case above the 'artscat' file hangs at precisely 55% each
 time, at exactly 32768 bytes. H. A clue, perhaps?

 One of the sending systems is a RedHat box, the other is FC2. The
 Fedora system is running rsync 2.6.2, the RH box is now running 2.6.8
 after hoping that an upgrade from 2.5.5 would cure the problem. No
 change.

 That leaves the receiving system as the common link, so I suspect the
 problem may lie there or with the HD. It's a G4 Mac running OS X
 10.4.6 with the standard install of rsync 2.6.3. It's a fresh install
 of the OS and the machine will be used solely as a backup server. I'm
 backing up to a LaCie 250GB Ethernet disk, formatted with the Ext3
 filesystem. I don't have enough free space on the receiver's internal
 HD to test the entire backup, but the backup of just the /usr/bin
 directory completes successfully if I back up to the internal HD
 instead of the Ethernet disk. As you can see above, the transfer
 hangs only six files into a 1205 file run when backing up to the
 LaCie. However it works fine on other large directories.

 I'm out of ideas on this one, except for possibly installing a larger
 internal HD and temporarily forgetting about the LaCie.

 Any ideas? I'd be grateful for any insights you can provide.

 Gawain
 --
 To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync via ssh tunnel

[Radio Gong 2000 GmbH Co. KG [Technik]]

Hi there,

I want to put these two commands in just one command:

ssh -i ~/.ssh/id_dsa -C -l sascha -L 873:rsync-server:873 bridge-server
rsync -auz rsync://[EMAIL PROTECTED]/share /local-path

How can I do this. With Option --rsh (-e) rsync wants to execute the command 
on the remote machine and not on localhost as I want it...

rsync -au --rsh=ssh -C -l sascha -i ~/.ssh/id_dsa -L 873:rsync-server:873 
bridge-server [EMAIL PROTECTED]::share /tmp/

Or does anyone know how to wirte the 2 commands in a script and getting first 
the tunnel and after execute rsync on localhost???

Greetings 

Sascha
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync via ssh tunnel

[Wayne Davison]

On Wed, Sep 08, 2004 at 09:42:32AM +0200, Radio Gong 2000 GmbH  Co. KG [Technik] 
wrote:
 I want to put these two commands in just one command:
 
 ssh -i ~/.ssh/id_dsa -C -l sascha -L 873:rsync-server:873 bridge-server
 rsync -auz rsync://[EMAIL PROTECTED]/share /local-path

This was covered a few weeks ago on the mailing list.  While you can't
combine them into a single command, you can run a simple sleep remotely
via the first command and then have the second command use the forwarded
port before the sleep goes away (which will keep the connection open
after the sleep command terminates).  Like this (adding -f and sleep 40):

ssh -f -i ~/.ssh/id_dsa -C -l sascha -L 873:rsync-server:873 bridge-server sleep 40
rsync -auz rsync://[EMAIL PROTECTED]/share /local-path

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync via ssh script

[Andrzej Filip]

Tim Nonnast wrote:
does anybody know, how a bash shell script looks, which automatically 
enters the ssh password?

the rsync call should be:
rsync -avz -e ssh /home/johndoe/data/repository 
[EMAIL PROTECTED]:/home/johndoe/junk

the call causes a password question.

Is ist possible to automate that by a cron? If yes how should the script 
look to automatically enter the password?
Have you considered using key authentication in ssh ?

AFAIK it is a standard way of making ssh session available to cron scripts. 
The remote site may assign single command to session suthenticated with the 
the key.

--
Andrzej [en:Andrew] Adam Filip [EMAIL PROTECTED] [EMAIL PROTECTED]
http://anfi.homeunix.net/ http://slashdot.org/~anfi
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: rsync via ssh script

[Jason Ferguson]

I found this very useful for getting around this problem with osX

http://www.bombich.com/mactips/rsync.html

-Jason

-Original Message-
From: Andrzej Filip [mailto:[EMAIL PROTECTED]
Sent: Tuesday, March 23, 2004 7:46 AM
To: [EMAIL PROTECTED]
Subject: Re: rsync via ssh script


Tim Nonnast wrote:
 does anybody know, how a bash shell script looks, which automatically 
 enters the ssh password?
 
 the rsync call should be:
 rsync -avz -e ssh /home/johndoe/data/repository 
 [EMAIL PROTECTED]:/home/johndoe/junk
 
 the call causes a password question.
 
 Is ist possible to automate that by a cron? If yes how should the script  look to 
 automatically enter the password?

Have you considered using key authentication in ssh ?

AFAIK it is a standard way of making ssh session available to cron scripts. The remote 
site may assign single command to session suthenticated with the 
the key.

-- 
Andrzej [en:Andrew] Adam Filip [EMAIL PROTECTED] [EMAIL PROTECTED]
http://anfi.homeunix.net/ http://slashdot.org/~anfi
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


*
This e-mail message, including any attachments, is intended solely for the use of the 
addressee and may contain confidential information. 
If it is not intended for you, please inform the sender and delete the e-mail and any 
attachments immediately. Any review, retransmission, 
disclosure, copying or modification of it is strictly forbidden.  Please be advised 
that the views and opinions expressed in this e-mail may not reflect the views and 
opinions of Newsquest (Herald  Times) Limited or any of its subsidiary companies.  

Whilst we take reasonable precautions to ensure that our emails are 
free from viruses, we cannot be responsible for any viruses transmitted with this 
e-mail and recommend that you subject any incoming e-mail to your own virus checking 
procedures.  Use of this or any other e-mail facility signifies consent to any 
interception we might lawfully carry out to prevent abuse of these facilities.
***

--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: rsync via ssh

[Harala, Sauli]

Hi, 

We were just today making some promising tests
with F-Secure sshd2 v 3.0 and rsync 2.4.6 
in OSF1 (Digital Unix).

The rsync+'ssh2 v2.0' server on OSF1 
were hanging - but first small tests 
did not any more show this trouble.

This hanging was most often totally 
preventing using ssh+rsync as OSF1 side server 
Visa versa (HP/UX + ssh2 v2.0 as server and the OSF1
as client) was working OK.

sh



-Original Message-
From: IVANYI Ivan [mailto:[EMAIL PROTECTED]]
Sent: 20. syyskuuta 2001 9:21
To: Rsync
Subject: rsync via ssh


Hi,

We have 2 problems with rsync over ssh that we
haven't been able to replicate all the time.

rsync via ssh

[IVANYI Ivan]

Hi,

We have 2 problems with rsync over ssh that we
haven't been able to replicate all the time.


Basically, we have a developement server running
Solaris 2.5.1 and another running Solaris 2.6, the
development tree is shared via nfs. When minor
changes are made they are distibuted to various
mirror sites via a perl script which does some
checks (path, etc) and runs 'rsync -e ssh '
The script distributes the changed files to about
7 other machines.

All the sites have rsync 2.4.6, the development
servers use ssh 2.4.0


The first problem is sometimes rsyncing to an IRIX
6.5 machine the script seems to block (we debugged
the script so it doesn't seem to be the perl,
besides as I said basically it just runs rsync in
a loop). The files are transferred but after the
transfer rsync (or ssh) just seem to sit there
doing nothing and you have to ctrl-C so that the
script will continue. SSH on this machine has been
upgraded several times, rsync recompiled, binaries
downloaded... in order to see if that would fix
it. Funnily enough on another IRIX machine with
almost exactly the same setup has no problem. (I'm
sure you'll say find the differences!)

The second problem is to a linux machine (Redhat
7.1 with openssh 2.5.2p2). After transfer
sometimes text files end up 'not quite the same'
at the other end. 1 or 2 characters might be
different for example. rsyncing again will usually
fix this. For html and flat file databases
obviously this is a bit of a problem.


We haven't been able to determine under what
circumstances these problems occur, but I hope
someone might have some idea...


thanks

-- 
Ivan Ivanyi

Swiss Institute of Bioinformatics
1, rue Michel Servet
CH-1211 Genève 4
Switzerland

Tel: (+41 22) 702 58 33
Fax: (+41 22) 702 58 58
E-mail: [EMAIL PROTECTED]