Re: Rsync / ssh high cpu load
Hi,
ssh: OpenSSH_5.3p1
rsync: 3.0.6
OS: CentOS release 6.3
I got the same problem here, syncing 95M to amazon ec2 without
--bwlimit parameter. It should be not caused by cygwin. From 3.0.6 to
latest version, I did not find any ssh related problem fixed.
After the syncing job is over, ssh process did not quit and consumes
100% CPU. I collected other information, wish it may help.
related process:
root 29296 0.0 0.0 107612 964 pts/1TDec06 0:00 rsync
-arv -e ssh data r...@xxx.com:/tmp/data/ --exclude=.svn
root 29297 99.9 0.0 59760 3476 pts/1RDec06 770:20 ssh
-l root xxx.com rsync --server -vlogDtpre.iLs . /tmp/data/
root 29306 0.0 0.0 55160 2028 ?Ss Dec06 0:00 scp
-f -- /yyy/zzz.sql
# strace -fF -v -p 29297
Process 29297 attached - interrupt to quit
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
rt_sigreturn(0x16) = -1 EINTR (Interrupted system call)
ioctl(4, SNDCTL_TMR_CONTINUE or TCSETSF, {c_iflags=0x500,
c_oflags=0x5, c_cflags=0xbf, c_lflags=0x8a3b, c_line=0,
c_cc=\x03\x1c\x7f\x15\x04\x00\x01\x00\x11\x13\x1a\x00\x12\x0f\x17\x16\x00\x00\x00})
= ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
rt_sigreturn(0x16) = -1 EINTR (Interrupted system call)
ioctl(4, SNDCTL_TMR_CONTINUE or TCSETSF, {c_iflags=0x500,
c_oflags=0x5, c_cflags=0xbf, c_lflags=0x8a3b, c_line=0,
c_cc=\x03\x1c\x7f\x15\x04\x00\x01\x00\x11\x13\x1a\x00\x12\x0f\x17\x16\x00\x00\x00})
= ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
rt_sigreturn(0x16) = -1 EINTR (Interrupted system call)
(repeat)
-Oorspronkelijk bericht-
Van: Jan Alphenaar [mailto:jan.alphenaar at dotcolour.com]
Verzonden: woensdag 17 februari 2010 23:17
Aan: 'rsync at lists.samba.org'
Onderwerp: RE: Rsync / ssh high cpu load
Hi again,
Still working on this one. When I rsync/ssh a big file to my server over the
local network (100Mbit/s) with the --bwlimit=1000 I only have a small CPU
load. SSH takes about 10% and rsync approximately 0-5%. In the end the
transfer speed usually is around the value I set in bwlimit.
When I transfer the same file to a remote machine, where the bandwidth is
limited to roughly 140KB/s and the bandwidth limit is set to something below
this value I get around the same values which I have on the local network,
SSH 10% and rsync somewhere between 0-5%.
If I change the bwlimit value to 150KB/s or higher the SSH load goes up to
70% and rsync to 30% fully consuming my CPU.
I think the problem is pretty clear now. Does anybody have an idea, what the
next step could be to the resolution of this ?
Regards,
Jan
-Oorspronkelijk bericht-
Van: Jan Alphenaar [mailto:jan.alphenaar at dotcolour.com]
Verzonden: woensdag 17 februari 2010 12:17
Aan: 'rsync at lists.samba.org'
Onderwerp: RE: Rsync / ssh high cpu load
Hi,
In an attempt to narrow this down a bit I installed a RedHat machine with
openssh-5.3p1 and rsync-3.0.7 and did the same test as below.
Transferring data, both up and downstream, are not giving me any CPU load.
Although there is a small version difference in the rsync versions, I would
say that this is caused by the cygwin layer.
I will mention this in the cygwin list.
Still, if anybody has any ideas how to solve the issue below, please let me
know.
Warm regards,
Jan
-Oorspronkelijk bericht-
Van: Jan Alphenaar [mailto:jan.alphenaar at dotcolour.com]
Verzonden: dinsdag 16 februari 2010 17:31
Aan: 'rsync at lists.samba.org'
Onderwerp: Rsync / ssh high cpu load
Hi everybody,
I am using Cygwin rsync/ssh to synchronize my files from my home computer to
my server. I raised a question on the Cygwin list as well, but thought this
list would be appropriate too.
The issue is that the CPU on the sending side is fully allocated by
rsync/ssh (both taking 50%) during a file transfer. This is the command I
use:
rsync -e ssh * user@server:/data
After some more testing on this I figured out that rsync/ssh is only taking
the full cpu if the bandwidth allocated to the session exceeds the true
bandwidth available on the link.
For example, if the uplink is 128KB/s and I pass in the --bwlimit=24
parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu,
but --bwlimit=132 does. Note this is the first install, the /data directory
on the receiving side is empty.
Would this be because of packet loss on the link and packets have to be
retransmitted ? I will do a quick test tonight with rsync/ssh between two
linux boxes (without cygwin), see what that gives me.
Note that I am using these versions:
Cygwin: 1.7.1-1
ssh: OpenSSH_5.3p1
rsync: 3.0.6
Has anybody seen this before ?
Thanks in advance.
Regards,
Jan
--
Phone: 18601240631
Blog: http://liruqi.info
Weibo: http://weibo.com/liruqi
--
Phone: 18601240631
Blog: http://liruqi.info
Weibo: http://weibo.com/liruqi
--
Please use reply-all for most replies to avoid omitting
RE: Rsync / ssh high cpu load
Hi, Today I did another interesting test in an attempt to resolve this issue. With Microsoft Process Explorer (downloadable from the Microsoft website) it is possible to 'pause' and 'resume' processes. When I start a rsync/ssh session to my server and pause the SSH process. The rsync process consumes 100% CPU, while the IO is 0. In the case I pause rsync and leave SSH running, there is no CPU load and no IO. In the first case rsync cannot send any data into the SSH tunnel. I think rsync goes into a CPU consuming loop while it waits for access to the SSH tunnel. This is somehow the same as when rsync tries to send data with SSH and it has to wait because it is exceeding the bwlimit variable. The next step for me would be to dive into the rsync code, which is something I really try to avoid at this moment. Any volunteers ? Regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: woensdag 17 februari 2010 23:17 Aan: 'rsync@lists.samba.org' Onderwerp: RE: Rsync / ssh high cpu load Hi again, Still working on this one. When I rsync/ssh a big file to my server over the local network (100Mbit/s) with the --bwlimit=1000 I only have a small CPU load. SSH takes about 10% and rsync approximately 0-5%. In the end the transfer speed usually is around the value I set in bwlimit. When I transfer the same file to a remote machine, where the bandwidth is limited to roughly 140KB/s and the bandwidth limit is set to something below this value I get around the same values which I have on the local network, SSH 10% and rsync somewhere between 0-5%. If I change the bwlimit value to 150KB/s or higher the SSH load goes up to 70% and rsync to 30% fully consuming my CPU. I think the problem is pretty clear now. Does anybody have an idea, what the next step could be to the resolution of this ? Regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: woensdag 17 februari 2010 12:17 Aan: 'rsync@lists.samba.org' Onderwerp: RE: Rsync / ssh high cpu load Hi, In an attempt to narrow this down a bit I installed a RedHat machine with openssh-5.3p1 and rsync-3.0.7 and did the same test as below. Transferring data, both up and downstream, are not giving me any CPU load. Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. I will mention this in the cygwin list. Still, if anybody has any ideas how to solve the issue below, please let me know. Warm regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: dinsdag 16 februari 2010 17:31 Aan: 'rsync@lists.samba.org' Onderwerp: Rsync / ssh high cpu load Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync / ssh high cpu load
Hi, In an attempt to narrow this down a bit I installed a RedHat machine with openssh-5.3p1 and rsync-3.0.7 and did the same test as below. Transferring data, both up and downstream, are not giving me any CPU load. Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. I will mention this in the cygwin list. Still, if anybody has any ideas how to solve the issue below, please let me know. Warm regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: dinsdag 16 februari 2010 17:31 Aan: 'rsync@lists.samba.org' Onderwerp: Rsync / ssh high cpu load Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync / ssh high cpu load
Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. When using rsync over SSH you may find that the CPU load is substantial depending upon multiple factors including but not limited to your processor, disks and network. Hopefully, someone on the cygwin mailing list will be able to provide further details. However, high CPU load for when using rsync over SSH is not something that I would be overly surprised about. One possibility would be to nice the process. - This email is protected by LBackup, an open source backup solution. Free as in freedom; LBackup is licensed under the GNU GPL http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync / ssh high cpu load
Hi, On Thu, 18 Feb 2010, Henri Shustak wrote: Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. When using rsync over SSH you may find that the CPU load is substantial depending upon multiple factors including but not limited to your processor, disks and network. Hopefully, someone on the cygwin mailing list will be able to provide further details. However, high CPU load for when using rsync over SSH is not something that I would be overly surprised about. One possibility would be to nice the process. This would not help if the high CPU load is a result of i/o wait. Viele Gruesse Eberhard Moenkeberg (emoe...@gwdg.de, e...@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoe...@gwdg.de Tel.: +49 (0)551 201-1551 - Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL:http://www.gwdg.de E-Mail: g...@gwdg.de Tel.: +49 (0)551 201-1510Fax:+49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Bernhard Neumair Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 - -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync / ssh high cpu load
Hi again, Still working on this one. When I rsync/ssh a big file to my server over the local network (100Mbit/s) with the --bwlimit=1000 I only have a small CPU load. SSH takes about 10% and rsync approximately 0-5%. In the end the transfer speed usually is around the value I set in bwlimit. When I transfer the same file to a remote machine, where the bandwidth is limited to roughly 140KB/s and the bandwidth limit is set to something below this value I get around the same values which I have on the local network, SSH 10% and rsync somewhere between 0-5%. If I change the bwlimit value to 150KB/s or higher the SSH load goes up to 70% and rsync to 30% fully consuming my CPU. I think the problem is pretty clear now. Does anybody have an idea, what the next step could be to the resolution of this ? Regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: woensdag 17 februari 2010 12:17 Aan: 'rsync@lists.samba.org' Onderwerp: RE: Rsync / ssh high cpu load Hi, In an attempt to narrow this down a bit I installed a RedHat machine with openssh-5.3p1 and rsync-3.0.7 and did the same test as below. Transferring data, both up and downstream, are not giving me any CPU load. Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. I will mention this in the cygwin list. Still, if anybody has any ideas how to solve the issue below, please let me know. Warm regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: dinsdag 16 februari 2010 17:31 Aan: 'rsync@lists.samba.org' Onderwerp: Rsync / ssh high cpu load Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
