Issue with spaces and strange characters (è)
Hi All, I have been looking at this thing for a while now and I think it is a bug in rsync 3.1.2 version 31. But before I log a defect, I need some confirmation. My rsync solution is running in Cygwin on my Windows 10 PC to my linux server. If required, I am rsyncing my files back onto my PC. The directory path contains spaces in the name and the file name contains strange characters (è), for example: F:\Directory Name\What is wereldorièntatie.doc If I rsync this to the server without the option -s everything works as expected. I can also rsync it back to my pc as long as I do not use the option -s. It will not work in both directions if I do use the -s option. Now I have the following file, note I replaced the è for a normal e. F:\Directory Name\What is wereldorientatie.doc For syncing this file I do need the -s option, otherwise it breaks. Obviously this behavior is not consistent and breaking my script, if I want to restore my files. I have tons of debugging for this, if required I can send this on the list, but I think this small description might cover it as well. Can anybody say something usefull about it ? Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync / ssh high cpu load
Hi, Today I did another interesting test in an attempt to resolve this issue. With Microsoft Process Explorer (downloadable from the Microsoft website) it is possible to 'pause' and 'resume' processes. When I start a rsync/ssh session to my server and pause the SSH process. The rsync process consumes 100% CPU, while the IO is 0. In the case I pause rsync and leave SSH running, there is no CPU load and no IO. In the first case rsync cannot send any data into the SSH tunnel. I think rsync goes into a CPU consuming loop while it waits for access to the SSH tunnel. This is somehow the same as when rsync tries to send data with SSH and it has to wait because it is exceeding the bwlimit variable. The next step for me would be to dive into the rsync code, which is something I really try to avoid at this moment. Any volunteers ? Regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: woensdag 17 februari 2010 23:17 Aan: 'rsync@lists.samba.org' Onderwerp: RE: Rsync / ssh high cpu load Hi again, Still working on this one. When I rsync/ssh a big file to my server over the local network (100Mbit/s) with the --bwlimit=1000 I only have a small CPU load. SSH takes about 10% and rsync approximately 0-5%. In the end the transfer speed usually is around the value I set in bwlimit. When I transfer the same file to a remote machine, where the bandwidth is limited to roughly 140KB/s and the bandwidth limit is set to something below this value I get around the same values which I have on the local network, SSH 10% and rsync somewhere between 0-5%. If I change the bwlimit value to 150KB/s or higher the SSH load goes up to 70% and rsync to 30% fully consuming my CPU. I think the problem is pretty clear now. Does anybody have an idea, what the next step could be to the resolution of this ? Regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: woensdag 17 februari 2010 12:17 Aan: 'rsync@lists.samba.org' Onderwerp: RE: Rsync / ssh high cpu load Hi, In an attempt to narrow this down a bit I installed a RedHat machine with openssh-5.3p1 and rsync-3.0.7 and did the same test as below. Transferring data, both up and downstream, are not giving me any CPU load. Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. I will mention this in the cygwin list. Still, if anybody has any ideas how to solve the issue below, please let me know. Warm regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: dinsdag 16 februari 2010 17:31 Aan: 'rsync@lists.samba.org' Onderwerp: Rsync / ssh high cpu load Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync / ssh high cpu load
Hi, In an attempt to narrow this down a bit I installed a RedHat machine with openssh-5.3p1 and rsync-3.0.7 and did the same test as below. Transferring data, both up and downstream, are not giving me any CPU load. Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. I will mention this in the cygwin list. Still, if anybody has any ideas how to solve the issue below, please let me know. Warm regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: dinsdag 16 februari 2010 17:31 Aan: 'rsync@lists.samba.org' Onderwerp: Rsync / ssh high cpu load Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync / ssh high cpu load
Hi again, Still working on this one. When I rsync/ssh a big file to my server over the local network (100Mbit/s) with the --bwlimit=1000 I only have a small CPU load. SSH takes about 10% and rsync approximately 0-5%. In the end the transfer speed usually is around the value I set in bwlimit. When I transfer the same file to a remote machine, where the bandwidth is limited to roughly 140KB/s and the bandwidth limit is set to something below this value I get around the same values which I have on the local network, SSH 10% and rsync somewhere between 0-5%. If I change the bwlimit value to 150KB/s or higher the SSH load goes up to 70% and rsync to 30% fully consuming my CPU. I think the problem is pretty clear now. Does anybody have an idea, what the next step could be to the resolution of this ? Regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: woensdag 17 februari 2010 12:17 Aan: 'rsync@lists.samba.org' Onderwerp: RE: Rsync / ssh high cpu load Hi, In an attempt to narrow this down a bit I installed a RedHat machine with openssh-5.3p1 and rsync-3.0.7 and did the same test as below. Transferring data, both up and downstream, are not giving me any CPU load. Although there is a small version difference in the rsync versions, I would say that this is caused by the cygwin layer. I will mention this in the cygwin list. Still, if anybody has any ideas how to solve the issue below, please let me know. Warm regards, Jan -Oorspronkelijk bericht- Van: Jan Alphenaar [mailto:jan.alphen...@dotcolour.com] Verzonden: dinsdag 16 februari 2010 17:31 Aan: 'rsync@lists.samba.org' Onderwerp: Rsync / ssh high cpu load Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync / ssh high cpu load
Hi everybody, I am using Cygwin rsync/ssh to synchronize my files from my home computer to my server. I raised a question on the Cygwin list as well, but thought this list would be appropriate too. The issue is that the CPU on the sending side is fully allocated by rsync/ssh (both taking 50%) during a file transfer. This is the command I use: rsync -e ssh * user@server:/data After some more testing on this I figured out that rsync/ssh is only taking the full cpu if the bandwidth allocated to the session exceeds the true bandwidth available on the link. For example, if the uplink is 128KB/s and I pass in the --bwlimit=24 parameter the cpu is around 0%. Even --bw-limit=126 does not take 100% cpu, but --bwlimit=132 does. Note this is the first install, the /data directory on the receiving side is empty. Would this be because of packet loss on the link and packets have to be retransmitted ? I will do a quick test tonight with rsync/ssh between two linux boxes (without cygwin), see what that gives me. Note that I am using these versions: Cygwin: 1.7.1-1 ssh: OpenSSH_5.3p1 rsync: 3.0.6 Has anybody seen this before ? Thanks in advance. Regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync spawning a child process when pulling files ?
Hi, I am using rsync 3.0.4 on a WinXP machine (with CygWin) to synchronize my files to a RedHat Linux server. Everything is working perfectly, but there is one thing I do not understand. Apparently there is a difference in pushing and pulling files with rsync. When I push my files to the server with this command, I see one rsync process on my WinXP machine, just as expected rsync -rlt -e ssh -T Test.pst user@ip address:/data But when I pull my files from the server to my WinXP machine with this command, I see two rsync processes on my WinXP machine. rsync -rlt -e ssh -T user@ip address:/data/Test.pst . Can anybody confirm that this is the default behavior ? If yes, what would be the reason for spawning an extra child rsync process ? Any information is appreciated. Warm regards, Jan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
