Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Ken, Thank you for the reply, this is great news! Looking forward to speedier rsyncs with Cygwin. Keith On Thu, Sep 16, 2021, 16:49 Ken Brown via Cygwin wrote: > On 9/16/2021 6:00 PM, Keith Christian via Cygwin wrote: > > I've been following his thread with interest both here and on the Cygwin > > developers list. I, too rsync between Cygwin and Linux machines. > > > > Lots of good debate showing the genius of the folks that support the > cygwin > > infrastructure. > > > > Looks like the lively discussion on both lists has stopped. What is the > > final consensus? > > > > Should we look for a new updates to the cygwin dll, cygrunsrv, ssh, > rsync, > > etc? > > All that's needed is an update to the cygwin package. There should be a > test > release coming very soon, possibly tomorrow if nothing unexpected shows up. > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 9/16/2021 6:00 PM, Keith Christian via Cygwin wrote: I've been following his thread with interest both here and on the Cygwin developers list. I, too rsync between Cygwin and Linux machines. Lots of good debate showing the genius of the folks that support the cygwin infrastructure. Looks like the lively discussion on both lists has stopped. What is the final consensus? Should we look for a new updates to the cygwin dll, cygrunsrv, ssh, rsync, etc? All that's needed is an update to the cygwin package. There should be a test release coming very soon, possibly tomorrow if nothing unexpected shows up. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I've been following his thread with interest both here and on the Cygwin developers list. I, too rsync between Cygwin and Linux machines. Lots of good debate showing the genius of the folks that support the cygwin infrastructure. Looks like the lively discussion on both lists has stopped. What is the final consensus? Should we look for a new updates to the cygwin dll, cygrunsrv, ssh, rsync, etc? -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
05.09.2021 17:11, Brian Inglis: The suggestion was intended as a tip to ensure *complete* locally rebuilt package contents are installed, Setup has its "from_cwd" installation mode for precisely that reason: installing a local package without the need to create a full install hierarchy and setup.ini files. Due to Windows "improvements", my system upgraded a few years ago is just as "fast" as the 10 year old system it replaced. ;^> Cygwin Setup upgrades can take as long as Windows Update installing the latest patches. If you haven't already, put your Cygwin installation on SSD and exclude the Cygwin root (and local mirror directory) from real-time virus scans. I do monthly updates on slow machines (CPU is a 1.8GHz IvyBridge Celeron) and that takes around two to five minutes depending on how large the update is in any given month (the man-db update happens in the background, so is not part of these figures). -- Achim. (on the road :-) -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 2021-09-05 02:18, Achim Gratz wrote: 04.09.2021 18:45, Brian Inglis: [...] then to install all binary packages for dogfooding: Would you please stop telling folks to do things that potentially breaks their systems? There are quite a few more steps to take if you want to emulate what setup does. Then again you cannot do that on a live Cygwin system unless you _really_ know what exactly it is you're doing and which packages can or cannot be touched that way. This is a downside of having only a general and not also a "technical" mailing list to discuss and diagnose "technical" issues and approaches. The suggestion was intended as a tip to ensure *complete* locally rebuilt package contents are installed, by folks who are already having issues with their installation, rebuilding Cygwin and packages, and replacing the Cygwin dll etc. and packages to diagnose those issues. So hopefully they are also aware of preremove and postinstall scripts, in packages where they are provided, and the /etc/setup/PKG.lst.gz manifests for files to be removed before installation, and provided after installation. For the latter I would expect technical folks to make use of them but leave them to Cygwin Setup programs for officially released package upgrades. [Also some of us do not have the appropriate technical background and expertise to get a Cygwin overlay mirror working reliably from the available instructions (if it was easy enough, someone would have provided a local-overlay-mirror package to do it years ago), nor systems fast enough to run Cygwin Setup for every package install. Due to Windows "improvements", my system upgraded a few years ago is just as "fast" as the 10 year old system it replaced. ;^> Cygwin Setup upgrades can take as long as Windows Update installing the latest patches. Some of my approaches, suggestions, and participation here are coloured by those limitations (also by expensive, slow, uncompetitive networks).] -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.] -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
04.09.2021 18:45, Brian Inglis: [...] then to install all binary packages for dogfooding: Would you please stop telling folks to do things that potentially breaks their systems? There are quite a few more steps to take if you want to emulate what setup does. Then again you cannot do that on a live Cygwin system unless you _really_ know what exactly it is you're doing and which packages can or cannot be touched that way. -- Achim. (on the road :-) -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 2021-09-03 14:59, Chris Roehrig wrote: On Fri Sep 3 2021, at 12:55 PM, Corinna Vinschen wrote: On Sep 2 12:03, Chris Roehrig wrote: On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin wrote: On 9/1/2021 5:11 PM, Chris Roehrig wrote: I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch. Maybe, but it could also be a Cygwin bug. I'll do a bisection of the Cygwin sources to see if I can track it down. I did some more tests and it still doesn't completely work: procps -ef # works procps -eo user,stime,tty,time,args # works procps -eo pid # fails with that same mmap() error I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args). Since you're building Cygwin by yourself anyway, can you do me a favor and try this: $ git revert 2f05de4dbf9c and see if that fixes your issue? I got procps working I think (both with and without the revert). I think the problem might just be that I wasn't also copying the rebuilt /bin/cygprocps-8.dll to match the /bin/procps.exe. There's some tricky renaming that make install does, so I did 'make install DESTDIR=/tmp/install' and copied just those two files. I'm guessing it would all work on a properly fully installed build. If you are building packages in the same way as scallywag with: $ cygport PKG.cygport download all check then to install all binary packages for dogfooding: $ for tar in PKG-VER-1.ARCH/dist/PKG/**/*PKG*-VER.tar.*z* do tar -x -C / -f $tar # * -C / installs usr/... under /usr/ done for example, I just did: $ for tar in \ dash-0.5.11.5-1.x86_64/dist/dash/**/*dash*-0.5.11.5-1.tar.*z* do tar -x -C / -f $tar done -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.] -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Am 03.09.2021 um 22:59 schrieb Chris Roehrig: I got procps working I think (both with and without the revert). That likely wasn't what Corinna wanted to know, though. Please re-install the procps-ng, cygwin and cygwin-devel packages from setup (and revert any other alterations you may have made). Then check that the error has gone away, rebuild and install the new cygwin1.dll with and without the revert mentioned by Corinna and check for the error under these two conditions as well. -- Achim. (on the road :-) -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Fri Sep 3 2021, at 12:55 PM, Corinna Vinschen wrote: > [resent, this time with the ML in To] > > On Sep 2 12:03, Chris Roehrig wrote: >> >> On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin >> wrote: >>> On 9/1/2021 5:11 PM, Chris Roehrig wrote: I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch. >>> >>> Maybe, but it could also be a Cygwin bug. I'll do a bisection of the >>> Cygwin sources to see if I can track it down. >> >> I did some more tests and it still doesn't completely work: >> procps -ef # works >> procps -eo user,stime,tty,time,args # works >> procps -eo pid # fails with that same mmap() error >> >> I also rebuilt it all using cygport and it gives the same error >> (pscommand.exe with no args). > > Since you're building Cygwin by yourself anyway, can you do me a favor > and try this: > > $ git revert 2f05de4dbf9c > > and see if that fixes your issue? I got procps working I think (both with and without the revert). I think the problem might just be that I wasn't also copying the rebuilt /bin/cygprocps-8.dll to match the /bin/procps.exe.There's some tricky renaming that make install does, so I did 'make install DESTDIR=/tmp/install' and copied just those two files. I'm guessing it would all work on a properly fully installed build. -- Chris -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
[resent, this time with the ML in To] On Sep 2 12:03, Chris Roehrig wrote: > > On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin wrote: > > On 9/1/2021 5:11 PM, Chris Roehrig wrote: > >> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so > >> maybe the stock procps package is incompatible with the current master > >> branch. > > > > Maybe, but it could also be a Cygwin bug. I'll do a bisection of the > > Cygwin sources to see if I can track it down. > > I did some more tests and it still doesn't completely work: > procps -ef # works > procps -eo user,stime,tty,time,args # works > procps -eo pid # fails with that same mmap() error > > I also rebuilt it all using cygport and it gives the same error > (pscommand.exe with no args). Since you're building Cygwin by yourself anyway, can you do me a favor and try this: $ git revert 2f05de4dbf9c and see if that fixes your issue? Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 9/2/2021 3:03 PM, Chris Roehrig wrote: On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin wrote: On 9/1/2021 5:11 PM, Chris Roehrig wrote: I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch. Maybe, but it could also be a Cygwin bug. I'll do a bisection of the Cygwin sources to see if I can track it down. I did some more tests and it still doesn't completely work: procps -ef # works procps -eo user,stime,tty,time,args # works procps -eo pid # fails with that same mmap() error I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args). I'm still seeing the error too, and I'm even seeing it if I rebuild Cygwin 3.2.0 with the current gcc. I don't know what the problem is. P.S. The custom on this list is not to top-post. Thanks. Sorry, I didn't realize there was an etiquette.Fixed. Thanks! Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin wrote: > On 9/1/2021 5:11 PM, Chris Roehrig wrote: >> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe >> the stock procps package is incompatible with the current master branch. > > Maybe, but it could also be a Cygwin bug. I'll do a bisection of the Cygwin > sources to see if I can track it down. I did some more tests and it still doesn't completely work: procps -ef # works procps -eo user,stime,tty,time,args # works procps -eo pid # fails with that same mmap() error I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args). > P.S. The custom on this list is not to top-post. Thanks. Sorry, I didn't realize there was an etiquette.Fixed. -- Chris -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 9/1/2021 5:11 PM, Chris Roehrig wrote: I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatibility with the current master branch. Maybe, but it could also be a Cygwin bug. I'll do a bisection of the Cygwin sources to see if I can track it down. Ken P.S. The custom on this list is not to top-post. Thanks. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatibility with the current master branch. (However, I built it against the stock /usr/include, not the current branch...) I first needed to 'make /proc/libprocps.la', and there was an undefined siginfo_t.si_int in ./lib/test_process.c which I just commented-out to get it to build. -- Chris On Tue Aug 31 2021, at 1:23 PM, Chris Roehrig wrote: > I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed > get 100MB/s transfers using both rsync and scp (without pipe_byte). > (It turns out last time I forgot 'make install' -- Doh!) > > I still get the procps error however. > > > On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig wrote: > >> Thanks, I did some more tests: >> scp also shows no improvement with topic/pipe.I tried running >> cygsshd with CYGWIN=pipe_byte as well as empty (in the registry >> HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), >> using net stop cygsshd + net start cygsshd to restart it. Not sure if that >> CYGWIN only applies to cygsshd or to all cygwin tasks. >> >> I get the procps error just running 'procps' command with no args from >> mintty (/usr/bin/procps). Looking at the procps-ng source (procps-ng >> 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears >> to be a mmap() failure. >> >> I'm wondering if I built/installed cygwin1.dll correctly because I don't get >> the procps error using the latest stock cygwin1.dll as installed by >> setup_x86_64. >> >> -- Chris >> >> >> >> On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin >> wrote: >> >>> On 8/30/2021 7:58 PM, Chris Roehrig wrote: I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT): 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. 2. I get the following error from procps: procps:ps/output.c:2195: please report this bug (I also get this using the main branch build). I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first). >>> >>> Thanks for testing. >>> >>> We're still working on this. (The discussion has moved to the >>> cygwin-developers mailing list.) I'll let you know when it's stable, and >>> maybe you can try again. >>> >>> As to the procps error, can you give details so that someone can try to >>> reproduce it and debug it? >>> >>> Ken >>> >>> -- >>> Problem reports: https://cygwin.com/problems.html >>> FAQ: https://cygwin.com/faq/ >>> Documentation:https://cygwin.com/docs.html >>> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple >> >> >> -- >> Problem reports: https://cygwin.com/problems.html >> FAQ: https://cygwin.com/faq/ >> Documentation:https://cygwin.com/docs.html >> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I've found you must also copy the matching cygwin-console-helper.exe for everything to work correctly! On 2021-08-31 14:23, Chris Roehrig wrote: I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed get 100MB/s transfers using both rsync and scp (without pipe_byte). (It turns out last time I forgot 'make install' -- Doh!) I still get the procps error however. On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig wrote: Thanks, I did some more tests: scp also shows no improvement with topic/pipe.I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it. Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks. I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps). Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure. I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_ On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin wrote: On 8/30/2021 7:58 PM, Chris Roehrig wrote: I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT): 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. 2. I get the following error from procps: procps:ps/output.c:2195: please report this bug (I also get this using the main branch build). I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first). Thanks for testing. We're still working on this. (The discussion has moved to the cygwin-developers mailing list.) I'll let you know when it's stable, and maybe you can try again. As to the procps error, can you give details so that someone can try to reproduce it and debug it? -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.] -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed get 100MB/s transfers using both rsync and scp (without pipe_byte). (It turns out last time I forgot 'make install' -- Doh!) I still get the procps error however. On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig wrote: > Thanks, I did some more tests: > scp also shows no improvement with topic/pipe.I tried running > cygsshd with CYGWIN=pipe_byte as well as empty (in the registry > HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), > using net stop cygsshd + net start cygsshd to restart it. Not sure if that > CYGWIN only applies to cygsshd or to all cygwin tasks. > > I get the procps error just running 'procps' command with no args from mintty > (/usr/bin/procps). Looking at the procps-ng source (procps-ng 3.3.17-1 is > what setup_x86_64 reports as my installed package), it appears to be a mmap() > failure. > > I'm wondering if I built/installed cygwin1.dll correctly because I don't get > the procps error using the latest stock cygwin1.dll as installed by > setup_x86_64. > > -- Chris > > > > On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin > wrote: > >> On 8/30/2021 7:58 PM, Chris Roehrig wrote: >>> I got it to build and tried out the topic/pipe branch (checked out on >>> Monday around 4:30pm PDT): >>> 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. >>> 2. I get the following error from procps: procps:ps/output.c:2195: >>> please report this bug >>> (I also get this using the main branch build). >>> I first updated my cygwin normally using setup-x86_64.exe, then built and >>> copied only the cygwin1.dll into /bin from the install/bin directory >>> (quitting all cygwin tasks first). >> >> Thanks for testing. >> >> We're still working on this. (The discussion has moved to the >> cygwin-developers mailing list.) I'll let you know when it's stable, and >> maybe you can try again. >> >> As to the procps error, can you give details so that someone can try to >> reproduce it and debug it? >> >> Ken >> >> -- >> Problem reports: https://cygwin.com/problems.html >> FAQ: https://cygwin.com/faq/ >> Documentation:https://cygwin.com/docs.html >> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Thanks, I did some more tests: scp also shows no improvement with topic/pipe.I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it. Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks. I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps). Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure. I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_x86_64. -- Chris On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin wrote: > On 8/30/2021 7:58 PM, Chris Roehrig wrote: >> I got it to build and tried out the topic/pipe branch (checked out on Monday >> around 4:30pm PDT): >> 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. >> 2. I get the following error from procps: procps:ps/output.c:2195: please >> report this bug >> (I also get this using the main branch build). >> I first updated my cygwin normally using setup-x86_64.exe, then built and >> copied only the cygwin1.dll into /bin from the install/bin directory >> (quitting all cygwin tasks first). > > Thanks for testing. > > We're still working on this. (The discussion has moved to the > cygwin-developers mailing list.) I'll let you know when it's stable, and > maybe you can try again. > > As to the procps error, can you give details so that someone can try to > reproduce it and debug it? > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/30/2021 7:58 PM, Chris Roehrig wrote: I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT): 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. 2. I get the following error from procps: procps:ps/output.c:2195: please report this bug (I also get this using the main branch build). I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first). Thanks for testing. We're still working on this. (The discussion has moved to the cygwin-developers mailing list.) I'll let you know when it's stable, and maybe you can try again. As to the procps error, can you give details so that someone can try to reproduce it and debug it? Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT): 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. 2. I get the following error from procps: procps:ps/output.c:2195: please report this bug (I also get this using the main branch build). I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first). On Sun Aug 29 2021, at 3:24 PM, Ken Brown via Cygwin wrote: > On 8/29/2021 3:24 PM, Chris Roehrig wrote: >> I'd be happy to test it out if you can give me some commands to git it and >> build it to replace my existing stock Cygwin installation. >> (Or it is just the cygwin.dll I'd need to replace?) >> My daily backup scripts use a lot of pipes and named pipes. > > It's just cygwin1.dll that you'd need to replace. Instructions for building > the latter can be found at > > https://www.cygwin.com/faq.html#faq.programming.building-cygwin > > but they're out of date in two respects: > > 1. Currently the build does *not* ignore errors in building the documentation. > > 2. In addition to the packages listed in the FAQ, you'll need the following: > > texlive-collection-latexrecommended > texlive-collection-fontsrecommended > texlive-collection-pictures > python38-lxml > python38-ply > > These instructions will produce cygwin1.dll in the bin subdirectory of your > install directory, built from the git master branch. > > If you want to try the topic/pipe branch, just go to the newlib-cygwin git > repo that you cloned, issue the command 'git checkout topic/pipe', and > rebuild. > > I'll be glad to help if you run into problems. > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sun, 29 Aug 2021 22:15:29 -0400
Ken Brown wrote:
> On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> > On Mon, 30 Aug 2021 09:13:14 +0900
> > Takashi Yano wrote:
> >> On Sun, 29 Aug 2021 17:04:56 -0400
> >> Ken Brown wrote:
> >>> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 18:41:02 +0900
> Takashi Yano wrote:
> > On Sat, 28 Aug 2021 10:43:27 +0200
> > Corinna Vinschen wrote:
> >> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> >>> On Fri, 27 Aug 2021 12:00:50 -0400
> >>> Ken Brown wrote:
> Two years ago I thought I needed nt_create to avoid problems when
> calling
> set_pipe_non_blocking. Are you saying that's not an issue? Is
> set_pipe_non_blocking unnecessary? Is that the point of your
> modification to
> raw_read?
> >>>
> >>> Yes. Instead of making windows read function itself non-blocking,
> >>> it is possible to check if the pipe can be read before read using
> >>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> >>> returned.
> >>
> >> The problem is this:
> >>
> >> if (PeekNamedPipe())
> >> ReadFile(blocking);
> >>
> >> is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another
> >> thread from draining the pipe between the PeekNamedPipe and the
> >> ReadFile
> >> call. And as soon as ReadFile runs, it hangs indefinitely and we can't
> >> stop it via a signal.
> >
> > Hmm, you are right. Mutex guard seems to be necessary like pty code
> > if we go this way.
>
> I have found that set_pipe_non_blocking() succeeds for both read and
> write pipes if the write pipe is created by CreateNamedPipe() and the
> read pipe is created by CreateFile() contrary to the current create()
> code. Therefore, not only nt_create() but also PeekNamedPipe() become
> unnecessary.
>
> Please see the revised patch attached.
> >>>
> >>> I haven't had a chance to test this myself yet, but occurs to me that we
> >>> might
> >>> have a different problem after this patch: Does the write handle that we
> >>> get
> >>> from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
> >>
> >> I have just checked this, and the answer is "No". Due to this problem,
> >> NtQueryInformationFile() call in select() fails on the write pipe.
> >>
> >> It seems that we need more consideration...
> >
> > We have two easy options:
> > 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
> > 2) Use nt_create() again and forget C# program issue.
>
> I vote for 2), but let's see what Corinna thinks.
BTW. what's wrong if just:
static int
nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
DWORD psize, int64_t *unique_id)
{
if (r && w)
{
static volatile ULONG pipe_unique_id;
LONG id = InterlockedIncrement ((LONG *) _unique_id);
if (unique_id)
*unique_id = ((int64_t) id << 32 | GetCurrentProcessId ());
if (!CreatePipe (r, w, sa_ptr, psize))
{
*r = *w = NULL;
return GetLastError ();
}
}
return 0;
}
?
In my environment, I cannot find any defects.
- No performance degradation.
- set_pipe_non_blocking() works for both read and write pipes.
- NtQueryInformationFile() in select() works for both r/w pipes.
- Piping C# program works.
Is naming the pipe really necessary?
--
Takashi Yano
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
On Mon, 30 Aug 2021 09:13:14 +0900
Takashi Yano wrote:
On Sun, 29 Aug 2021 17:04:56 -0400
Ken Brown wrote:
On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:
On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:
On Aug 28 02:21, Takashi Yano via Cygwin wrote:
On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:
Two years ago I thought I needed nt_create to avoid problems when calling
set_pipe_non_blocking. Are you saying that's not an issue? Is
set_pipe_non_blocking unnecessary? Is that the point of your modification to
raw_read?
Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.
The problem is this:
if (PeekNamedPipe())
ReadFile(blocking);
is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call. And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.
Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.
I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.
Please see the revised patch attached.
I haven't had a chance to test this myself yet, but occurs to me that we might
have a different problem after this patch: Does the write handle that we get
from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
I have just checked this, and the answer is "No". Due to this problem,
NtQueryInformationFile() call in select() fails on the write pipe.
It seems that we need more consideration...
We have two easy options:
1) Configure the pipe with PIPE_ACCESS_DUPLEX.
2) Use nt_create() again and forget C# program issue.
I vote for 2), but let's see what Corinna thinks.
Even without this problem, select() for writing pipe has a bug
and does not wrok as expected. The following patch seems to be
needed.
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0..ac2fd227e 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h,
bool writing)
that. This means that a pipe could still block since you could
be trying to write more to the pipe than is available in the
buffer but that is the hazard of select(). */
- fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
if (fpli.WriteQuotaAvailable > 0)
{
paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
I agree.
Ken
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Mon, 30 Aug 2021 09:13:14 +0900
Takashi Yano wrote:
> On Sun, 29 Aug 2021 17:04:56 -0400
> Ken Brown wrote:
> > On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > > On Sat, 28 Aug 2021 18:41:02 +0900
> > > Takashi Yano wrote:
> > >> On Sat, 28 Aug 2021 10:43:27 +0200
> > >> Corinna Vinschen wrote:
> > >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > On Fri, 27 Aug 2021 12:00:50 -0400
> > Ken Brown wrote:
> > > Two years ago I thought I needed nt_create to avoid problems when
> > > calling
> > > set_pipe_non_blocking. Are you saying that's not an issue? Is
> > > set_pipe_non_blocking unnecessary? Is that the point of your
> > > modification to
> > > raw_read?
> >
> > Yes. Instead of making windows read function itself non-blocking,
> > it is possible to check if the pipe can be read before read using
> > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > returned.
> > >>>
> > >>> The problem is this:
> > >>>
> > >>>if (PeekNamedPipe())
> > >>> ReadFile(blocking);
> > >>>
> > >>> is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another
> > >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > >>> call. And as soon as ReadFile runs, it hangs indefinitely and we can't
> > >>> stop it via a signal.
> > >>
> > >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> > >> if we go this way.
> > >
> > > I have found that set_pipe_non_blocking() succeeds for both read and
> > > write pipes if the write pipe is created by CreateNamedPipe() and the
> > > read pipe is created by CreateFile() contrary to the current create()
> > > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > > unnecessary.
> > >
> > > Please see the revised patch attached.
> >
> > I haven't had a chance to test this myself yet, but occurs to me that we
> > might
> > have a different problem after this patch: Does the write handle that we
> > get
> > from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
>
> I have just checked this, and the answer is "No". Due to this problem,
> NtQueryInformationFile() call in select() fails on the write pipe.
>
> It seems that we need more consideration...
We have two easy options:
1) Configure the pipe with PIPE_ACCESS_DUPLEX.
2) Use nt_create() again and forget C# program issue.
Even without this problem, select() for writing pipe has a bug
and does not wrok as expected. The following patch seems to be
needed.
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0..ac2fd227e 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h,
bool writing)
that. This means that a pipe could still block since you could
be trying to write more to the pipe than is available in the
buffer but that is the hazard of select(). */
- fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
if (fpli.WriteQuotaAvailable > 0)
{
paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
--
Takashi Yano
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sun, 29 Aug 2021 17:04:56 -0400 Ken Brown wrote: > On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: > > On Sat, 28 Aug 2021 18:41:02 +0900 > > Takashi Yano wrote: > >> On Sat, 28 Aug 2021 10:43:27 +0200 > >> Corinna Vinschen wrote: > >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote: > On Fri, 27 Aug 2021 12:00:50 -0400 > Ken Brown wrote: > > Two years ago I thought I needed nt_create to avoid problems when > > calling > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > set_pipe_non_blocking unnecessary? Is that the point of your > > modification to > > raw_read? > > Yes. Instead of making windows read function itself non-blocking, > it is possible to check if the pipe can be read before read using > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > returned. > >>> > >>> The problem is this: > >>> > >>>if (PeekNamedPipe()) > >>> ReadFile(blocking); > >>> > >>> is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile > >>> call. And as soon as ReadFile runs, it hangs indefinitely and we can't > >>> stop it via a signal. > >> > >> Hmm, you are right. Mutex guard seems to be necessary like pty code > >> if we go this way. > > > > I have found that set_pipe_non_blocking() succeeds for both read and > > write pipes if the write pipe is created by CreateNamedPipe() and the > > read pipe is created by CreateFile() contrary to the current create() > > code. Therefore, not only nt_create() but also PeekNamedPipe() become > > unnecessary. > > > > Please see the revised patch attached. > > I haven't had a chance to test this myself yet, but occurs to me that we > might > have a different problem after this patch: Does the write handle that we get > from CreateNamedPipe() have FILE_READ_ATTRIBUTES access? I have just checked this, and the answer is "No". Due to this problem, NtQueryInformationFile() call in select() fails on the write pipe. It seems that we need more consideration... -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/29/2021 4:41 AM, Takashi Yano wrote: Hi Ken, On Sat, 28 Aug 2021 16:55:52 -0400 Ken Brown wrote: On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 13:58:08 +0200 Corinna Vinschen wrote: On Aug 28 18:41, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 10:43:27 +0200 Corinna Vinschen wrote: On Aug 28 02:21, Takashi Yano via Cygwin wrote: On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Hmm, you are right. Mutex guard seems to be necessary like pty code if we go this way. Is a blocking ReadFile actually faster than a non-blocking read? Or does it mainly depend on BYTE vs. MESSAGE mode? Actually, I don't think so. Perhaps it is not essential problem of overlapped I/O but something is wrong with current pipe code. What if the pipe is created non-blocking and stays non-blocking all the time and uses BYTE mode all the time? Just as sockets, it would always only emulate blocking mode. Wouldn't that drop code size a lot and fix most problems? If 'non-blocking' means overlapped I/O, only the problem will be: https://cygwin.com/pipermail/cygwin/2021-March/247987.html Sorry if that wasn't clear, but I was not talking about overlapped I/O, which we should get rid off, but of real non-blocking mode, which Windows pipes are fortunately capable of. Do you mean, PIPE_NOWAIT flag? If this flags is specified in the read pipe, non-cygwin apps cannot read the pipe correctly. While waiting for Corinna's response to this, I have one more question. Do you understand why nt_create() failed and you had to revert to create()? Was it an access problem because nt_create requested FILE_WRITE_ATTRIBUTES? Or did I make some careless mistake in writing nt_create? I am sorry but no. I don't understand why piping C# program via the pipe created by nt_create() has the issue. I tried to change setup parameters in nt_create(), however, I did not succeed it to work. I also couldn't find any mistake in nt_create() so far. Win32 programs which use ReadFile() and WriteFile() work even with the pipe created by nt_create() as well as overlapped I/O. What does C# program differ from legacy win32 program at all? I don't know. By the way, when I introduced nt_create(), my preference would have been to simply change create() to use the NT API, but I was afraid to do that because I didn't want to take a chance on breaking something. That's still my preference, if we can find a way to work around this problem with C# programs. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/29/2021 3:24 PM, Chris Roehrig wrote: I'd be happy to test it out if you can give me some commands to git it and build it to replace my existing stock Cygwin installation. (Or it is just the cygwin.dll I'd need to replace?) My daily backup scripts use a lot of pipes and named pipes. It's just cygwin1.dll that you'd need to replace. Instructions for building the latter can be found at https://www.cygwin.com/faq.html#faq.programming.building-cygwin but they're out of date in two respects: 1. Currently the build does *not* ignore errors in building the documentation. 2. In addition to the packages listed in the FAQ, you'll need the following: texlive-collection-latexrecommended texlive-collection-fontsrecommended texlive-collection-pictures python38-lxml python38-ply These instructions will produce cygwin1.dll in the bin subdirectory of your install directory, built from the git master branch. If you want to try the topic/pipe branch, just go to the newlib-cygwin git repo that you cloned, issue the command 'git checkout topic/pipe', and rebuild. I'll be glad to help if you run into problems. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/29/2021 3:37 PM, Takashi Yano wrote: On Sun, 29 Aug 2021 11:57:04 -0400 Ken Brown wrote: On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 18:41:02 +0900 Takashi Yano wrote: On Sat, 28 Aug 2021 10:43:27 +0200 Corinna Vinschen wrote: On Aug 28 02:21, Takashi Yano via Cygwin wrote: On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Hmm, you are right. Mutex guard seems to be necessary like pty code if we go this way. I have found that set_pipe_non_blocking() succeeds for both read and write pipes if the write pipe is created by CreateNamedPipe() and the read pipe is created by CreateFile() contrary to the current create() code. Therefore, not only nt_create() but also PeekNamedPipe() become unnecessary. Please see the revised patch attached. That's a great idea. I've applied your two patches to the topic/pipe branch. I also rebased it and did a forced push in order to bring in Corinna's loader script fix. So you'll have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'. Does this now fix all known problems with pipes? Only the small thing remaining is pipe mode. If the pipe mode is changed to byte mode, the following issue will be solved. https://cygwin.com/pipermail/cygwin/2021-March/247987.html How about the simple patch attached? The comment in pipe code says: Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE. This *seems* to more closely mimic Linux pipe behavior and is definitely required for pty handling since fhandler_pty_master writes to the pipe in chunks, terminated by newline when CANON mode is specified. This mentions about pty behaiviour in canonical mode, however, the pty pipe is created as message mode even with this patch. Are there any other reasons that message mode is preferred for pipe? No idea. All I remember is that there was a lot of discussion around the time that it was decided to use PIPE_TYPE_MESSAGE by default. Corinna probably remembers the reasons. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 18:41:02 +0900 Takashi Yano wrote: On Sat, 28 Aug 2021 10:43:27 +0200 Corinna Vinschen wrote: On Aug 28 02:21, Takashi Yano via Cygwin wrote: On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Hmm, you are right. Mutex guard seems to be necessary like pty code if we go this way. I have found that set_pipe_non_blocking() succeeds for both read and write pipes if the write pipe is created by CreateNamedPipe() and the read pipe is created by CreateFile() contrary to the current create() code. Therefore, not only nt_create() but also PeekNamedPipe() become unnecessary. Please see the revised patch attached. I haven't had a chance to test this myself yet, but occurs to me that we might have a different problem after this patch: Does the write handle that we get from CreateNamedPipe() have FILE_READ_ATTRIBUTES access? Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sun, 29 Aug 2021 11:57:04 -0400 Ken Brown wrote: > On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: > > On Sat, 28 Aug 2021 18:41:02 +0900 > > Takashi Yano wrote: > >> On Sat, 28 Aug 2021 10:43:27 +0200 > >> Corinna Vinschen wrote: > >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote: > On Fri, 27 Aug 2021 12:00:50 -0400 > Ken Brown wrote: > > Two years ago I thought I needed nt_create to avoid problems when > > calling > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > set_pipe_non_blocking unnecessary? Is that the point of your > > modification to > > raw_read? > > Yes. Instead of making windows read function itself non-blocking, > it is possible to check if the pipe can be read before read using > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > returned. > >>> > >>> The problem is this: > >>> > >>>if (PeekNamedPipe()) > >>> ReadFile(blocking); > >>> > >>> is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile > >>> call. And as soon as ReadFile runs, it hangs indefinitely and we can't > >>> stop it via a signal. > >> > >> Hmm, you are right. Mutex guard seems to be necessary like pty code > >> if we go this way. > > > > I have found that set_pipe_non_blocking() succeeds for both read and > > write pipes if the write pipe is created by CreateNamedPipe() and the > > read pipe is created by CreateFile() contrary to the current create() > > code. Therefore, not only nt_create() but also PeekNamedPipe() become > > unnecessary. > > > > Please see the revised patch attached. > > That's a great idea. > > I've applied your two patches to the topic/pipe branch. I also rebased it > and > did a forced push in order to bring in Corinna's loader script fix. So > you'll > have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'. > > Does this now fix all known problems with pipes? Only the small thing remaining is pipe mode. If the pipe mode is changed to byte mode, the following issue will be solved. https://cygwin.com/pipermail/cygwin/2021-March/247987.html How about the simple patch attached? The comment in pipe code says: Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE. This *seems* to more closely mimic Linux pipe behavior and is definitely required for pty handling since fhandler_pty_master writes to the pipe in chunks, terminated by newline when CANON mode is specified. This mentions about pty behaiviour in canonical mode, however, the pty pipe is created as message mode even with this patch. Are there any other reasons that message mode is preferred for pipe? -- Takashi Yano 0003-Cygwin-pipe-Set-byte-mode-as-default-rather-than-mes.patch Description: Binary data -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I'd be happy to test it out if you can give me some commands to git it and build it to replace my existing stock Cygwin installation. (Or it is just the cygwin.dll I'd need to replace?) My daily backup scripts use a lot of pipes and named pipes. -- Chris On Sun Aug 29 2021, at 8:57 AM, Ken Brown via Cygwin wrote: > > Aside from that, I'm wondering how and when to merge the new pipe > implementation to master. It obviously needs much more widespread testing > than it's gotten so far. I'm a little nervous about it because I haven't > thought about the details for two years, and no one other than me has tested > it until a few days ago. > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 18:41:02 +0900 Takashi Yano wrote: On Sat, 28 Aug 2021 10:43:27 +0200 Corinna Vinschen wrote: On Aug 28 02:21, Takashi Yano via Cygwin wrote: On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Hmm, you are right. Mutex guard seems to be necessary like pty code if we go this way. I have found that set_pipe_non_blocking() succeeds for both read and write pipes if the write pipe is created by CreateNamedPipe() and the read pipe is created by CreateFile() contrary to the current create() code. Therefore, not only nt_create() but also PeekNamedPipe() become unnecessary. Please see the revised patch attached. That's a great idea. I've applied your two patches to the topic/pipe branch. I also rebased it and did a forced push in order to bring in Corinna's loader script fix. So you'll have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'. Does this now fix all known problems with pipes? Corinna, do you still see any benefit to switching to PIPE_NOWAIT? AFAICT, it wouldn't decrease the code size at this point, so the only question is whether it might improve performance. If you think it's worth trying, I'd be glad to code it up on a new branch, and we could compare the two. Aside from that, I'm wondering how and when to merge the new pipe implementation to master. It obviously needs much more widespread testing than it's gotten so far. I'm a little nervous about it because I haven't thought about the details for two years, and no one other than me has tested it until a few days ago. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Aug 29 00:43, Takashi Yano via Cygwin wrote: > On Sat, 28 Aug 2021 13:58:08 +0200 > Corinna Vinschen wrote: > > On Aug 28 18:41, Takashi Yano via Cygwin wrote: > > > On Sat, 28 Aug 2021 10:43:27 +0200 > > > Corinna Vinschen wrote: > > > > On Aug 28 02:21, Takashi Yano via Cygwin wrote: > > > > > On Fri, 27 Aug 2021 12:00:50 -0400 > > > > > Ken Brown wrote: > > > > > > Two years ago I thought I needed nt_create to avoid problems when > > > > > > calling > > > > > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > > > > > set_pipe_non_blocking unnecessary? Is that the point of your > > > > > > modification to > > > > > > raw_read? > > > > > > > > > > Yes. Instead of making windows read function itself non-blocking, > > > > > it is possible to check if the pipe can be read before read using > > > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > > > > returned. > > > > > > > > The problem is this: > > > > > > > > if (PeekNamedPipe()) > > > > ReadFile(blocking); > > > > > > > > is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > > > > thread from draining the pipe between the PeekNamedPipe and the ReadFile > > > > call. And as soon as ReadFile runs, it hangs indefinitely and we can't > > > > stop it via a signal. > > > > > > Hmm, you are right. Mutex guard seems to be necessary like pty code > > > if we go this way. > > > > > > > Is a blocking ReadFile actually faster than a non-blocking read? Or > > > > does it mainly depend on BYTE vs. MESSAGE mode? > > > > > > Actually, I don't think so. Perhaps it is not essential problem of > > > overlapped I/O but something is wrong with current pipe code. > > > > > > > What if the pipe is created non-blocking and stays non-blocking all the > > > > time and uses BYTE mode all the time? Just as sockets, it would always > > > > only emulate blocking mode. Wouldn't that drop code size a lot and fix > > > > most problems? > > > > > > If 'non-blocking' means overlapped I/O, only the problem will be: > > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > > > > Sorry if that wasn't clear, but I was not talking about overlapped I/O, > > which we should get rid off, but of real non-blocking mode, which > > Windows pipes are fortunately capable of. > > Do you mean, PIPE_NOWAIT flag? If this flags is specified in > the read pipe, non-cygwin apps cannot read the pipe correctly. Do you mean after execve? If the child is a non-Cygwin process, we could run a loop over the stdio descriptors in child_info_spawn::worker setting pipes to blocking. But that's a corner problem, easily solved. It's more important that it works nicely for Cygwin processes. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sat, 28 Aug 2021 18:41:02 +0900 Takashi Yano wrote: > On Sat, 28 Aug 2021 10:43:27 +0200 > Corinna Vinschen wrote: > > On Aug 28 02:21, Takashi Yano via Cygwin wrote: > > > On Fri, 27 Aug 2021 12:00:50 -0400 > > > Ken Brown wrote: > > > > Two years ago I thought I needed nt_create to avoid problems when > > > > calling > > > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > > > set_pipe_non_blocking unnecessary? Is that the point of your > > > > modification to > > > > raw_read? > > > > > > Yes. Instead of making windows read function itself non-blocking, > > > it is possible to check if the pipe can be read before read using > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > > returned. > > > > The problem is this: > > > > if (PeekNamedPipe()) > > ReadFile(blocking); > > > > is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > > thread from draining the pipe between the PeekNamedPipe and the ReadFile > > call. And as soon as ReadFile runs, it hangs indefinitely and we can't > > stop it via a signal. > > Hmm, you are right. Mutex guard seems to be necessary like pty code > if we go this way. I have found that set_pipe_non_blocking() succeeds for both read and write pipes if the write pipe is created by CreateNamedPipe() and the read pipe is created by CreateFile() contrary to the current create() code. Therefore, not only nt_create() but also PeekNamedPipe() become unnecessary. Please see the revised patch attached. -- Takashi Yano v2-0002-Cygwin-pipe-Revert-to-create-rather-than-nt_creat.patch Description: Binary data -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Hi Ken, On Sat, 28 Aug 2021 16:55:52 -0400 Ken Brown wrote: > On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote: > > On Sat, 28 Aug 2021 13:58:08 +0200 > > Corinna Vinschen wrote: > >> On Aug 28 18:41, Takashi Yano via Cygwin wrote: > >>> On Sat, 28 Aug 2021 10:43:27 +0200 > >>> Corinna Vinschen wrote: > On Aug 28 02:21, Takashi Yano via Cygwin wrote: > > On Fri, 27 Aug 2021 12:00:50 -0400 > > Ken Brown wrote: > >> Two years ago I thought I needed nt_create to avoid problems when > >> calling > >> set_pipe_non_blocking. Are you saying that's not an issue? Is > >> set_pipe_non_blocking unnecessary? Is that the point of your > >> modification to > >> raw_read? > > > > Yes. Instead of making windows read function itself non-blocking, > > it is possible to check if the pipe can be read before read using > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > returned. > > The problem is this: > > if (PeekNamedPipe()) > ReadFile(blocking); > > is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > thread from draining the pipe between the PeekNamedPipe and the ReadFile > call. And as soon as ReadFile runs, it hangs indefinitely and we can't > stop it via a signal. > >>> > >>> Hmm, you are right. Mutex guard seems to be necessary like pty code > >>> if we go this way. > >>> > Is a blocking ReadFile actually faster than a non-blocking read? Or > does it mainly depend on BYTE vs. MESSAGE mode? > >>> > >>> Actually, I don't think so. Perhaps it is not essential problem of > >>> overlapped I/O but something is wrong with current pipe code. > >>> > What if the pipe is created non-blocking and stays non-blocking all the > time and uses BYTE mode all the time? Just as sockets, it would always > only emulate blocking mode. Wouldn't that drop code size a lot and fix > most problems? > >>> > >>> If 'non-blocking' means overlapped I/O, only the problem will be: > >>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html > >> > >> Sorry if that wasn't clear, but I was not talking about overlapped I/O, > >> which we should get rid off, but of real non-blocking mode, which > >> Windows pipes are fortunately capable of. > > > > Do you mean, PIPE_NOWAIT flag? If this flags is specified in > > the read pipe, non-cygwin apps cannot read the pipe correctly. > > While waiting for Corinna's response to this, I have one more question. Do > you > understand why nt_create() failed and you had to revert to create()? Was it > an > access problem because nt_create requested FILE_WRITE_ATTRIBUTES? Or did I > make > some careless mistake in writing nt_create? I am sorry but no. I don't understand why piping C# program via the pipe created by nt_create() has the issue. I tried to change setup parameters in nt_create(), however, I did not succeed it to work. I also couldn't find any mistake in nt_create() so far. Win32 programs which use ReadFile() and WriteFile() work even with the pipe created by nt_create() as well as overlapped I/O. What does C# program differ from legacy win32 program at all? -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 13:58:08 +0200 Corinna Vinschen wrote: On Aug 28 18:41, Takashi Yano via Cygwin wrote: On Sat, 28 Aug 2021 10:43:27 +0200 Corinna Vinschen wrote: On Aug 28 02:21, Takashi Yano via Cygwin wrote: On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Hmm, you are right. Mutex guard seems to be necessary like pty code if we go this way. Is a blocking ReadFile actually faster than a non-blocking read? Or does it mainly depend on BYTE vs. MESSAGE mode? Actually, I don't think so. Perhaps it is not essential problem of overlapped I/O but something is wrong with current pipe code. What if the pipe is created non-blocking and stays non-blocking all the time and uses BYTE mode all the time? Just as sockets, it would always only emulate blocking mode. Wouldn't that drop code size a lot and fix most problems? If 'non-blocking' means overlapped I/O, only the problem will be: https://cygwin.com/pipermail/cygwin/2021-March/247987.html Sorry if that wasn't clear, but I was not talking about overlapped I/O, which we should get rid off, but of real non-blocking mode, which Windows pipes are fortunately capable of. Do you mean, PIPE_NOWAIT flag? If this flags is specified in the read pipe, non-cygwin apps cannot read the pipe correctly. While waiting for Corinna's response to this, I have one more question. Do you understand why nt_create() failed and you had to revert to create()? Was it an access problem because nt_create requested FILE_WRITE_ATTRIBUTES? Or did I make some careless mistake in writing nt_create? Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sat, 28 Aug 2021 13:58:08 +0200 Corinna Vinschen wrote: > On Aug 28 18:41, Takashi Yano via Cygwin wrote: > > On Sat, 28 Aug 2021 10:43:27 +0200 > > Corinna Vinschen wrote: > > > On Aug 28 02:21, Takashi Yano via Cygwin wrote: > > > > On Fri, 27 Aug 2021 12:00:50 -0400 > > > > Ken Brown wrote: > > > > > Two years ago I thought I needed nt_create to avoid problems when > > > > > calling > > > > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > > > > set_pipe_non_blocking unnecessary? Is that the point of your > > > > > modification to > > > > > raw_read? > > > > > > > > Yes. Instead of making windows read function itself non-blocking, > > > > it is possible to check if the pipe can be read before read using > > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > > > returned. > > > > > > The problem is this: > > > > > > if (PeekNamedPipe()) > > > ReadFile(blocking); > > > > > > is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > > > thread from draining the pipe between the PeekNamedPipe and the ReadFile > > > call. And as soon as ReadFile runs, it hangs indefinitely and we can't > > > stop it via a signal. > > > > Hmm, you are right. Mutex guard seems to be necessary like pty code > > if we go this way. > > > > > Is a blocking ReadFile actually faster than a non-blocking read? Or > > > does it mainly depend on BYTE vs. MESSAGE mode? > > > > Actually, I don't think so. Perhaps it is not essential problem of > > overlapped I/O but something is wrong with current pipe code. > > > > > What if the pipe is created non-blocking and stays non-blocking all the > > > time and uses BYTE mode all the time? Just as sockets, it would always > > > only emulate blocking mode. Wouldn't that drop code size a lot and fix > > > most problems? > > > > If 'non-blocking' means overlapped I/O, only the problem will be: > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > > Sorry if that wasn't clear, but I was not talking about overlapped I/O, > which we should get rid off, but of real non-blocking mode, which > Windows pipes are fortunately capable of. Do you mean, PIPE_NOWAIT flag? If this flags is specified in the read pipe, non-cygwin apps cannot read the pipe correctly. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/28/2021 4:43 AM, Corinna Vinschen via Cygwin wrote: On Aug 28 02:21, Takashi Yano via Cygwin wrote: On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Is a blocking ReadFile actually faster than a non-blocking read? Or does it mainly depend on BYTE vs. MESSAGE mode? What if the pipe is created non-blocking and stays non-blocking all the time and uses BYTE mode all the time? Just as sockets, it would always only emulate blocking mode. Wouldn't that drop code size a lot and fix most problems? It would certainly reduce the code size. I think it's worth a try, and we can see if any problems remain. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Aug 28 18:41, Takashi Yano via Cygwin wrote: > On Sat, 28 Aug 2021 10:43:27 +0200 > Corinna Vinschen wrote: > > On Aug 28 02:21, Takashi Yano via Cygwin wrote: > > > On Fri, 27 Aug 2021 12:00:50 -0400 > > > Ken Brown wrote: > > > > Two years ago I thought I needed nt_create to avoid problems when > > > > calling > > > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > > > set_pipe_non_blocking unnecessary? Is that the point of your > > > > modification to > > > > raw_read? > > > > > > Yes. Instead of making windows read function itself non-blocking, > > > it is possible to check if the pipe can be read before read using > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > > returned. > > > > The problem is this: > > > > if (PeekNamedPipe()) > > ReadFile(blocking); > > > > is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > > thread from draining the pipe between the PeekNamedPipe and the ReadFile > > call. And as soon as ReadFile runs, it hangs indefinitely and we can't > > stop it via a signal. > > Hmm, you are right. Mutex guard seems to be necessary like pty code > if we go this way. > > > Is a blocking ReadFile actually faster than a non-blocking read? Or > > does it mainly depend on BYTE vs. MESSAGE mode? > > Actually, I don't think so. Perhaps it is not essential problem of > overlapped I/O but something is wrong with current pipe code. > > > What if the pipe is created non-blocking and stays non-blocking all the > > time and uses BYTE mode all the time? Just as sockets, it would always > > only emulate blocking mode. Wouldn't that drop code size a lot and fix > > most problems? > > If 'non-blocking' means overlapped I/O, only the problem will be: > https://cygwin.com/pipermail/cygwin/2021-March/247987.html Sorry if that wasn't clear, but I was not talking about overlapped I/O, which we should get rid off, but of real non-blocking mode, which Windows pipes are fortunately capable of. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sat, 28 Aug 2021 10:43:27 +0200 Corinna Vinschen wrote: > On Aug 28 02:21, Takashi Yano via Cygwin wrote: > > On Fri, 27 Aug 2021 12:00:50 -0400 > > Ken Brown wrote: > > > Two years ago I thought I needed nt_create to avoid problems when calling > > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > > set_pipe_non_blocking unnecessary? Is that the point of your > > > modification to > > > raw_read? > > > > Yes. Instead of making windows read function itself non-blocking, > > it is possible to check if the pipe can be read before read using > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > returned. > > The problem is this: > > if (PeekNamedPipe()) > ReadFile(blocking); > > is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another > thread from draining the pipe between the PeekNamedPipe and the ReadFile > call. And as soon as ReadFile runs, it hangs indefinitely and we can't > stop it via a signal. Hmm, you are right. Mutex guard seems to be necessary like pty code if we go this way. > Is a blocking ReadFile actually faster than a non-blocking read? Or > does it mainly depend on BYTE vs. MESSAGE mode? Actually, I don't think so. Perhaps it is not essential problem of overlapped I/O but something is wrong with current pipe code. > What if the pipe is created non-blocking and stays non-blocking all the > time and uses BYTE mode all the time? Just as sockets, it would always > only emulate blocking mode. Wouldn't that drop code size a lot and fix > most problems? If 'non-blocking' means overlapped I/O, only the problem will be: https://cygwin.com/pipermail/cygwin/2021-March/247987.html -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Aug 28 02:21, Takashi Yano via Cygwin wrote: > On Fri, 27 Aug 2021 12:00:50 -0400 > Ken Brown wrote: > > Two years ago I thought I needed nt_create to avoid problems when calling > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > set_pipe_non_blocking unnecessary? Is that the point of your modification > > to > > raw_read? > > Yes. Instead of making windows read function itself non-blocking, > it is possible to check if the pipe can be read before read using > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > returned. The problem is this: if (PeekNamedPipe()) ReadFile(blocking); is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another thread from draining the pipe between the PeekNamedPipe and the ReadFile call. And as soon as ReadFile runs, it hangs indefinitely and we can't stop it via a signal. Is a blocking ReadFile actually faster than a non-blocking read? Or does it mainly depend on BYTE vs. MESSAGE mode? What if the pipe is created non-blocking and stays non-blocking all the time and uses BYTE mode all the time? Just as sockets, it would always only emulate blocking mode. Wouldn't that drop code size a lot and fix most problems? Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sat, 28 Aug 2021 11:00:24 +0900 Takashi Yano wrote: > On Sat, 28 Aug 2021 02:21:11 +0900 > Takashi Yano wrote: > > > On Fri, 27 Aug 2021 12:00:50 -0400 > > Ken Brown wrote: > > > > > On 8/27/2021 7:24 AM, Takashi Yano wrote: > > > > On Thu, 26 Aug 2021 18:18:29 -0400 > > > > Ken Brown wrote: > > > >> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: > > > [...] > > > >> In case you want to try out my proposed change, I've just rebased the > > > >> patches to > > > >> the current master and pushed them to a new topic/pipe branch. > > > > > > > > Hi Ken, > > > > > > > > Thanks much! I tested topic/pipe branch. > > > > > > > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > > > > yano@linux-server's password: > > > > test.dat 100% 100MB 95.9MB/s > > > > 00:01 > > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > > > yano@linux-server's password: > > > > test.dat 100% 100MB 8.0MB/s > > > > 00:12 > > > > > > > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > > > > yano@cygwin-PC's password: > > > > test.dat 100% 100MB 109.7MB/s > > > > 00:00 > > > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > > > > yano@cygwin-PC's password: > > > > test.dat 100% 100MB 31.4MB/s > > > > 00:03 > > > > > > > > As shown above, outgoing transfer-rate has been improved upto near > > > > theoretical limit. However, incoming transfer-rate is not improved > > > > much. > > > > > > > > I digged further and found the first patch attached solves the issue > > > > as follows. > > > > > > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > > > yano@linux-server's password: > > > > test.dat 100% 100MB 112.8MB/s > > > > 00:00 > > > > > > > > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:. > > > > yano@cygwin-PC's password: > > > > test.dat 100% 100MB 102.5MB/s > > > > 00:00 > > > > > > Great! > > > > > > > I also tested the case: > > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > > > which seems to be the same issue with > > > https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app > > > > > > > > Unfortunately, topic/pipe does not help. > > > > > > > > I confirmed that applying the second patch attached, which reverts > > > > to create() rather than nt_create(), and setting CYGWIN=pipe_byte > > > > fixes the problem. > > > > > > > > What do you think of this alternative implementation which does > > > > not use nt_create()? > > > > > > Two years ago I thought I needed nt_create to avoid problems when calling > > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > > set_pipe_non_blocking unnecessary? Is that the point of your > > > modification to > > > raw_read? > > > > Yes. Instead of making windows read function itself non-blocking, > > it is possible to check if the pipe can be read before read using > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > > returned. > > As for writint to pipe, set_pipe_non_blocking seems to take effect > and be necessary. So, the following patch seems to be appropriate. diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc index b3af4a0a0..82f5de3f4 100644 --- a/winsup/cygwin/fhandler_pipe.cc +++ b/winsup/cygwin/fhandler_pipe.cc @@ -48,6 +48,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking) IO_STATUS_BLOCK io; FILE_PIPE_INFORMATION fpi; + if ((openflags & O_ACCMODE) == O_RDONLY) +return; /* Do nothing for read pipe */ fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE; fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION : FILE_PIPE_QUEUE_OPERATION; -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Sat, 28 Aug 2021 02:21:11 +0900 Takashi Yano wrote: > On Fri, 27 Aug 2021 12:00:50 -0400 > Ken Brown wrote: > > > On 8/27/2021 7:24 AM, Takashi Yano wrote: > > > On Thu, 26 Aug 2021 18:18:29 -0400 > > > Ken Brown wrote: > > >> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: > > [...] > > >> In case you want to try out my proposed change, I've just rebased the > > >> patches to > > >> the current master and pushed them to a new topic/pipe branch. > > > > > > Hi Ken, > > > > > > Thanks much! I tested topic/pipe branch. > > > > > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > > > yano@linux-server's password: > > > test.dat 100% 100MB 95.9MB/s > > > 00:01 > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > > yano@linux-server's password: > > > test.dat 100% 100MB 8.0MB/s > > > 00:12 > > > > > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > > > yano@cygwin-PC's password: > > > test.dat 100% 100MB 109.7MB/s > > > 00:00 > > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > > > yano@cygwin-PC's password: > > > test.dat 100% 100MB 31.4MB/s > > > 00:03 > > > > > > As shown above, outgoing transfer-rate has been improved upto near > > > theoretical limit. However, incoming transfer-rate is not improved > > > much. > > > > > > I digged further and found the first patch attached solves the issue > > > as follows. > > > > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > > yano@linux-server's password: > > > test.dat 100% 100MB 112.8MB/s > > > 00:00 > > > > > > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:. > > > yano@cygwin-PC's password: > > > test.dat 100% 100MB 102.5MB/s > > > 00:00 > > > > Great! > > > > > I also tested the case: > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > > which seems to be the same issue with > > https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app > > > > > > Unfortunately, topic/pipe does not help. > > > > > > I confirmed that applying the second patch attached, which reverts > > > to create() rather than nt_create(), and setting CYGWIN=pipe_byte > > > fixes the problem. > > > > > > What do you think of this alternative implementation which does > > > not use nt_create()? > > > > Two years ago I thought I needed nt_create to avoid problems when calling > > set_pipe_non_blocking. Are you saying that's not an issue? Is > > set_pipe_non_blocking unnecessary? Is that the point of your modification > > to > > raw_read? > > Yes. Instead of making windows read function itself non-blocking, > it is possible to check if the pipe can be read before read using > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is > returned. As for writint to pipe, set_pipe_non_blocking seems to take effect and be necessary. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Fri, 27 Aug 2021 12:00:50 -0400 Ken Brown wrote: > On 8/27/2021 7:24 AM, Takashi Yano wrote: > > On Thu, 26 Aug 2021 18:18:29 -0400 > > Ken Brown wrote: > >> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: > [...] > >> In case you want to try out my proposed change, I've just rebased the > >> patches to > >> the current master and pushed them to a new topic/pipe branch. > > > > Hi Ken, > > > > Thanks much! I tested topic/pipe branch. > > > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > > yano@linux-server's password: > > test.dat 100% 100MB 95.9MB/s 00:01 > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > yano@linux-server's password: > > test.dat 100% 100MB 8.0MB/s 00:12 > > > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > > yano@cygwin-PC's password: > > test.dat 100% 100MB 109.7MB/s 00:00 > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > > yano@cygwin-PC's password: > > test.dat 100% 100MB 31.4MB/s 00:03 > > > > As shown above, outgoing transfer-rate has been improved upto near > > theoretical limit. However, incoming transfer-rate is not improved > > much. > > > > I digged further and found the first patch attached solves the issue > > as follows. > > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > yano@linux-server's password: > > test.dat 100% 100MB 112.8MB/s 00:00 > > > > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:. > > yano@cygwin-PC's password: > > test.dat 100% 100MB 102.5MB/s 00:00 > > Great! > > > I also tested the case: > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > which seems to be the same issue with > https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app > > > > Unfortunately, topic/pipe does not help. > > > > I confirmed that applying the second patch attached, which reverts > > to create() rather than nt_create(), and setting CYGWIN=pipe_byte > > fixes the problem. > > > > What do you think of this alternative implementation which does > > not use nt_create()? > > Two years ago I thought I needed nt_create to avoid problems when calling > set_pipe_non_blocking. Are you saying that's not an issue? Is > set_pipe_non_blocking unnecessary? Is that the point of your modification to > raw_read? Yes. Instead of making windows read function itself non-blocking, it is possible to check if the pipe can be read before read using PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is returned. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/27/2021 7:24 AM, Takashi Yano wrote: On Thu, 26 Aug 2021 18:18:29 -0400 Ken Brown wrote: On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: [...] In case you want to try out my proposed change, I've just rebased the patches to the current master and pushed them to a new topic/pipe branch. Hi Ken, Thanks much! I tested topic/pipe branch. [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 95.9MB/s 00:01 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 8.0MB/s 00:12 yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 109.7MB/s 00:00 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 31.4MB/s 00:03 As shown above, outgoing transfer-rate has been improved upto near theoretical limit. However, incoming transfer-rate is not improved much. I digged further and found the first patch attached solves the issue as follows. [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 112.8MB/s 00:00 yano@linux-server2:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 102.5MB/s 00:00 Great! I also tested the case: https://cygwin.com/pipermail/cygwin/2021-March/247987.html which seems to be the same issue with https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app Unfortunately, topic/pipe does not help. I confirmed that applying the second patch attached, which reverts to create() rather than nt_create(), and setting CYGWIN=pipe_byte fixes the problem. What do you think of this alternative implementation which does not use nt_create()? Two years ago I thought I needed nt_create to avoid problems when calling set_pipe_non_blocking. Are you saying that's not an issue? Is set_pipe_non_blocking unnecessary? Is that the point of your modification to raw_read? Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Thu, 26 Aug 2021 18:18:29 -0400 Ken Brown wrote: > On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: > > On 8/25/2021 5:29 PM, Takashi Yano wrote: > >> On Wed, 25 Aug 2021 13:52:19 -0400 > >> Ken Brown wrote: > >>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: > On Tue, 24 Aug 2021 12:49:52 -0700 > Chris Roehrig wrote: > > I have a network of Windows, Linux and Mac machines and I use rsync to > > synchronize various directories between them. > > > > I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) > > only > > when the remote endpoint is Cygwin rsync over sshd (with both a Linux > > or > > Cygwin rsync client). In all other scenarios, I get the full 100MB/s > > as > > expected from gigabit ethernet. This has been an ongoing problem for > > me > > for a couple of years over several Windows and Cygwin versions, and I'd > > like to try to fix it. > > > > If I run rsync --daemon --no-detach under mintty in the foreground on > > the > > remote Windows endpoint, I get the full 100 MB/s transfers, so it > > seems > > like it has something to do with rsync.exe running in the background > > under > > the cygrunsrv+sshd service (which was installed normally using > > ssh-host-config). > > > > If I do: > > pv /dev/zero | ssh $WINHOST "cat > /dev/null" > > or even > > pv /dev/urandom | ssh $WINHOST md5sum > > I also get the full 100 MB/s transfers, so it doesn't look like sshd > > itself > > is being throttled by bandwidth or CPU. > > > > The machines have less than 15% CPU utilization while transferring, > > with > > each of the 4 cores less than 30%, so it doesn't look to be CPU issue. > > In Task Manager, sshd.exe and rsync.exe seem to be running normally > > using > > only few percent CPU, and show Power Throttling=Disabled, > > Priority=Normal. Setting their Priority to High doesn't seem to > > change > > things. > > > > Looking in Resource Monitor on the remote endpoint, the network usage > > is > > pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it > > sure > > looks to me as if rsync is somehow being bandwidth-throttled when run > > in > > the background under cygsshd. > > > > It's almost as if rsync has an implicit --bwlimit override when it is > > run > > from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes > > no > > difference). > > > > > > Any ideas? Not sure where to go from here. > > In cygwin, just scp is very slow. > > The transfer speed in my environment is as follows. > The tests were done with 100MB of test.dat file. > > (1-1) From cygwin-PC, > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > yano@linux-server's password: > test.dat 100% 100MB 4.0MB/s > 00:24 > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > yano@linux-server's password: > test.dat 100% 100MB 8.0MB/s > 00:12 > > (1-2) From linux-server, > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > yano@cygwin-PC's password: > test.dat 100% 100MB 4.0MB/s > 00:24 > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > yano@cygwin-PC's password: > test.dat 100% 100MB 4.1MB/s > 00:24 > > > I looked into this problem, and noticed that this is caused > by cygwin pipe implementation. Pipe in cygwin is configured > with FILE_FLAG_OVERLAPPED. > > If the pipe is configured without FILE_FLAG_OVERLAPPED, > the transfer speed is much improved as follows. > > > (2-1) From cygwin-PC, > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > yano@linux-server's password: > test.dat 100% 100MB 85.5MB/s > 00:01 > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > yano@linux-server's password: > test.dat 100% 100MB 69.7MB/s > 00:01 > > (2-2) From linux-server, > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > yano@cygwin-PC's password: > test.dat 100% 100MB 80.1MB/s > 00:01 > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > yano@cygwin-PC's password: > test.dat 100% 100MB 57.7MB/s > 00:01 > > I am not sure why this happens and how to fix this. > >>> > >>> A couple years ago I had an idea for changing the pipe implementation to > >>> avoid > >>> overlapped I/O: > >>> >
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: On 8/25/2021 5:29 PM, Takashi Yano wrote: On Wed, 25 Aug 2021 13:52:19 -0400 Ken Brown wrote: On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: On Tue, 24 Aug 2021 12:49:52 -0700 Chris Roehrig wrote: I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). If I do: pv /dev/zero | ssh $WINHOST "cat > /dev/null" or even pv /dev/urandom | ssh $WINHOST md5sum I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). Any ideas? Not sure where to go from here. In cygwin, just scp is very slow. The transfer speed in my environment is as follows. The tests were done with 100MB of test.dat file. (1-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 4.0MB/s 00:24 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 8.0MB/s 00:12 (1-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 4.0MB/s 00:24 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 4.1MB/s 00:24 I looked into this problem, and noticed that this is caused by cygwin pipe implementation. Pipe in cygwin is configured with FILE_FLAG_OVERLAPPED. If the pipe is configured without FILE_FLAG_OVERLAPPED, the transfer speed is much improved as follows. (2-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 85.5MB/s 00:01 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 69.7MB/s 00:01 (2-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 80.1MB/s 00:01 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 57.7MB/s 00:01 I am not sure why this happens and how to fix this. A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. Interesting. It will be also helpfull for: https://cygwin.com/pipermail/cygwin/2021-March/247987.html which seems to be the same issue with https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app However, I wonder why scp dislikes overlapped I/O. I agree that it would be good to understand this. When I first proposed the change, I was thinking in terms of code simplification. If it also improves performance (which we don't know yet), it becomes a higher priority, but in that case it would be nice to understand why it improves performance. Hi Takashi, In case you want to try out my proposed change, I've just rebased
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/25/2021 5:29 PM, Takashi Yano wrote: On Wed, 25 Aug 2021 13:52:19 -0400 Ken Brown wrote: On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: On Tue, 24 Aug 2021 12:49:52 -0700 Chris Roehrig wrote: I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). If I do: pv /dev/zero | ssh $WINHOST "cat > /dev/null" or even pv /dev/urandom | ssh $WINHOST md5sum I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). Any ideas?Not sure where to go from here. In cygwin, just scp is very slow. The transfer speed in my environment is as follows. The tests were done with 100MB of test.dat file. (1-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 4.0MB/s 00:24 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 8.0MB/s 00:12 (1-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 4.0MB/s 00:24 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 4.1MB/s 00:24 I looked into this problem, and noticed that this is caused by cygwin pipe implementation. Pipe in cygwin is configured with FILE_FLAG_OVERLAPPED. If the pipe is configured without FILE_FLAG_OVERLAPPED, the transfer speed is much improved as follows. (2-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 85.5MB/s 00:01 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 69.7MB/s 00:01 (2-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 80.1MB/s 00:01 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 57.7MB/s 00:01 I am not sure why this happens and how to fix this. A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. Interesting. It will be also helpfull for: https://cygwin.com/pipermail/cygwin/2021-March/247987.html which seems to be the same issue with https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app However, I wonder why scp dislikes overlapped I/O. I agree that it would be good to understand this. When I first proposed the change, I was thinking in terms of code simplification. If it also improves performance (which we don't know yet), it becomes a higher priority, but in that case it would be nice to understand why it improves performance. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/25/2021 4:33 PM, Mario Emmenlauer wrote: On 25.08.21 19:52, Ken Brown via Cygwin wrote: A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. This sounds quite interesting! Do I assume correctly that these changes may likely lead to a general improvement also in other situations if the impact of the pipe is so big for this specific case? Pipes are most likely used in quite a lot of scenarios?! I'm not prepared to make any predictions. We won't really know without testing. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/25/2021 2:18 PM, Chris Roehrig wrote: On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin wrote: A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. Ken I'm not familiar enough with the innards of rsync, sshd or cygwin to know how this would work. Is it possible to have a new CYGWIN environment option to switch the pipe behaviour without requiring changes to the ssh or rsync source code (and without breaking any existing stuff)? My proposed change would be purely internal to Cygwin. There should be no user-visible change in behavior except, perhaps, improved performance. (And we don't know yet whether there would be improved performance.) So I don't see a need for a new CYGWIN option. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Wed, 25 Aug 2021 13:52:19 -0400 Ken Brown wrote: > On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: > > On Tue, 24 Aug 2021 12:49:52 -0700 > > Chris Roehrig wrote: > >> I have a network of Windows, Linux and Mac machines and I use rsync to > >> synchronize various directories between them. > >> > >> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only > >> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or > >> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as > >> expected from gigabit ethernet. This has been an ongoing problem for me > >> for a couple of years over several Windows and Cygwin versions, and I'd > >> like to try to fix it. > >> > >> If I run rsync --daemon --no-detach under mintty in the foreground on the > >> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems > >> like it has something to do with rsync.exe running in the background under > >> the cygrunsrv+sshd service (which was installed normally using > >> ssh-host-config). > >> > >> If I do: > >>pv /dev/zero | ssh $WINHOST "cat > /dev/null" > >> or even > >>pv /dev/urandom | ssh $WINHOST md5sum > >> I also get the full 100 MB/s transfers, so it doesn't look like sshd > >> itself is being throttled by bandwidth or CPU. > >> > >> The machines have less than 15% CPU utilization while transferring, with > >> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. > >> In Task Manager, sshd.exe and rsync.exe seem to be running normally using > >> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. > >> Setting their Priority to High doesn't seem to change things. > >> > >> Looking in Resource Monitor on the remote endpoint, the network usage is > >> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure > >> looks to me as if rsync is somehow being bandwidth-throttled when run in > >> the background under cygsshd. > >> > >> It's almost as if rsync has an implicit --bwlimit override when it is run > >> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no > >> difference). > >> > >> > >> Any ideas?Not sure where to go from here. > > > > In cygwin, just scp is very slow. > > > > The transfer speed in my environment is as follows. > > The tests were done with 100MB of test.dat file. > > > > (1-1) From cygwin-PC, > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > > yano@linux-server's password: > > test.dat 100% 100MB 4.0MB/s 00:24 > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > yano@linux-server's password: > > test.dat 100% 100MB 8.0MB/s 00:12 > > > > (1-2) From linux-server, > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > > yano@cygwin-PC's password: > > test.dat 100% 100MB 4.0MB/s 00:24 > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > > yano@cygwin-PC's password: > > test.dat 100% 100MB 4.1MB/s 00:24 > > > > > > I looked into this problem, and noticed that this is caused > > by cygwin pipe implementation. Pipe in cygwin is configured > > with FILE_FLAG_OVERLAPPED. > > > > If the pipe is configured without FILE_FLAG_OVERLAPPED, > > the transfer speed is much improved as follows. > > > > > > (2-1) From cygwin-PC, > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > > yano@linux-server's password: > > test.dat 100% 100MB 85.5MB/s 00:01 > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > > yano@linux-server's password: > > test.dat 100% 100MB 69.7MB/s 00:01 > > > > (2-2) From linux-server, > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > > yano@cygwin-PC's password: > > test.dat 100% 100MB 80.1MB/s 00:01 > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > > yano@cygwin-PC's password: > > test.dat 100% 100MB 57.7MB/s 00:01 > > > > I am not sure why this happens and how to fix this. > > A couple years ago I had an idea for changing the pipe implementation to > avoid > overlapped I/O: > >https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html >https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html > > I never followed up on it. But if you think it might help with this problem, > I > could dust it off and try to finish it. Interesting. It will be also helpfull for: https://cygwin.com/pipermail/cygwin/2021-March/247987.html which seems to be the same issue with https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app However, I wonder why scp dislikes overlapped I/O. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 25.08.21 19:52, Ken Brown via Cygwin wrote: A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. This sounds quite interesting! Do I assume correctly that these changes may likely lead to a general improvement also in other situations if the impact of the pipe is so big for this specific case? Pipes are most likely used in quite a lot of scenarios?! All the best, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
It looks like there was a previous (2013) patch and attempt to add a pipe_nooverlap CYGWIN option that was rejected by the maintainers: https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app Is this something that can be revisited?It seems to be affecting pure-UNIX situations (in my case) which is Cygwin's core audience. - Chris On Wed Aug 25 2021, at 11:18 AM, Chris Roehrig wrote: > On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin > wrote: >> A couple years ago I had an idea for changing the pipe implementation to >> avoid overlapped I/O: >> >> https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html >> https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html >> >> I never followed up on it. But if you think it might help with this >> problem, I could dust it off and try to finish it. >> >> Ken > > I'm not familiar enough with the innards of rsync, sshd or cygwin to know how > this would work. > Is it possible to have a new CYGWIN environment option to switch the pipe > behaviour without requiring changes to the ssh or rsync source code (and > without breaking any existing stuff)? > > - Chris > > > >> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: >>> On Tue, 24 Aug 2021 12:49:52 -0700 >>> Chris Roehrig wrote: I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). If I do: pv /dev/zero | ssh $WINHOST "cat > /dev/null" or even pv /dev/urandom | ssh $WINHOST md5sum I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). Any ideas?Not sure where to go from here. >>> In cygwin, just scp is very slow. >>> The transfer speed in my environment is as follows. >>> The tests were done with 100MB of test.dat file. >>> (1-1) From cygwin-PC, >>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >>> yano@linux-server's password: >>> test.dat 100% 100MB 4.0MB/s 00:24 >>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >>> yano@linux-server's password: >>> test.dat 100% 100MB 8.0MB/s 00:12 >>> (1-2) From linux-server, >>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 4.0MB/s 00:24 >>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 4.1MB/s 00:24 >>> I looked into this problem, and noticed that this is caused >>> by cygwin pipe implementation. Pipe in cygwin is configured >>> with FILE_FLAG_OVERLAPPED. >>> If the pipe is configured without FILE_FLAG_OVERLAPPED, >>> the transfer speed is much improved as follows. >>> (2-1) From cygwin-PC, >>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >>> yano@linux-server's password: >>> test.dat 100% 100MB 85.5MB/s 00:01 >>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >>> yano@linux-server's password: >>> test.dat 100% 100MB 69.7MB/s 00:01 >>> (2-2) From linux-server, >>> yano@linux-server:~$ scp
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin wrote: > A couple years ago I had an idea for changing the pipe implementation to > avoid overlapped I/O: > > https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html > https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html > > I never followed up on it. But if you think it might help with this problem, > I could dust it off and try to finish it. > > Ken I'm not familiar enough with the innards of rsync, sshd or cygwin to know how this would work. Is it possible to have a new CYGWIN environment option to switch the pipe behaviour without requiring changes to the ssh or rsync source code (and without breaking any existing stuff)? - Chris > On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: >> On Tue, 24 Aug 2021 12:49:52 -0700 >> Chris Roehrig wrote: >>> I have a network of Windows, Linux and Mac machines and I use rsync to >>> synchronize various directories between them. >>> >>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >>> expected from gigabit ethernet. This has been an ongoing problem for me >>> for a couple of years over several Windows and Cygwin versions, and I'd >>> like to try to fix it. >>> >>> If I run rsync --daemon --no-detach under mintty in the foreground on the >>> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >>> like it has something to do with rsync.exe running in the background under >>> the cygrunsrv+sshd service (which was installed normally using >>> ssh-host-config). >>> >>> If I do: >>> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >>> or even >>> pv /dev/urandom | ssh $WINHOST md5sum >>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >>> is being throttled by bandwidth or CPU. >>> >>> The machines have less than 15% CPU utilization while transferring, with >>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >>> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >>> Setting their Priority to High doesn't seem to change things. >>> >>> Looking in Resource Monitor on the remote endpoint, the network usage is >>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >>> looks to me as if rsync is somehow being bandwidth-throttled when run in >>> the background under cygsshd. >>> >>> It's almost as if rsync has an implicit --bwlimit override when it is run >>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >>> difference). >>> >>> >>> Any ideas?Not sure where to go from here. >> In cygwin, just scp is very slow. >> The transfer speed in my environment is as follows. >> The tests were done with 100MB of test.dat file. >> (1-1) From cygwin-PC, >> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >> yano@linux-server's password: >> test.dat 100% 100MB 4.0MB/s 00:24 >> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >> yano@linux-server's password: >> test.dat 100% 100MB 8.0MB/s 00:12 >> (1-2) From linux-server, >> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >> yano@cygwin-PC's password: >> test.dat 100% 100MB 4.0MB/s 00:24 >> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >> yano@cygwin-PC's password: >> test.dat 100% 100MB 4.1MB/s 00:24 >> I looked into this problem, and noticed that this is caused >> by cygwin pipe implementation. Pipe in cygwin is configured >> with FILE_FLAG_OVERLAPPED. >> If the pipe is configured without FILE_FLAG_OVERLAPPED, >> the transfer speed is much improved as follows. >> (2-1) From cygwin-PC, >> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >> yano@linux-server's password: >> test.dat 100% 100MB 85.5MB/s 00:01 >> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >> yano@linux-server's password: >> test.dat 100% 100MB 69.7MB/s 00:01 >> (2-2) From linux-server, >> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >> yano@cygwin-PC's password: >> test.dat 100% 100MB 80.1MB/s 00:01 >> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >> yano@cygwin-PC's password: >> test.dat 100% 100MB 57.7MB/s 00:01 >> I am not sure why this happens and how to fix this. > > > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info:
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: On Tue, 24 Aug 2021 12:49:52 -0700 Chris Roehrig wrote: I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). If I do: pv /dev/zero | ssh $WINHOST "cat > /dev/null" or even pv /dev/urandom | ssh $WINHOST md5sum I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). Any ideas?Not sure where to go from here. In cygwin, just scp is very slow. The transfer speed in my environment is as follows. The tests were done with 100MB of test.dat file. (1-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 4.0MB/s 00:24 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 8.0MB/s 00:12 (1-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 4.0MB/s 00:24 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 4.1MB/s 00:24 I looked into this problem, and noticed that this is caused by cygwin pipe implementation. Pipe in cygwin is configured with FILE_FLAG_OVERLAPPED. If the pipe is configured without FILE_FLAG_OVERLAPPED, the transfer speed is much improved as follows. (2-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 85.5MB/s 00:01 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 69.7MB/s 00:01 (2-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 80.1MB/s 00:01 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 57.7MB/s 00:01 I am not sure why this happens and how to fix this. A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. Ken -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
The FILE_FLAG_OVERLAPPED avenue looks promising. I get exactly the same results as you using 'scp': 4MB/s in either direction (with either remote endpoint.) I guess the next step is to install a build environment and build rsync and sshd ... On Wed Aug 25 2021, at 4:18 AM, Takashi Yano via Cygwin wrote: > On Tue, 24 Aug 2021 12:49:52 -0700 > Chris Roehrig wrote: >> I have a network of Windows, Linux and Mac machines and I use rsync to >> synchronize various directories between them. >> >> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >> expected from gigabit ethernet. This has been an ongoing problem for me for >> a couple of years over several Windows and Cygwin versions, and I'd like to >> try to fix it. >> >> If I run rsync --daemon --no-detach under mintty in the foreground on the >> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >> like it has something to do with rsync.exe running in the background under >> the cygrunsrv+sshd service (which was installed normally using >> ssh-host-config). >> >> If I do: >> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >> or even >> pv /dev/urandom | ssh $WINHOST md5sum >> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >> is being throttled by bandwidth or CPU. >> >> The machines have less than 15% CPU utilization while transferring, with >> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >> Setting their Priority to High doesn't seem to change things. >> >> Looking in Resource Monitor on the remote endpoint, the network usage is >> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >> looks to me as if rsync is somehow being bandwidth-throttled when run in >> the background under cygsshd. >> >> It's almost as if rsync has an implicit --bwlimit override when it is run >> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >> difference). >> >> >> Any ideas?Not sure where to go from here. > > In cygwin, just scp is very slow. > > The transfer speed in my environment is as follows. > The tests were done with 100MB of test.dat file. > > (1-1) From cygwin-PC, > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > yano@linux-server's password: > test.dat 100% 100MB 4.0MB/s 00:24 > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > yano@linux-server's password: > test.dat 100% 100MB 8.0MB/s 00:12 > > (1-2) From linux-server, > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > yano@cygwin-PC's password: > test.dat 100% 100MB 4.0MB/s 00:24 > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > yano@cygwin-PC's password: > test.dat 100% 100MB 4.1MB/s 00:24 > > > I looked into this problem, and noticed that this is caused > by cygwin pipe implementation. Pipe in cygwin is configured > with FILE_FLAG_OVERLAPPED. > > If the pipe is configured without FILE_FLAG_OVERLAPPED, > the transfer speed is much improved as follows. > > > (2-1) From cygwin-PC, > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > yano@linux-server's password: > test.dat 100% 100MB 85.5MB/s 00:01 > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > yano@linux-server's password: > test.dat 100% 100MB 69.7MB/s 00:01 > > (2-2) From linux-server, > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > yano@cygwin-PC's password: > test.dat 100% 100MB 80.1MB/s 00:01 > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > yano@cygwin-PC's password: > test.dat 100% 100MB 57.7MB/s 00:01 > > I am not sure why this happens and how to fix this. > Any idea? > > -- > Takashi Yano > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
It was set to "Programs".Changing it to "Background services" didn't make a difference. TCPOptimizer can adjust 2 registry entries that I think are related to that Windows Setting: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile] "NetworkThrottlingIndex"=dword:000a "SystemResponsiveness"=dword:0014 The first rate-limits IP packets; the second reserves a percent (0-100) of CPU scheduling for background tasks. Changing either of those values made no difference. I think the CPU scheduling reservation only applies when there's CPU contention (not on an idle system) so I wouldn't expect it to make a difference when CPU/core usage is low. On Tue Aug 24 2021, at 8:20 PM, Mark Geisert wrote: > NightStrike via Cygwin wrote: >> Older versions of windows had a setting to optimize the OS for either >> background services or foreground applications. One of the things this did >> was throttle network usage. I don't know if windows 10 has the same setting >> though. > > Yes, it does. Getting to it is a pain. Search the "Settings" thingy for > "Advanced system settings". That will bring up the System Properties dialog > you used to be able to get to from the Control Panel. Click the Advanced > tab. Under Performance, click Settings... . Click the Advanced tab there. > Right in that dialog there's a Processor Scheduling box where you select for > best performance of "Programs" or "Background services". > > It would be interesting to hear what it's currently set to before you try > changing it. Try to keep all your testcases the same so we're dealing with > apples to apples. > > ..mark > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Tue, 24 Aug 2021 12:49:52 -0700 Chris Roehrig wrote: > I have a network of Windows, Linux and Mac machines and I use rsync to > synchronize various directories between them. > > I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only > when the remote endpoint is Cygwin rsync over sshd (with both a Linux or > Cygwin rsync client). In all other scenarios, I get the full 100MB/s as > expected from gigabit ethernet. This has been an ongoing problem for me for > a couple of years over several Windows and Cygwin versions, and I'd like to > try to fix it. > > If I run rsync --daemon --no-detach under mintty in the foreground on the > remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like > it has something to do with rsync.exe running in the background under the > cygrunsrv+sshd service (which was installed normally using ssh-host-config). > > If I do: > pv /dev/zero | ssh $WINHOST "cat > /dev/null" > or even > pv /dev/urandom | ssh $WINHOST md5sum > I also get the full 100 MB/s transfers, so it doesn't look like sshd itself > is being throttled by bandwidth or CPU. > > The machines have less than 15% CPU utilization while transferring, with each > of the 4 cores less than 30%, so it doesn't look to be CPU issue. > In Task Manager, sshd.exe and rsync.exe seem to be running normally using > only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. > Setting their Priority to High doesn't seem to change things. > > Looking in Resource Monitor on the remote endpoint, the network usage is > pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure > looks to me as if rsync is somehow being bandwidth-throttled when run in the > background under cygsshd. > > It's almost as if rsync has an implicit --bwlimit override when it is run > from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no > difference). > > > Any ideas?Not sure where to go from here. In cygwin, just scp is very slow. The transfer speed in my environment is as follows. The tests were done with 100MB of test.dat file. (1-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 4.0MB/s 00:24 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 8.0MB/s 00:12 (1-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 4.0MB/s 00:24 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 4.1MB/s 00:24 I looked into this problem, and noticed that this is caused by cygwin pipe implementation. Pipe in cygwin is configured with FILE_FLAG_OVERLAPPED. If the pipe is configured without FILE_FLAG_OVERLAPPED, the transfer speed is much improved as follows. (2-1) From cygwin-PC, [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 85.5MB/s 00:01 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 69.7MB/s 00:01 (2-2) From linux-server, yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 80.1MB/s 00:01 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 57.7MB/s 00:01 I am not sure why this happens and how to fix this. Any idea? -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
NightStrike via Cygwin wrote: Older versions of windows had a setting to optimize the OS for either background services or foreground applications. One of the things this did was throttle network usage. I don't know if windows 10 has the same setting though. Yes, it does. Getting to it is a pain. Search the "Settings" thingy for "Advanced system settings". That will bring up the System Properties dialog you used to be able to get to from the Control Panel. Click the Advanced tab. Under Performance, click Settings... . Click the Advanced tab there. Right in that dialog there's a Processor Scheduling box where you select for best performance of "Programs" or "Background services". It would be interesting to hear what it's currently set to before you try changing it. Try to keep all your testcases the same so we're dealing with apples to apples. ..mark -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Here's the other direction with rsync also in --daemon mode: # ON WINDOWS (from a mintty terminal): dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000 # create 2GB file printf "[TMP]\npath = /tmp\n" > /etc/rsyncd.conf # create rsyncd.conf rsync --daemon --no-detach # run rsync in daemon mode # ON LINUX: rm -f /tmp/bigfile.bin; rsync -Pva $WINHOST:/tmp/bigfile.bin /tmp/ # 4MB/s rm -f /tmp/bigfile.bin; rsync -Pva rsync://$WINHOST/TMP/bigfile.bin /tmp/ # 100MB/s I was thinking that there might be some Windows Policy that rate-limits some syscalls for background process, but I also tried: # ON WINDOWS (Administrator mintty): cygrunsrv -E cygsshd# exit sshd service /usr/sbin/sshd -D # run sshd in "foreground" (GUI app) and I still get 4MB/s. Maybe some IPC rate-limit issue between sshd and rsync (but that isn't there between e.g. sshd and md5sum... ?) On Tue Aug 24 2021, at 7:02 PM, NightStrike wrote: > Older versions of windows had a setting to optimize the OS for either > background services or foreground applications. One of the things this did > was throttle network usage. I don't know if windows 10 has the same setting > though. I read about a NetworkThrottlingIndex that limits "non-multimedia" packets to 10 per millisecond which can be disabled by setting: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile] "NetworkThrottlingIndex"=dword: but this made no difference to my tests. On Tue Aug 24 2021, at 5:35 PM, Chris Roehrig wrote: > Here's my test in a nutshell: > > # ON WINDOWS: install Cygwin and enable cygsshd > ssh-host-config -y > # set up authorized keys, etc to make things easier > > # LINUX: Create a 2GB random file on Linux: > dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000 > > # WINDOWS: rsync "pull" to cygwin > LINUXHOST=mylinuxhost > rsync -Pva $LINUXHOST:/tmp/bigfile.bin /tmp/# 100MB/s full > speed > rm /tmp/bigfile.bin > > # LINUX: rsync "push" to cygwin (from Linux machine) > WINHOST=mywindowshost > rsync -Pva /tmp/bigfile.bin $WINHOST:/tmp/ # slow, > dropping to 4MB/s > > > I get the same results transferring the other direction (WIN -> LINUX): The > Cygwin remote endpoint is always slow. > > > > > > On Tue Aug 24 2021, at 4:43 PM, Mark Geisert wrote: > >> Chris Roehrig wrote: >>> I have a network of Windows, Linux and Mac machines and I use rsync to >>> synchronize various directories between them. >>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >>> expected from gigabit ethernet. This has been an ongoing problem for me >>> for a couple of years over several Windows and Cygwin versions, and I'd >>> like to try to fix it. >>> If I run rsync --daemon --no-detach under mintty in the foreground on the >>> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >>> like it has something to do with rsync.exe running in the background under >>> the cygrunsrv+sshd service (which was installed normally using >>> ssh-host-config). >>> If I do: >>> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >>> or even >>> pv /dev/urandom | ssh $WINHOST md5sum >>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >>> is being throttled by bandwidth or CPU. >>> The machines have less than 15% CPU utilization while transferring, with >>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >>> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >>> Setting their Priority to High doesn't seem to change things. >>> Looking in Resource Monitor on the remote endpoint, the network usage is >>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >>> looks to me as if rsync is somehow being bandwidth-throttled when run in >>> the background under cygsshd. >>> It's almost as if rsync has an implicit --bwlimit override when it is run >>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >>> difference). >>> Any ideas?Not sure where to go from here. >> >> You're not the first to report this. I don't have any quick answer. But >> can you give one or two simple examples of commands that give slow transfers >> in your environment? Simple like your 'pv' examples, if possible, using >> whatever method that works. >> >> ..mark >> >> -- >> Problem reports:
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Tue, Aug 24, 2021, 09:50 Chris Roehrig wrote: > I have a network of Windows, Linux and Mac machines and I use rsync to > synchronize various directories between them. > > I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only > when the remote endpoint is Cygwin rsync over sshd (with both a Linux or > Cygwin rsync client). In all other scenarios, I get the full 100MB/s as > expected from gigabit ethernet. This has been an ongoing problem for me > for a couple of years over several Windows and Cygwin versions, and I'd > like to try to fix it. > > If I run rsync --daemon --no-detach under mintty in the foreground on the > remote Windows endpoint, I get the full 100 MB/s transfers, so it seems > like it has something to do with rsync.exe running in the background under > the cygrunsrv+sshd service (which was installed normally using > ssh-host-config). > > If I do: > pv /dev/zero | ssh $WINHOST "cat > /dev/null" > or even > pv /dev/urandom | ssh $WINHOST md5sum > I also get the full 100 MB/s transfers, so it doesn't look like sshd > itself is being throttled by bandwidth or CPU. > > The machines have less than 15% CPU utilization while transferring, with > each of the 4 cores less than 30%, so it doesn't look to be CPU issue. > In Task Manager, sshd.exe and rsync.exe seem to be running normally using > only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. > Setting their Priority to High doesn't seem to change things. > > Looking in Resource Monitor on the remote endpoint, the network usage is > pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure > looks to me as if rsync is somehow being bandwidth-throttled when run in > the background under cygsshd. > > It's almost as if rsync has an implicit --bwlimit override when it is run > from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no > difference). > > > Any ideas?Not sure where to go from here. > > Thanks, > -- Chris > > > Windows 10 v2004 (64-bit) > CYGWIN_NT-10.0 3.2.0(0.340/5/3) 2021-03-29 08:42 x86_64 Cygwin > rsync version 3.2.4dev protocol version 31 > Linux Mint 20 > Older versions of windows had a setting to optimize the OS for either background services or foreground applications. One of the things this did was throttle network usage. I don't know if windows 10 has the same setting though. > -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Here's my test in a nutshell: # ON WINDOWS: install Cygwin and enable cygsshd ssh-host-config -y # set up authorized keys, etc to make things easier # LINUX: Create a 2GB random file on Linux: dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000 # WINDOWS: rsync "pull" to cygwin LINUXHOST=mylinuxhost rsync -Pva $LINUXHOST:/tmp/bigfile.bin /tmp/# 100MB/s full speed rm /tmp/bigfile.bin # LINUX: rsync "push" to cygwin (from Linux machine) WINHOST=mywindowshost rsync -Pva /tmp/bigfile.bin $WINHOST:/tmp/ # slow, dropping to 4MB/s I get the same results transferring the other direction (WIN -> LINUX): The Cygwin remote endpoint is always slow. On Tue Aug 24 2021, at 4:43 PM, Mark Geisert wrote: > Chris Roehrig wrote: >> I have a network of Windows, Linux and Mac machines and I use rsync to >> synchronize various directories between them. >> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >> expected from gigabit ethernet. This has been an ongoing problem for me for >> a couple of years over several Windows and Cygwin versions, and I'd like to >> try to fix it. >> If I run rsync --daemon --no-detach under mintty in the foreground on the >> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >> like it has something to do with rsync.exe running in the background under >> the cygrunsrv+sshd service (which was installed normally using >> ssh-host-config). >> If I do: >> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >> or even >> pv /dev/urandom | ssh $WINHOST md5sum >> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >> is being throttled by bandwidth or CPU. >> The machines have less than 15% CPU utilization while transferring, with >> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >> Setting their Priority to High doesn't seem to change things. >> Looking in Resource Monitor on the remote endpoint, the network usage is >> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >> looks to me as if rsync is somehow being bandwidth-throttled when run in >> the background under cygsshd. >> It's almost as if rsync has an implicit --bwlimit override when it is run >> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >> difference). >> Any ideas?Not sure where to go from here. > > You're not the first to report this. I don't have any quick answer. But can > you give one or two simple examples of commands that give slow transfers in > your environment? Simple like your 'pv' examples, if possible, using > whatever method that works. > > ..mark > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Chris Roehrig wrote: I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). If I do: pv /dev/zero | ssh $WINHOST "cat > /dev/null" or even pv /dev/urandom | ssh $WINHOST md5sum I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). Any ideas?Not sure where to go from here. You're not the first to report this. I don't have any quick answer. But can you give one or two simple examples of commands that give slow transfers in your environment? Simple like your 'pv' examples, if possible, using whatever method that works. ..mark -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
