Re: scp and ssh 'cat' stalls at 64k bytes
On 2023-06-24 00:12, Kevin Schnitzius wrote:
On Friday, June 23, 2023 at 10:00:02 PM EDT, Chris Roehrig via Cygwin
wrote:
Thanks. There must be some issue with my setup. Very odd that 'pv'
works, but 'cat' does not. ldd shows they use identical libs. I guess
I'll start with the pv and cat source.
Try this first:
/usr/bin/dd if=/dev/zero bs=1 count=65536 | /usr/bin/ssh localhost '/usr/bin/cat
> /dev/null'
Good thought. No change. NB: 65536 always works fine. Anything
greater (e.g. 65537) does not work.
Also, I assume you are running Cygwin's bash in a mintty term...
Yes, same results with both mintty (Administrator or not) or slogged in
remotely.
I found another one that works: tee.
So these work fine (and /dev/null can be replaced by any file or
omitted; doesn't matter):
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'wc' # WORKS
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'tee /dev/null | cat
> /dev/null' # WORKS
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'pv | cat >
/dev/null' # WORKS
dd if=/dev/urandom bs=1 count=65536 | ssh localhost 'cat > /dev/null'
# WORKS
These don't:
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'cat | wc' # STALLS
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'cat > /dev/null'
# STALLS
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'awk {print} >
/dev/null' # STALLS
dd if=/dev/urandom bs=1 count=65537 | ssh localhost 'sed {} > /dev/null'
# STALLS
So pv, tee, and wc work, but cat, sed, awk get hung up.
I also tried a simple python 3.9 implementation of 'cat' which works,
and a gcc-compiled getchar/putchar one which stalls (which gives me
something to work with).
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: scp and ssh 'cat' stalls at 64k bytes
On 2023-06-23 22:35, Brian Inglis wrote:
On 2023-06-23 20:19, Dan Harkless via Cygwin wrote:
Before you resort to trawling through source, did you try a fresh
Cygwin install (either to a different directory, or after temporarily
moving your current tree)? Sometimes, e.g. if you use the same
"packages" directory between two different versions of Windows, I've
found that Cygwin can get screwed up in strange ways.
Good thought. Yes, one of my machines is a new Win11 with a fresh Cygwin.
On each system have you run /usr/bin/ssh-user-config, and run
/usr/bin/ssh-host-config from an elevated admin account, customized or
added host keys appropriately to /etc/sshd_config, /etc/ssh_config,
~/.ssh/{config,known_hosts,authorized_keys}, /etc/ssh/ssh_host_*_key.pub?
I think they're all set up correctly. ssh/slogin/rsync/scp all work
great and have been for years. It's only this weird issue in specific
cases with files > 64K.
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: scp and ssh 'cat' stalls at 64k bytes
On 2023-06-23 18:26, Dan Harkless via Cygwin wrote: On 6/23/2023 5:19 PM, Chris Roehrig via Cygwin wrote: No worries; I imagine most people don't run sshd on cygwin. Hmm, I'd generally think the opposite, at least for users coming from more UNIXey / Linuxey backgrounds. It looks to me like the issue involves i/o between sshd and its sub-processes on cygwin. Is anyone else able to successfully scp large files (>64KB) to cygwin as the remote receiving end (with the latest cygwin)? Yes, with all Cygwin64 updates, I was able to scp a file of a few MB from Linux to Windows 10 with no issues. I also tried your 'cat' version, and that worked fine too. -- Dan Harkless http://harkless.org/dan/ Thanks. There must be some issue with my setup. Very odd that 'pv' works, but 'cat' does not. ldd shows they use identical libs. I guess I'll start with the pv and cat source. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: scp and ssh 'cat' stalls at 64k bytes
No worries; I imagine most people don't run sshd on cygwin. It looks to
me like the issue involves i/o between sshd and its sub-processes on cygwin.
Is anyone else able to successfully scp large files (>64KB) to cygwin as
the remote receiving end (with the latest cygwin)?
On 2023-06-23 16:27, Voris, Ben wrote:
I did not and cannot. Sorry.
Fwiw, pushing from Cygwin to the remote also worked.
-Original Message-
From: Chris Roehrig
Sent: 23 June 2023 17:23
To: Voris, Ben
Subject: Re: scp and ssh 'cat' stalls at 64k bytes
Did you execute the scp on the remote (e.g. Linux) host? It only
fails when cygwin is the remote and target end (i.e. "pushing" to a
cygwin host); "pulling" works fine.
On 2023-06-23 16:07, Voris, Ben wrote:
Using scp from OpenSSH_9.3p1, 30 May 2023, copied a 8317351936 byte file from
a remote to Cygwin. sha1sum says it arrived correctly.
Windows Firewall is running.
-Original Message-----
From: Chris Roehrig
Sent: 23 June 2023 13:42
To: cygwin@cygwin.com
Subject: Re: scp and ssh 'cat' stalls at 64k bytes
On 2023-06-23 08:28, Brian Inglis wrote:
On 2023-06-23 00:26, Chris Roehrig via Cygwin wrote:
I've upgraded cygwin recently (from a much older version) and am
encountering a new problem on all my Win10/WIn11 machines.
With openssh and pv installed on cygwin (3.4.7-1):
dd if=/dev/zero bs=1 count=65536 | ssh localhost 'cat > /dev/null'
# works
dd if=/dev/zero bs=1 count=65537 | ssh localhost 'cat > /dev/null'
# stalls (and anything larger)
dd if=/dev/zero bs=1 count=65537 | ssh localhost 'pv > /dev/null'
# replace 'cat' with 'pv' and it works
This happens with or without Windows Firewall enabled, with any input
64k, and also remotely from Linux.
It also seems to affect scp to cygwin which stalls if the file is >= 64k
In case equivalent Windows services are running, I check for, shut
down, and disable the following Windows services I have seen, or seen
warnings about, before trying to start Cygwin services:
ssh ssh-agent sshbroker sshproxy sshd sshdbroker sshdproxy
Do you take similar precautions on the Windows systems with issues?
The only service listed is ssh-agent which is disabled and stopped.
TaskManager did show many zombie/stuck cat.exe and scp.exe which I
terminated, but that did not help. Nor did rebooting.
I use ssh, rsync, slogin frequently on these machines to/from windows
and linux and rsync (over ssh) gigabytes of stuff daily to/from Linux
and other windows machines with no issues. It is only this case of
reading more than 64K from stdin over ssh (and scp) that is hanging.
Please show at least the output of `uname -srvmo`,
CYGWIN_NT-10.0-19045 3.4.7-1.x86_64 2023-06-16 14:04 UTC x86_64 Cygwin
and if possible provide a text attachment of the output log from
running `cygcheck -hrsv > cygcheck.out`, as suggested under the
Problem reports link below.
I've attached the file.
I take it then you can't replicate this? Can you confirm whether this
command works successfully (i.e. exits without hanging):
dd if=/dev/zero bs=1 count=65537 | ssh localhost 'cat > /dev/null'
Are you able to 'scp' large files to cygwin?
It also hangs if I replace 'cat' with 'sed {}' or 'awk {print}', but not
with 'pv'. Those all passthrough stdin>stdout and should behave the
same as cat. (pv is basically cat with a progress bar which is omitted
if there's no controlling terminal like in this case).
The curious fact that 'pv' works makes me think it is a Cygwin library
problem and not anything related to firewall/network/Windows.
I've verified that cat, pv, and sshd all have the same ACLs. e.g.
aquila[77]% icacls pv.exe
pv.exe AQUILA\croehrig:(F)
BUILTIN\Administrators:(RX)
Everyone:(RX)
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: scp and ssh 'cat' stalls at 64k bytes
On 2023-06-23 08:28, Brian Inglis wrote:
On 2023-06-23 00:26, Chris Roehrig via Cygwin wrote:
I've upgraded cygwin recently (from a much older version) and am
encountering a new problem on all my Win10/WIn11 machines.
With openssh and pv installed on cygwin (3.4.7-1):
dd if=/dev/zero bs=1 count=65536 | ssh localhost 'cat > /dev/null'
# works
dd if=/dev/zero bs=1 count=65537 | ssh localhost 'cat > /dev/null'
# stalls (and anything larger)
dd if=/dev/zero bs=1 count=65537 | ssh localhost 'pv > /dev/null'
# replace 'cat' with 'pv' and it works
This happens with or without Windows Firewall enabled, with any input
> 64k, and also remotely from Linux.
It also seems to affect scp to cygwin which stalls if the file is >= 64k
In case equivalent Windows services are running, I check for, shut
down, and disable the following Windows services I have seen, or seen
warnings about, before trying to start Cygwin services:
ssh ssh-agent sshbroker sshproxy sshd sshdbroker sshdproxy
Do you take similar precautions on the Windows systems with issues?
The only service listed is ssh-agent which is disabled and stopped.
TaskManager did show many zombie/stuck cat.exe and scp.exe which I
terminated, but that did not help. Nor did rebooting.
I use ssh, rsync, slogin frequently on these machines to/from windows
and linux and rsync (over ssh) gigabytes of stuff daily to/from Linux
and other windows machines with no issues. It is only this case of
reading more than 64K from stdin over ssh (and scp) that is hanging.
Please show at least the output of `uname -srvmo`,
CYGWIN_NT-10.0-19045 3.4.7-1.x86_64 2023-06-16 14:04 UTC x86_64 Cygwin
and if possible provide a text attachment of the output log from
running `cygcheck -hrsv > cygcheck.out`, as suggested under the
Problem reports link below.
I've attached the file.
I take it then you can't replicate this? Can you confirm whether this
command works successfully (i.e. exits without hanging):
dd if=/dev/zero bs=1 count=65537 | ssh localhost 'cat > /dev/null'
Are you able to 'scp' large files to cygwin?
It also hangs if I replace 'cat' with 'sed {}' or 'awk {print}', but not
with 'pv'. Those all passthrough stdin>stdout and should behave the
same as cat. (pv is basically cat with a progress bar which is omitted
if there's no controlling terminal like in this case).
The curious fact that 'pv' works makes me think it is a Cygwin library
problem and not anything related to firewall/network/Windows.
I've verified that cat, pv, and sshd all have the same ACLs. e.g.
aquila[77]% icacls pv.exe
pv.exe AQUILA\croehrig:(F)
BUILTIN\Administrators:(RX)
Everyone:(RX)
Cygwin Configuration Diagnostics
Current System Time: Fri Jun 23 11:12:55 2023
Windows 10 Professional Ver 10.0 Build 19045
Path: C:\cygwin\home\croehrig\bin\WIN
C:\cygwin\home\croehrig\bin
C:\cygwin\usr\local\bin
C:\cygwin\bin
C:\Windows\system32
C:\Windows
Output from C:\cygwin\bin\id.exe
UID: 601(croehrig)
GID: 601(croehrig)
601(croehrig)
1099(wnone)
114(Local account and member of Administrators group)
80(admin)
1001(chrises)
1002(house)
544(Administrators)
545(Users)
2(NETWORK)
11(Authenticated Users)
15(This Organization)
113(Local account)
4095(CurrentSession)
262154(NTLM Authentication)
405504(High Mandatory Level)
SysDir: C:\Windows\system32
WinDir: C:\Windows
Here's some environment variables that may affect cygwin:
PWD = '/home/croehrig'
HOME = '/home/croehrig'
USER = 'croehrig'
Here's the rest of your environment variables:
ProgramFiles(x86) = 'C:\Program Files (x86)'
CommonProgramFiles(x86) = 'C:\Program Files (x86)\Common Files'
asl.log = 'Destination=file'
SHELL = '/bin/bash'
NUMBER_OF_PROCESSORS = '16'
PROCESSOR_LEVEL = '25'
LESS = '-X -R'
USERDOMAIN_ROAMINGPROFILE = 'AQUILA'
PROGRAMFILES = 'C:\Program Files'
PATHEXT = '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW'
OS = 'Windows_NT'
HOMEDRIVE = 'C:'
EDITOR = 'vim'
USERDOMAIN = 'AQUILA'
USERPROFILE = 'C:\Users\croehrig'
LOGNAME = 'croehrig'
VS140COMNTOOLS = 'C:\Program Files (x86)\Microsoft Visual Studio
14.0\Common7\Tools\'
CommonProgramFiles = 'C:\Program Files\Common Files'
ALLUSERSPROFILE = 'C:\ProgramData'
CommonProgramW6432 = 'C:\Program Files\Common Files'
USERNAME = 'croehrig'
OneDrive = 'C:\Users\croehrig\OneDrive'
COMSPEC = 'C:\Windows\system32\cmd.exe'
APPDATA = 'C:\Users\croehrig\AppData\Roaming'
SYSTEMROOT = 'C:\Windows'
LOCALAPPDATA = 'C:\Users\croehrig\AppData\Local'
SSH_CONNECTION = '192.168.32.4 34164 192.168.32.26 22'
COMPUTERNAME = 'AQUILA'
LOGONSERVER = '\\AQUILA'
ISROOT = 'yes'
PSModulePath =
'%ProgramFiles%\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules'
DISPLAY = ':0.0'
SHLVL = '0'
PROCESSOR_REVISION = '2102'
DriverData = 'C:\Windows\System32\Drivers\DriverData'
PROCESSOR_IDENTIFIER = 'AMD64 Family 25 Model 33 Stepping 2,
scp and ssh 'cat' stalls at 64k bytes
I've upgraded cygwin recently (from a much older version) and am encountering a new problem on all my Win10/WIn11 machines. With openssh and pv installed on cygwin (3.4.7-1): dd if=/dev/zero bs=1 count=65536 | ssh localhost 'cat > /dev/null' # works dd if=/dev/zero bs=1 count=65537 | ssh localhost 'cat > /dev/null' # stalls (and anything larger) dd if=/dev/zero bs=1 count=65537 | ssh localhost 'pv > /dev/null' # replace 'cat' with 'pv' and it works This happens with or without Windows Firewall enabled, with any input > 64k, and also remotely from Linux. It also seems to affect scp to cygwin which stalls if the file is >= 64k -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: SSH connection from Linux to Windows by CYGSSHD: port 22
I recently had to add the following lines to my Cygwin /etc/sshd_config to re-enable RSA in order for my older machines to connect: HostKeyAlgorithms +ssh-rsa PubkeyAcceptedAlgorithms +ssh-rsa -- Chris On 2022-03-31 06:18, Andrey Repin wrote: Greetings, Greco Giovanni! must port 22 on Windows server be enabled in a bidirectional way to establish a connection with RSA key exchange? I have a Linux server on a vlan and a Windows server on another vlan, those vlans are connected thru a firewall, where port 22 is enabled from Linux server to Windows server unidirectionally. Connection with user and password works, but not with RSA key exchange: is the problem located on port 22 unidirectional enabling? No, it is most likely because you are connecting to Microsoft provided OpenSSH. `netstat -aon` and `ps ax` will tell you more. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Help with standalone samba SID-uid mapping
On Mon Jan 17 2022, at 4:04 AM, Corinna Vinschen wrote: > On Jan 14 11:57, Chris Roehrig wrote: >> On Fri Jan 14 2022, at 2:04 AM, Corinna Vinschen >> wrote: >>> These look like your standard Windows SIDs, so they are your SIDs for >>> users cristina and croehrig on Windows. They should show up as such in >>> ls -l output, unless the SID is actuall wrong, e. g., they map to your >>> accounts on another machine or something like that. >> >> No those are the SIDs supplied by the Samba server (see below for my local >> Windows SIDs). Here they are directly on the Linux machine: >> housesrv[11]% smbcacls --numeric //housesrv/Users croehrig >> Enter WORKGROUP\croehrig's password: >> REVISION:1 >> CONTROL:0x9004 >> OWNER:S-1-5-21-751087815-2087572193-42305691-1000 >> GROUP:S-1-22-2-601 >> ACL:S-1-5-21-751087815-2087572193-42305691-1000:0/0x0/0x001f01ff >> ACL:S-1-22-2-601:0/0x0/0x001200a9 >> ACL:S-1-1-0:0/0x0/0x001200a9 >> >> (I think that Samba now uses a more complex IDMAP algorithm than when >> the Cygwin document above was written and now provides a full domain >> component to its SIDs.) > > That may be so, but in my installation, Samba reports the Unix User ID > as owner, i. e. > > $ icacls server\\corinna\\foo > \\server\corinna\foo S-1-22-1-500:(R,W,D,WDAC,WO) > S-1-22-2-11125:(R) > Everyone:(R) > > and that's with Samba 4.15.3. I'm doing the mapping via the AD > uidNumber and gidNumber fields. I'm using this setup for so long that I > don't remember if I ever saw a "normal", Windows-like SID for the user > returned by Samba. I never ran winbindd, up until Samba 4.15.3, which > was the first one forcing me to do so when using AD support. I'm no Samba expert, but maybe your /var/lib/samba/private/secrets.tdb file predates that IDMAP change...? What does 'net getdomainsid' say on your samba host? housesrv[2]% sudo net getdomainsid SID for local machine HOUSESRV is: S-1-5-21-751087815-2087572193-42305691 SID for domain WORKGROUP is: S-1-5-21-.. > >> I just added those SIDs to /etc/passwd and /etc/groups (double >> entries now) and it now works for the user, but (oddly) not the group: >> >> tyto[6]% ls -l //housesrv/Users/ ## NB: this is >> a UNC path to the samba share >> total 0 >> drwxr-xr-x 1 cristina Unix_Group+603 0 Jan 12 16:06 cristina >> drwxr-xr-x 1 croehrig Unix_Group+601 0 Jan 14 09:18 croehrig >> [...] >> tyto[10]% cat /etc/group >> croehrig:S-1-22-2-601:601: >> cristina:S-1-22-2-603:603: >> croehrig:S-1-5-21-1290748074-662758565-4273641972-1006:601: >> cristina:S-1-5-21-1290748074-662758565-4273641972-1008:603: > > Hmm, that's weird. I just tried this myself. First I created a stock > /etc/group file with all local and AD accounts. Next I changed > /etc/nsswitch.conf: > > - group: db > + group: files > > Exit/restart Cygwin. `ls -l' now prints > > -rw-r--r-- 1 corinna Unknown+Group 13342 Jan 17 10:46 //calimero/corinna/foo > > Now I add this line to /etc/group: > > mygroup:S-1-22-2-11125:11125: > > Exit/restart Cygwin. Now `ls -l' prints > > -rw-r--r-- 1 corinna mygroup 13342 Jan 17 10:46 //calimero/corinna/foo > > So it works, apparently. Did you set `group: db' in /etc/nsswitch.conf, > by any chance? That did the trick. My nsswitch.conf was the default (no lines; only comments), but everything works great now once I change it to group: files Seems odd that changing it back to 'group: files db' causes the groups to revert to the Unix_Group+601 form (as if the files weren't resolving it satisfactorily). Thanks for your help looking into this! [Update: cygsshd service no longer permits logins (closes connection immediately) when using 'group: files' (but it does work when running as /var/sbin/sshd -Dd).I'll have to get syslog-ng set up to try do debug this further...] > > > Corinna > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Help with standalone samba SID-uid mapping
On Fri Jan 14 2022, at 2:04 AM, Corinna Vinschen wrote: > On Jan 13 14:39, Chris Roehrig wrote: >> I'm trying to set up samba (standalone) following these instructions: >> https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-samba >> >> but I'm having no luck getting my samba user/groups to appear correctly >> using the comment field as described in the document. >> >> I'm using samba 4.13.14 on Ubuntu 20.04 with security = user (smbpasswd). >> winbindd is not installed and I'm not using any LDAP or AD anywhere. >> >> E.g. here is what is on the server (croehrig:croehrig = 601:601; >> cristina:cristina = 603:603) >> housesrv[3]% ls -l /House/Users >> total 17 >> drwxr-xr-x 9 cristina cristina 22 Jan 12 16:06 cristina >> drwxr-xr-x 30 croehrig croehrig 53 Jan 13 09:47 croehrig >> >> >> Here are the ACLs and SIDs when looking on the windows client: >> tyto[5]% icacls housesrv\\Users\\\* >> \\housesrv\Users\cristina S-1-5-21-751087815-2087572193-42305691-1001:(F) >> S-1-22-2-603:(RX) >> Everyone:(RX) >> >> \\housesrv\Users\croehrig S-1-5-21-751087815-2087572193-42305691-1000:(F) >> S-1-22-2-601:(RX) >> Everyone:(RX) >> >> As you can see, the gid is mapping to the S-1-22-2- as described >> in the document above, but the uid is using a domain-specific SID with >> different RIDs. > > These look like your standard Windows SIDs, so they are your SIDs for > users cristina and croehrig on Windows. They should show up as such in > ls -l output, unless the SID is actuall wrong, e. g., they map to your > accounts on another machine or something like that. No those are the SIDs supplied by the Samba server (see below for my local Windows SIDs). Here they are directly on the Linux machine: housesrv[11]% smbcacls --numeric //housesrv/Users croehrig Enter WORKGROUP\croehrig's password: REVISION:1 CONTROL:0x9004 OWNER:S-1-5-21-751087815-2087572193-42305691-1000 GROUP:S-1-22-2-601 ACL:S-1-5-21-751087815-2087572193-42305691-1000:0/0x0/0x001f01ff ACL:S-1-22-2-601:0/0x0/0x001200a9 ACL:S-1-1-0:0/0x0/0x001200a9 (I think that Samba now uses a more complex IDMAP algorithm than when the Cygwin document above was written and now provides a full domain component to its SIDs.) I just added those SIDs to /etc/passwd and /etc/groups (double entries now) and it now works for the user, but (oddly) not the group: tyto[6]% ls -l //housesrv/Users/## NB: this is a UNC path to the samba share total 0 drwxr-xr-x 1 cristina Unix_Group+603 0 Jan 12 16:06 cristina drwxr-xr-x 1 croehrig Unix_Group+601 0 Jan 14 09:18 croehrig NB: I'm only having issues with the files on the samba share. All my local files work fine and correctly show the user/group. Here are the SIDs on my local Windows machine (they are different on each of my Windows clients): tyto[7]% wmic useraccount get name,sid | grep cr cristinaS-1-5-21-1290748074-662758565-4273641972-1007 croehrigS-1-5-21-1290748074-662758565-4273641972-1002 tyto[8]% wmic group get name,sid | grep cr grp-cristina S-1-5-21-1290748074-662758565-4273641972-1008 grp-croehrig S-1-5-21-1290748074-662758565-4273641972-1006 tyto[9]% cat /etc/passwd croehrig:*:601:601:HOUSESRV\croehrig,S-1-5-21-751087815-2087572193-42305691-1000:/home/croehrig:/bin/bash cristina:*:603:603:HOUSESRV\cristina,S-1-5-21-751087815-2087572193-42305691-1001:/home/cristina:/bin/bash croehrig:*:601:601:U-TYTO\croehrig,S-1-5-21-1290748074-662758565-4273641972-1002:/home/croehrig:/bin/bash cristina:*:603:603:U-TYTO\cristina,S-1-5-21-1290748074-662758565-4273641972-1007:/home/cristina:/bin/bash tyto[10]% cat /etc/group croehrig:S-1-22-2-601:601: cristina:S-1-22-2-603:603: croehrig:S-1-5-21-1290748074-662758565-4273641972-1006:601: cristina:S-1-5-21-1290748074-662758565-4273641972-1008:603: admin:S-1-5-21-1290748074-662758565-4273641972-1004:80: chrises:S-1-5-21-1290748074-662758565-4273641972-1003:1001: house:S-1-5-21-1290748074-662758565-4273641972-1005:1002: nobody:S-1-5-21-1290748074-662758565-4273641972-513:99: Any idea why those first two /etc/group entries are not working?The rest work fine (on local files only of course). I tried temporarily deleting the 3rd and 4th entries in case there was an issue with double entries but it made no difference (even after restarting all cygwin processes). > >> On the windows client I have the same users and groups set up locally >> (SAM) with appropriate SID mappings to the same uid/gids (601/603) in >> the Cygwin /etc/passwd and /etc/group. This has all been working >> well to ensure e.g.
Help with standalone samba SID-uid mapping
I'm trying to set up samba (standalone) following these instructions:
https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-samba
but I'm having no luck getting my samba user/groups to appear correctly using
the comment field as described in the document.
I'm using samba 4.13.14 on Ubuntu 20.04 with security = user (smbpasswd).
winbindd is not installed and I'm not using any LDAP or AD anywhere.
E.g. here is what is on the server (croehrig:croehrig = 601:601;
cristina:cristina = 603:603)
housesrv[3]% ls -l /House/Users
total 17
drwxr-xr-x 9 cristina cristina 22 Jan 12 16:06 cristina
drwxr-xr-x 30 croehrig croehrig 53 Jan 13 09:47 croehrig
Here are the ACLs and SIDs when looking on the windows client:
tyto[5]% icacls housesrv\\Users\\\*
\\housesrv\Users\cristina S-1-5-21-751087815-2087572193-42305691-1001:(F)
S-1-22-2-603:(RX)
Everyone:(RX)
\\housesrv\Users\croehrig S-1-5-21-751087815-2087572193-42305691-1000:(F)
S-1-22-2-601:(RX)
Everyone:(RX)
As you can see, the gid is mapping to the S-1-22-2- as described in the
document above, but
the uid is using a domain-specific SID with different RIDs.
On the windows client I have the same users and groups set up locally (SAM)
with appropriate SID mappings to the same uid/gids (601/603) in the Cygwin
/etc/passwd and /etc/group. This has all been working well to ensure e.g.
rsync preserves permissions and ownership between cygwin and Linux. (The
windows groups are called 'grp-croehrig' and 'grp-cristina' since windows users
and groups share a namespace, but they are mapped to 'croehrig' and 'cristina'
in /etc/group).
Here is how the SMB share looks under Cygwin:
tyto[6]% ls -l //housesrv/Users/
total 0
drwxr-xr-x 1 Unknown+User Unix_Group+603 0 Jan 12 16:06 cristina
drwxr-xr-x 1 Unknown+User Unix_Group+601 0 Jan 13 09:47 croehrig
tyto[7]% ls -ln //housesrv/Users/
total 0
drwxr-xr-x 1 4294967295 4278190683 0 Jan 12 16:06 cristina
drwxr-xr-x 1 4294967295 4278190681 0 Jan 13 09:47 croehrig
I have added the SAM desc/comment field as described in the document above:
i.e.
net localgroup grp-croehrig /comment:''
net user croehrig /comment:''
and restarted all Cygwin processes, but it doesn't seem to have any effect
('net user croehrig' shows the comment is indeed present). I guess I don't
understand how that comment field works.
Anyone have any advice?
Thanks,
-- Chris
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
cat fifo hang [Re: [ANNOUNCEMENT] cygwin 3.3.0-0.2.6c1f49f83fde (TEST)]
On Sun Oct 17 2021, at 9:19 AM, Ken Brown via Cygwin wrote:
> On 10/16/2021 1:42 PM, Chris Roehrig wrote:
>> On Mon Sep 27 2021, at 7:26 AM, Ken Brown via Cygwin
>> wrote:
>>> On 9/26/2021 8:57 PM, Chris Roehrig wrote:
>>>> I have installed this (completely this time) and have encountered no
>>>> issues with it. I'm getting full gigabit speeds with my rsync transfers.
>>>> Looks great!
>>>
>>> Thanks for testing.
>> I've encountered a crash that might be related. I had previously been
>> having occasional crashes/hangs of cat.exe over the years, but this is the
>> first time I've ever gotten an error message:
>> cygwin error: 0 [fifo_reader] cat 11398 C:\cygwin\bin\cat.exe: *** fatal
>> error - Can't add a client handler, Win32 error 123
>
> This isn't a crash in the usual sense. It's the Cygwin fifo code issuing a
> fatal error because an attempt to create a new Windows pipe instance failed.
> And it's in code that's been around for a while, so it's not related to the
> new pipe implementation.
>
>> cat here is reading from a fifo created with mkfifo.
>> I've only encountered it once (out of daily runs over the last couple weeks)
>> and don't know how to replicate it. Possibly a race?Looks like my
>> script has tried to mitigate this with a sleep 1 between the mkfifo and the
>> fork: cat < $fifo &
>
> The sleep shouldn't be necessary. If it is, there's a bug in the fifo code.
> Can you remove the sleep and see what happens? It would be great if that
> made it possible to replicate the problem.
Here's a script that pretty reliably hangs cat after some iterations.I
haven't yet gotten a repeat of that error message though.
It runs fine on Ubuntu 20.04 and Mac OS X 10.8.4.
#!/bin/bash
# take arg as number of iterations (default=100)
STEPS="${1-100}"
FIFO_PFX="/tmp/catfifo_"
FIFO_WAIT=0
STEP_WAIT=0
function mysleep() { if [ -n "$1" -a "$1" != "0" ]; then sleep "$1"; fi }
function cleanup(){
rm -f "$FIFO_PFX"*
}
trap cleanup EXIT
printf "Creating $STEPS fifo readers...\n"
for ((i=0; i"$fifo"
printf "FIFO %d\n" "$i" >&3
# close the file descriptor, wait for process to exit and clean up
exec 3>&-
wait $pid
rm -f "$fifo"
mysleep $STEP_WAIT
done
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [ANNOUNCEMENT] cygwin 3.3.0-0.2.6c1f49f83fde (TEST)
On Mon Sep 27 2021, at 7:26 AM, Ken Brown via Cygwin wrote: > On 9/26/2021 8:57 PM, Chris Roehrig wrote: >> I have installed this (completely this time) and have encountered no issues >> with it. I'm getting full gigabit speeds with my rsync transfers. Looks >> great! > > Thanks for testing. I've encountered a crash that might be related. I had previously been having occasional crashes/hangs of cat.exe over the years, but this is the first time I've ever gotten an error message: cygwin error: 0 [fifo_reader] cat 11398 C:\cygwin\bin\cat.exe: *** fatal error - Can't add a client handler, Win32 error 123 cat here is reading from a fifo created with mkfifo. I've only encountered it once (out of daily runs over the last couple weeks) and don't know how to replicate it. Possibly a race?Looks like my script has tried to mitigate this with a sleep 1 between the mkfifo and the fork: cat < $fifo & I only encounter this issue on Cygwin (it has always worked fine without the sleep on Linux and MacOS). -- Chris -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [ANNOUNCEMENT] cygwin 3.3.0-0.2.6c1f49f83fde (TEST)
I have installed this (completely this time) and have encountered no issues with it. I'm getting full gigabit speeds with my rsync transfers. Looks great! -- Chris On Fri Sep 24 2021, at 6:53 PM, Ken Brown via Cygwin-announce via Cygwin wrote: > The following packages have been uploaded to the Cygwin distribution > as test releases: > > * cygwin-3.3.0-0.2.6c1f49f83fde > * cygwin-devel-3.3.0-0.2.6c1f49f83fde > * cygwin-doc-3.3.0-0.2.6c1f49f83fde > > This is the second test release of the upcoming cygwin-3.3.0. The > only change from the first test release is that there have been > further changes to the internal pipe code. > > Please test! > > === > > What's new: > --- > > - An IP-sampling profiler named 'profiler' has been added. It can be > used to profile any Cygwin program along with any DLLs loaded. > > - A new tool 'gmondump' has been added. It can dump the raw > information of any "gmon.out" file created by profiler, ssp, or use > of the gcc/g++ option '-pg'. (Continue using gprof to get symbolic > profile displays.) > > - New GNU-specific APIs, slated to become part of the next POSIX > standard: pthread_cond_clockwait, pthread_mutex_clocklock, > pthread_rwlock_clockrdlock, pthread_rwlock_clockwrlock, > sem_clockwait. > > - New Solaris-specific APIs, slated to become part of the next POSIX > standard: sig2str, str2sig. > > > What changed: > - > > - The speed argument to cfsetspeed(3) can now be a numerical baud rate > rather than a Bnnn constant, as on Linux. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248887.html > > - The internal implementation of pipes has been overhauled; this > should result in improved performance. > Addresses: https://cygwin.com/pipermail/cygwin/2021-August/249238.html > > > Bug Fixes > - > > - Fix values returned by select(2) for shutdown sockets. > Addresses: > https://cygwin.com/pipermail/cygwin-developers/2021-April/012092.html > > - Introduce a new hypotl(3) function not suffering unnecessary > overflows. > Addresses: https://cygwin.com/pipermail/cygwin/2021-April/248302.html > > - Fix path handling for paths spanning native symlinks. > Addresses: https://cygwin.com/pipermail/cygwin/2021-April/248307.html > > - Fix tab position evaluation after console window resize. > > - Fix a regression in pseudo console handling, resulting in rlwrap not > being able to start a new pseudo console. > > - Handle two race conditions in pseudo console usage. > Addresses: https://cygwin.com/pipermail/cygwin/2021-April/248292.html > > - Fix a bug in recognizing a successful completion of connect(2) on a > datagram socket. > > - Fix connect(2) when called with an address structure whose family is > AF_UNSPEC. As specified by POSIX and Linux, this is allowed on > datagram sockets, and its effect is to reset the socket's peer > address. > > - Fix nanosleep(2) returning negative rem. NtQueryTimer appears to be > able to return a negative remaining time (less than the timer > resolution) for unsignalled timers. > > - Fix getifaddrs(3) returning address family 0 or IPv4 address 0. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248970.html > > - Fix getaddrinfo(3) to return valid ai_socktype and ai_protocol > values if the underlying GetAddrInfoW screws up. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248985.html > > - Fix duplicate /proc/partitions entries and (presumably) duplicate > PIDs in ps(1) output. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248998.html > https://cygwin.com/pipermail/cygwin/2021-August/249124.html > > === > > > Have fun, > > Ken Brown, on behalf of Corinna > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [ANNOUNCEMENT] cygwin 3.3.0-0.1.9814cfd8f693 (TEST)
I'm not sure how to try out a test release without risking destabilizing my system, but I re-cloned and rebuilt the master branch and installed just the cygwin1.dll and all looks good to me so far. No issues with any of my script pipes or fifos and I am getting full gigabit speeds with rsync. I too have been following all the pipe rework on Cygwin-developers and want to express my appreciation for Takashi, Ken and Corinna for tackling this.You folks have made Cygwin so much more usable for a lot of applications! -- Chris On Fri Sep 17 2021, at 1:33 PM, Ken Brown via Cygwin-announce via Cygwin wrote: > The following packages have been uploaded to the Cygwin distribution as test > releases: > > * cygwin-3.3.0-0.1.9814cfd8f693 > * cygwin-devel-3.3.0-0.1.9814cfd8f693 > * cygwin-doc-3.3.0-0.1.9814cfd8f693 > > This release comes with an overhaul of the internal pipe code. In theory, > there should be no user-visible changes except for bug fixes and performance > improvements. But of course there will be new bugs. Please test! > > === > > What's new: > --- > > - An IP-sampling profiler named 'profiler' has been added. It can be > used to profile any Cygwin program along with any DLLs loaded. > > - A new tool 'gmondump' has been added. It can dump the raw > information of any "gmon.out" file created by profiler, ssp, or use > of the gcc/g++ option '-pg'. (Continue using gprof to get symbolic > profile displays.) > > - New GNU-specific APIs, slated to become part of the next POSIX > standard: pthread_cond_clockwait, pthread_mutex_clocklock, > pthread_rwlock_clockrdlock, pthread_rwlock_clockwrlock, > sem_clockwait. > > - New Solaris-specific APIs, slated to become part of the next POSIX > standard: sig2str, str2sig. > > > What changed: > - > > - The speed argument to cfsetspeed(3) can now be a numerical baud rate > rather than a Bnnn constant, as on Linux. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248887.html > > - The internal implementation of pipes has been overhauled; this > should result in improved performance. > Addresses: https://cygwin.com/pipermail/cygwin/2021-August/249238.html > > > Bug Fixes > - > > - Fix values returned by select(2) for shutdown sockets. > Addresses: > https://cygwin.com/pipermail/cygwin-developers/2021-April/012092.html > > - Introduce a new hypotl(3) function not suffering unnecessary > overflows. > Addresses: https://cygwin.com/pipermail/cygwin/2021-April/248302.html > > - Fix path handling for paths spanning native symlinks. > Addresses: https://cygwin.com/pipermail/cygwin/2021-April/248307.html > > - Fix tab position evaluation after console window resize. > > - Fix a regression in pseudo console handling, resulting in rlwrap not > being able to start a new pseudo console. > > - Handle two race conditions in pseudo console usage. > Addresses: https://cygwin.com/pipermail/cygwin/2021-April/248292.html > > - Fix a bug in recognizing a successful completion of connect(2) on a > datagram socket. > > - Fix connect(2) when called with an address structure whose family is > AF_UNSPEC. As specified by POSIX and Linux, this is allowed on > datagram sockets, and its effect is to reset the socket's peer > address. > > - Fix nanosleep(2) returning negative rem. NtQueryTimer appears to be > able to return a negative remaining time (less than the timer > resolution) for unsignalled timers. > > - Fix getifaddrs(3) returning address family 0 or IPv4 address 0. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248970.html > > - Fix getaddrinfo(3) to return valid ai_socktype and ai_protocol > values if the underlying GetAddrInfoW screws up. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248985.html > > - Fix duplicate /proc/partitions entries and (presumably) duplicate > PIDs in ps(1) output. > Addresses: https://cygwin.com/pipermail/cygwin/2021-July/248998.html > https://cygwin.com/pipermail/cygwin/2021-August/249124.html > > === > > > Have fun, > > Ken Brown, on behalf of Corinna > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Fri Sep 3 2021, at 12:55 PM, Corinna Vinschen wrote: > [resent, this time with the ML in To] > > On Sep 2 12:03, Chris Roehrig wrote: >> >> On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin >> wrote: >>> On 9/1/2021 5:11 PM, Chris Roehrig wrote: >>>> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so >>>> maybe the stock procps package is incompatible with the current master >>>> branch. >>> >>> Maybe, but it could also be a Cygwin bug. I'll do a bisection of the >>> Cygwin sources to see if I can track it down. >> >> I did some more tests and it still doesn't completely work: >> procps -ef # works >> procps -eo user,stime,tty,time,args # works >> procps -eo pid # fails with that same mmap() error >> >> I also rebuilt it all using cygport and it gives the same error >> (pscommand.exe with no args). > > Since you're building Cygwin by yourself anyway, can you do me a favor > and try this: > > $ git revert 2f05de4dbf9c > > and see if that fixes your issue? I got procps working I think (both with and without the revert). I think the problem might just be that I wasn't also copying the rebuilt /bin/cygprocps-8.dll to match the /bin/procps.exe.There's some tricky renaming that make install does, so I did 'make install DESTDIR=/tmp/install' and copied just those two files. I'm guessing it would all work on a properly fully installed build. -- Chris -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin wrote: > On 9/1/2021 5:11 PM, Chris Roehrig wrote: >> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe >> the stock procps package is incompatible with the current master branch. > > Maybe, but it could also be a Cygwin bug. I'll do a bisection of the Cygwin > sources to see if I can track it down. I did some more tests and it still doesn't completely work: procps -ef # works procps -eo user,stime,tty,time,args # works procps -eo pid # fails with that same mmap() error I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args). > P.S. The custom on this list is not to top-post. Thanks. Sorry, I didn't realize there was an etiquette.Fixed. -- Chris -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatibility with the current master branch. (However, I built it against the stock /usr/include, not the current branch...) I first needed to 'make /proc/libprocps.la', and there was an undefined siginfo_t.si_int in ./lib/test_process.c which I just commented-out to get it to build. -- Chris On Tue Aug 31 2021, at 1:23 PM, Chris Roehrig wrote: > I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed > get 100MB/s transfers using both rsync and scp (without pipe_byte). > (It turns out last time I forgot 'make install' -- Doh!) > > I still get the procps error however. > > > On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig wrote: > >> Thanks, I did some more tests: >> scp also shows no improvement with topic/pipe.I tried running >> cygsshd with CYGWIN=pipe_byte as well as empty (in the registry >> HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), >> using net stop cygsshd + net start cygsshd to restart it. Not sure if that >> CYGWIN only applies to cygsshd or to all cygwin tasks. >> >> I get the procps error just running 'procps' command with no args from >> mintty (/usr/bin/procps). Looking at the procps-ng source (procps-ng >> 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears >> to be a mmap() failure. >> >> I'm wondering if I built/installed cygwin1.dll correctly because I don't get >> the procps error using the latest stock cygwin1.dll as installed by >> setup_x86_64. >> >> -- Chris >> >> >> >> On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin >> wrote: >> >>> On 8/30/2021 7:58 PM, Chris Roehrig wrote: >>>> I got it to build and tried out the topic/pipe branch (checked out on >>>> Monday around 4:30pm PDT): >>>> 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. >>>> 2. I get the following error from procps: procps:ps/output.c:2195: >>>> please report this bug >>>> (I also get this using the main branch build). >>>> I first updated my cygwin normally using setup-x86_64.exe, then built and >>>> copied only the cygwin1.dll into /bin from the install/bin directory >>>> (quitting all cygwin tasks first). >>> >>> Thanks for testing. >>> >>> We're still working on this. (The discussion has moved to the >>> cygwin-developers mailing list.) I'll let you know when it's stable, and >>> maybe you can try again. >>> >>> As to the procps error, can you give details so that someone can try to >>> reproduce it and debug it? >>> >>> Ken >>> >>> -- >>> Problem reports: https://cygwin.com/problems.html >>> FAQ: https://cygwin.com/faq/ >>> Documentation:https://cygwin.com/docs.html >>> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple >> >> >> -- >> Problem reports: https://cygwin.com/problems.html >> FAQ: https://cygwin.com/faq/ >> Documentation:https://cygwin.com/docs.html >> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed get 100MB/s transfers using both rsync and scp (without pipe_byte). (It turns out last time I forgot 'make install' -- Doh!) I still get the procps error however. On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig wrote: > Thanks, I did some more tests: > scp also shows no improvement with topic/pipe.I tried running > cygsshd with CYGWIN=pipe_byte as well as empty (in the registry > HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), > using net stop cygsshd + net start cygsshd to restart it. Not sure if that > CYGWIN only applies to cygsshd or to all cygwin tasks. > > I get the procps error just running 'procps' command with no args from mintty > (/usr/bin/procps). Looking at the procps-ng source (procps-ng 3.3.17-1 is > what setup_x86_64 reports as my installed package), it appears to be a mmap() > failure. > > I'm wondering if I built/installed cygwin1.dll correctly because I don't get > the procps error using the latest stock cygwin1.dll as installed by > setup_x86_64. > > -- Chris > > > > On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin > wrote: > >> On 8/30/2021 7:58 PM, Chris Roehrig wrote: >>> I got it to build and tried out the topic/pipe branch (checked out on >>> Monday around 4:30pm PDT): >>> 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. >>> 2. I get the following error from procps: procps:ps/output.c:2195: >>> please report this bug >>> (I also get this using the main branch build). >>> I first updated my cygwin normally using setup-x86_64.exe, then built and >>> copied only the cygwin1.dll into /bin from the install/bin directory >>> (quitting all cygwin tasks first). >> >> Thanks for testing. >> >> We're still working on this. (The discussion has moved to the >> cygwin-developers mailing list.) I'll let you know when it's stable, and >> maybe you can try again. >> >> As to the procps error, can you give details so that someone can try to >> reproduce it and debug it? >> >> Ken >> >> -- >> Problem reports: https://cygwin.com/problems.html >> FAQ: https://cygwin.com/faq/ >> Documentation:https://cygwin.com/docs.html >> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Thanks, I did some more tests: scp also shows no improvement with topic/pipe.I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it. Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks. I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps). Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure. I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_x86_64. -- Chris On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin wrote: > On 8/30/2021 7:58 PM, Chris Roehrig wrote: >> I got it to build and tried out the topic/pipe branch (checked out on Monday >> around 4:30pm PDT): >> 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. >> 2. I get the following error from procps: procps:ps/output.c:2195: please >> report this bug >> (I also get this using the main branch build). >> I first updated my cygwin normally using setup-x86_64.exe, then built and >> copied only the cygwin1.dll into /bin from the install/bin directory >> (quitting all cygwin tasks first). > > Thanks for testing. > > We're still working on this. (The discussion has moved to the > cygwin-developers mailing list.) I'll let you know when it's stable, and > maybe you can try again. > > As to the procps error, can you give details so that someone can try to > reproduce it and debug it? > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT): 1. I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec. 2. I get the following error from procps: procps:ps/output.c:2195: please report this bug (I also get this using the main branch build). I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first). On Sun Aug 29 2021, at 3:24 PM, Ken Brown via Cygwin wrote: > On 8/29/2021 3:24 PM, Chris Roehrig wrote: >> I'd be happy to test it out if you can give me some commands to git it and >> build it to replace my existing stock Cygwin installation. >> (Or it is just the cygwin.dll I'd need to replace?) >> My daily backup scripts use a lot of pipes and named pipes. > > It's just cygwin1.dll that you'd need to replace. Instructions for building > the latter can be found at > > https://www.cygwin.com/faq.html#faq.programming.building-cygwin > > but they're out of date in two respects: > > 1. Currently the build does *not* ignore errors in building the documentation. > > 2. In addition to the packages listed in the FAQ, you'll need the following: > > texlive-collection-latexrecommended > texlive-collection-fontsrecommended > texlive-collection-pictures > python38-lxml > python38-ply > > These instructions will produce cygwin1.dll in the bin subdirectory of your > install directory, built from the git master branch. > > If you want to try the topic/pipe branch, just go to the newlib-cygwin git > repo that you cloned, issue the command 'git checkout topic/pipe', and > rebuild. > > I'll be glad to help if you run into problems. > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I'd be happy to test it out if you can give me some commands to git it and build it to replace my existing stock Cygwin installation. (Or it is just the cygwin.dll I'd need to replace?) My daily backup scripts use a lot of pipes and named pipes. -- Chris On Sun Aug 29 2021, at 8:57 AM, Ken Brown via Cygwin wrote: > > Aside from that, I'm wondering how and when to merge the new pipe > implementation to master. It obviously needs much more widespread testing > than it's gotten so far. I'm a little nervous about it because I haven't > thought about the details for two years, and no one other than me has tested > it until a few days ago. > > Ken > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
It looks like there was a previous (2013) patch and attempt to add a pipe_nooverlap CYGWIN option that was rejected by the maintainers: https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app Is this something that can be revisited?It seems to be affecting pure-UNIX situations (in my case) which is Cygwin's core audience. - Chris On Wed Aug 25 2021, at 11:18 AM, Chris Roehrig wrote: > On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin > wrote: >> A couple years ago I had an idea for changing the pipe implementation to >> avoid overlapped I/O: >> >> https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html >> https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html >> >> I never followed up on it. But if you think it might help with this >> problem, I could dust it off and try to finish it. >> >> Ken > > I'm not familiar enough with the innards of rsync, sshd or cygwin to know how > this would work. > Is it possible to have a new CYGWIN environment option to switch the pipe > behaviour without requiring changes to the ssh or rsync source code (and > without breaking any existing stuff)? > > - Chris > > > >> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: >>> On Tue, 24 Aug 2021 12:49:52 -0700 >>> Chris Roehrig wrote: >>>> I have a network of Windows, Linux and Mac machines and I use rsync to >>>> synchronize various directories between them. >>>> >>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >>>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >>>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >>>> expected from gigabit ethernet. This has been an ongoing problem for me >>>> for a couple of years over several Windows and Cygwin versions, and I'd >>>> like to try to fix it. >>>> >>>> If I run rsync --daemon --no-detach under mintty in the foreground on the >>>> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >>>> like it has something to do with rsync.exe running in the background under >>>> the cygrunsrv+sshd service (which was installed normally using >>>> ssh-host-config). >>>> >>>> If I do: >>>>pv /dev/zero | ssh $WINHOST "cat > /dev/null" >>>> or even >>>>pv /dev/urandom | ssh $WINHOST md5sum >>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd >>>> itself is being throttled by bandwidth or CPU. >>>> >>>> The machines have less than 15% CPU utilization while transferring, with >>>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >>>> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >>>> Setting their Priority to High doesn't seem to change things. >>>> >>>> Looking in Resource Monitor on the remote endpoint, the network usage is >>>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >>>> looks to me as if rsync is somehow being bandwidth-throttled when run in >>>> the background under cygsshd. >>>> >>>> It's almost as if rsync has an implicit --bwlimit override when it is run >>>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >>>> difference). >>>> >>>> >>>> Any ideas?Not sure where to go from here. >>> In cygwin, just scp is very slow. >>> The transfer speed in my environment is as follows. >>> The tests were done with 100MB of test.dat file. >>> (1-1) From cygwin-PC, >>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >>> yano@linux-server's password: >>> test.dat 100% 100MB 4.0MB/s 00:24 >>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >>> yano@linux-server's password: >>> test.dat 100% 100MB 8.0MB/s 00:12 >>> (1-2) From linux-server, >>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 4.0MB/s 00:24 >>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >>> yano@cygwin-PC's password: >>>
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin wrote: > A couple years ago I had an idea for changing the pipe implementation to > avoid overlapped I/O: > > https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html > https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html > > I never followed up on it. But if you think it might help with this problem, > I could dust it off and try to finish it. > > Ken I'm not familiar enough with the innards of rsync, sshd or cygwin to know how this would work. Is it possible to have a new CYGWIN environment option to switch the pipe behaviour without requiring changes to the ssh or rsync source code (and without breaking any existing stuff)? - Chris > On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: >> On Tue, 24 Aug 2021 12:49:52 -0700 >> Chris Roehrig wrote: >>> I have a network of Windows, Linux and Mac machines and I use rsync to >>> synchronize various directories between them. >>> >>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >>> expected from gigabit ethernet. This has been an ongoing problem for me >>> for a couple of years over several Windows and Cygwin versions, and I'd >>> like to try to fix it. >>> >>> If I run rsync --daemon --no-detach under mintty in the foreground on the >>> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >>> like it has something to do with rsync.exe running in the background under >>> the cygrunsrv+sshd service (which was installed normally using >>> ssh-host-config). >>> >>> If I do: >>> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >>> or even >>> pv /dev/urandom | ssh $WINHOST md5sum >>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >>> is being throttled by bandwidth or CPU. >>> >>> The machines have less than 15% CPU utilization while transferring, with >>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >>> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >>> Setting their Priority to High doesn't seem to change things. >>> >>> Looking in Resource Monitor on the remote endpoint, the network usage is >>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >>> looks to me as if rsync is somehow being bandwidth-throttled when run in >>> the background under cygsshd. >>> >>> It's almost as if rsync has an implicit --bwlimit override when it is run >>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >>> difference). >>> >>> >>> Any ideas?Not sure where to go from here. >> In cygwin, just scp is very slow. >> The transfer speed in my environment is as follows. >> The tests were done with 100MB of test.dat file. >> (1-1) From cygwin-PC, >> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >> yano@linux-server's password: >> test.dat 100% 100MB 4.0MB/s 00:24 >> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >> yano@linux-server's password: >> test.dat 100% 100MB 8.0MB/s 00:12 >> (1-2) From linux-server, >> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >> yano@cygwin-PC's password: >> test.dat 100% 100MB 4.0MB/s 00:24 >> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >> yano@cygwin-PC's password: >> test.dat 100% 100MB 4.1MB/s 00:24 >> I looked into this problem, and noticed that this is caused >> by cygwin pipe implementation. Pipe in cygwin is configured >> with FILE_FLAG_OVERLAPPED. >> If the pipe is configured without FILE_FLAG_OVERLAPPED, >> the transfer speed is much improved as follows. >> (2-1) From cygwin-PC, >> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >> yano@linux-server's password: >> test.dat 100% 100MB 85.5MB/s 00:01 >> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >> yano@linux-server's password: >> test.dat 100% 100MB 69.7MB/s 00:01 >> (2-2) From linux-server
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
The FILE_FLAG_OVERLAPPED avenue looks promising. I get exactly the same results as you using 'scp': 4MB/s in either direction (with either remote endpoint.) I guess the next step is to install a build environment and build rsync and sshd ... On Wed Aug 25 2021, at 4:18 AM, Takashi Yano via Cygwin wrote: > On Tue, 24 Aug 2021 12:49:52 -0700 > Chris Roehrig wrote: >> I have a network of Windows, Linux and Mac machines and I use rsync to >> synchronize various directories between them. >> >> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >> expected from gigabit ethernet. This has been an ongoing problem for me for >> a couple of years over several Windows and Cygwin versions, and I'd like to >> try to fix it. >> >> If I run rsync --daemon --no-detach under mintty in the foreground on the >> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >> like it has something to do with rsync.exe running in the background under >> the cygrunsrv+sshd service (which was installed normally using >> ssh-host-config). >> >> If I do: >> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >> or even >> pv /dev/urandom | ssh $WINHOST md5sum >> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >> is being throttled by bandwidth or CPU. >> >> The machines have less than 15% CPU utilization while transferring, with >> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >> Setting their Priority to High doesn't seem to change things. >> >> Looking in Resource Monitor on the remote endpoint, the network usage is >> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >> looks to me as if rsync is somehow being bandwidth-throttled when run in >> the background under cygsshd. >> >> It's almost as if rsync has an implicit --bwlimit override when it is run >> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >> difference). >> >> >> Any ideas?Not sure where to go from here. > > In cygwin, just scp is very slow. > > The transfer speed in my environment is as follows. > The tests were done with 100MB of test.dat file. > > (1-1) From cygwin-PC, > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > yano@linux-server's password: > test.dat 100% 100MB 4.0MB/s 00:24 > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > yano@linux-server's password: > test.dat 100% 100MB 8.0MB/s 00:12 > > (1-2) From linux-server, > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > yano@cygwin-PC's password: > test.dat 100% 100MB 4.0MB/s 00:24 > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > yano@cygwin-PC's password: > test.dat 100% 100MB 4.1MB/s 00:24 > > > I looked into this problem, and noticed that this is caused > by cygwin pipe implementation. Pipe in cygwin is configured > with FILE_FLAG_OVERLAPPED. > > If the pipe is configured without FILE_FLAG_OVERLAPPED, > the transfer speed is much improved as follows. > > > (2-1) From cygwin-PC, > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > yano@linux-server's password: > test.dat 100% 100MB 85.5MB/s 00:01 > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > yano@linux-server's password: > test.dat 100% 100MB 69.7MB/s 00:01 > > (2-2) From linux-server, > yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > yano@cygwin-PC's password: > test.dat 100% 100MB 80.1MB/s 00:01 > yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > yano@cygwin-PC's password: > test.dat 100% 100MB 57.7MB/s 00:01 > > I am not sure why this happens and how to fix this. > Any idea? > > -- > Takashi Yano > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
It was set to "Programs".Changing it to "Background services" didn't make a difference. TCPOptimizer can adjust 2 registry entries that I think are related to that Windows Setting: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile] "NetworkThrottlingIndex"=dword:000a "SystemResponsiveness"=dword:0014 The first rate-limits IP packets; the second reserves a percent (0-100) of CPU scheduling for background tasks. Changing either of those values made no difference. I think the CPU scheduling reservation only applies when there's CPU contention (not on an idle system) so I wouldn't expect it to make a difference when CPU/core usage is low. On Tue Aug 24 2021, at 8:20 PM, Mark Geisert wrote: > NightStrike via Cygwin wrote: >> Older versions of windows had a setting to optimize the OS for either >> background services or foreground applications. One of the things this did >> was throttle network usage. I don't know if windows 10 has the same setting >> though. > > Yes, it does. Getting to it is a pain. Search the "Settings" thingy for > "Advanced system settings". That will bring up the System Properties dialog > you used to be able to get to from the Control Panel. Click the Advanced > tab. Under Performance, click Settings... . Click the Advanced tab there. > Right in that dialog there's a Processor Scheduling box where you select for > best performance of "Programs" or "Background services". > > It would be interesting to hear what it's currently set to before you try > changing it. Try to keep all your testcases the same so we're dealing with > apples to apples. > > ..mark > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Here's the other direction with rsync also in --daemon mode: # ON WINDOWS (from a mintty terminal): dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000 # create 2GB file printf "[TMP]\npath = /tmp\n" > /etc/rsyncd.conf # create rsyncd.conf rsync --daemon --no-detach # run rsync in daemon mode # ON LINUX: rm -f /tmp/bigfile.bin; rsync -Pva $WINHOST:/tmp/bigfile.bin /tmp/ # 4MB/s rm -f /tmp/bigfile.bin; rsync -Pva rsync://$WINHOST/TMP/bigfile.bin /tmp/ # 100MB/s I was thinking that there might be some Windows Policy that rate-limits some syscalls for background process, but I also tried: # ON WINDOWS (Administrator mintty): cygrunsrv -E cygsshd# exit sshd service /usr/sbin/sshd -D # run sshd in "foreground" (GUI app) and I still get 4MB/s. Maybe some IPC rate-limit issue between sshd and rsync (but that isn't there between e.g. sshd and md5sum... ?) On Tue Aug 24 2021, at 7:02 PM, NightStrike wrote: > Older versions of windows had a setting to optimize the OS for either > background services or foreground applications. One of the things this did > was throttle network usage. I don't know if windows 10 has the same setting > though. I read about a NetworkThrottlingIndex that limits "non-multimedia" packets to 10 per millisecond which can be disabled by setting: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile] "NetworkThrottlingIndex"=dword: but this made no difference to my tests. On Tue Aug 24 2021, at 5:35 PM, Chris Roehrig wrote: > Here's my test in a nutshell: > > # ON WINDOWS: install Cygwin and enable cygsshd > ssh-host-config -y > # set up authorized keys, etc to make things easier > > # LINUX: Create a 2GB random file on Linux: > dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000 > > # WINDOWS: rsync "pull" to cygwin > LINUXHOST=mylinuxhost > rsync -Pva $LINUXHOST:/tmp/bigfile.bin /tmp/# 100MB/s full > speed > rm /tmp/bigfile.bin > > # LINUX: rsync "push" to cygwin (from Linux machine) > WINHOST=mywindowshost > rsync -Pva /tmp/bigfile.bin $WINHOST:/tmp/ # slow, > dropping to 4MB/s > > > I get the same results transferring the other direction (WIN -> LINUX): The > Cygwin remote endpoint is always slow. > > > > > > On Tue Aug 24 2021, at 4:43 PM, Mark Geisert wrote: > >> Chris Roehrig wrote: >>> I have a network of Windows, Linux and Mac machines and I use rsync to >>> synchronize various directories between them. >>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >>> expected from gigabit ethernet. This has been an ongoing problem for me >>> for a couple of years over several Windows and Cygwin versions, and I'd >>> like to try to fix it. >>> If I run rsync --daemon --no-detach under mintty in the foreground on the >>> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >>> like it has something to do with rsync.exe running in the background under >>> the cygrunsrv+sshd service (which was installed normally using >>> ssh-host-config). >>> If I do: >>> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >>> or even >>> pv /dev/urandom | ssh $WINHOST md5sum >>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >>> is being throttled by bandwidth or CPU. >>> The machines have less than 15% CPU utilization while transferring, with >>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >>> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >>> Setting their Priority to High doesn't seem to change things. >>> Looking in Resource Monitor on the remote endpoint, the network usage is >>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >>> looks to me as if rsync is somehow being bandwidth-throttled when run in >>> the background under cygsshd. >>> It's almost as if rsync has an implicit --bwlimit override when it is run >>> from cygrunsrv+sshd
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Here's my test in a nutshell: # ON WINDOWS: install Cygwin and enable cygsshd ssh-host-config -y # set up authorized keys, etc to make things easier # LINUX: Create a 2GB random file on Linux: dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000 # WINDOWS: rsync "pull" to cygwin LINUXHOST=mylinuxhost rsync -Pva $LINUXHOST:/tmp/bigfile.bin /tmp/# 100MB/s full speed rm /tmp/bigfile.bin # LINUX: rsync "push" to cygwin (from Linux machine) WINHOST=mywindowshost rsync -Pva /tmp/bigfile.bin $WINHOST:/tmp/ # slow, dropping to 4MB/s I get the same results transferring the other direction (WIN -> LINUX): The Cygwin remote endpoint is always slow. On Tue Aug 24 2021, at 4:43 PM, Mark Geisert wrote: > Chris Roehrig wrote: >> I have a network of Windows, Linux and Mac machines and I use rsync to >> synchronize various directories between them. >> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only >> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or >> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as >> expected from gigabit ethernet. This has been an ongoing problem for me for >> a couple of years over several Windows and Cygwin versions, and I'd like to >> try to fix it. >> If I run rsync --daemon --no-detach under mintty in the foreground on the >> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems >> like it has something to do with rsync.exe running in the background under >> the cygrunsrv+sshd service (which was installed normally using >> ssh-host-config). >> If I do: >> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >> or even >> pv /dev/urandom | ssh $WINHOST md5sum >> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself >> is being throttled by bandwidth or CPU. >> The machines have less than 15% CPU utilization while transferring, with >> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >> In Task Manager, sshd.exe and rsync.exe seem to be running normally using >> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. >> Setting their Priority to High doesn't seem to change things. >> Looking in Resource Monitor on the remote endpoint, the network usage is >> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure >> looks to me as if rsync is somehow being bandwidth-throttled when run in >> the background under cygsshd. >> It's almost as if rsync has an implicit --bwlimit override when it is run >> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no >> difference). >> Any ideas?Not sure where to go from here. > > You're not the first to report this. I don't have any quick answer. But can > you give one or two simple examples of commands that give slow transfers in > your environment? Simple like your 'pv' examples, if possible, using > whatever method that works. > > ..mark > > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). If I do: pv /dev/zero | ssh $WINHOST "cat > /dev/null" or even pv /dev/urandom | ssh $WINHOST md5sum I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). Any ideas?Not sure where to go from here. Thanks, -- Chris Windows 10 v2004 (64-bit) CYGWIN_NT-10.0 3.2.0(0.340/5/3) 2021-03-29 08:42 x86_64 Cygwin rsync version 3.2.4dev protocol version 31 Linux Mint 20 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
