Re: Faster rsync?
On 29.08.23 14:32, Adam Kessel via Cygwin wrote: I've found rsync to be painfully slow on large folders -- hours to sync thousands of files, even when they already match size and --size-only is used. It's much faster between native Linux boxes. I've been using rsync, unison and similar tools on Windows and Linux since basically forever. In my humble opinion, the problem is the Windows file system performance, not the synchronization tools. As a separate example, try to download the boost source code, and extract the archive. I can do the extraction in way under a minute on Linux, but have to wait many many minutes on a similarly equipped Windows machine. Just my two cents. Mario -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Kind request to update 'nasm'
Dear All, the current version of OpenSSL (1.1.1s) fails to build for me with an error in nasm. The issue is reported upstream and supposedly fixed already, see https://bugzilla.nasm.us/show_bug.cgi?id=3392658 I've tested nasm 2.16.1 from the upstream win32 builds and it does indeed fix the problem. However I would prefer a Cygwin deployment over manually maintained archives. Could someone kindly consider updating 'nasm' to current latest 2.16, or anything including or higher than 2.15.01? All the best, Mario Emmenlauer -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 München http://www.biodataanalysis.de/ -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Could rm remove files and folders with colon in their name?
On 10/11/2021 21:39, Corinna Vinschen via Cygwin wrote: On Nov 10 21:24, Mario Emmenlauer wrote: On 10.11.21 14:49, Corinna Vinschen via Cygwin wrote: On Nov 10 10:45, Mario Emmenlauer wrote: Could 'rm' support removing files and folders that have a colon ':' in their name? I.e. I would like that 'rm -fr' would remove a full directory tree, including such folders. Currently it will correctly remove anything inside such folders, but not the folder itself. As an example, for the following structure: C:/root/folder/C:/inside/file.txt When using 'rm -fr root', afterwards I have: C:/root/folder/C: It works fine if the folder is called, say, "a:b", it just doesn't work for a name which looks like a drive letter "x:", apparently. That is indeed interesting, I was not aware of it! Then maybe the problem is not so hard to solve? That would be awesome! To the contrary. The problem is the ambiguity that "X:/foo" might be either the absolute POSIX path $CWD/X:/foo, or the absolute DOS path "X:\foo". I have a patch which fixes your case, but not much else. The problem is that we historically allow DOS paths as input at all. That was a bad decision from the start, but you can't easily change 25 years of history... Oh my, I see! All the more thanks for so quickly patching support for this use case. Its highly appreciated! All the best, Mario -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Could rm remove files and folders with colon in their name?
On 10.11.21 14:49, Corinna Vinschen via Cygwin wrote: > On Nov 10 10:45, Mario Emmenlauer wrote: >> Could 'rm' support removing files and folders that have a colon ':' in >> their name? I.e. I would like that 'rm -fr' would remove a full directory >> tree, including such folders. Currently it will correctly remove anything >> inside such folders, but not the folder itself. >> >> As an example, for the following structure: >> C:/root/folder/C:/inside/file.txt >> >> When using 'rm -fr root', afterwards I have: >> C:/root/folder/C: > > It works fine if the folder is called, say, "a:b", it just doesn't > work for a name which looks like a drive letter "x:", apparently. That is indeed interesting, I was not aware of it! Then maybe the problem is not so hard to solve? That would be awesome! > Funny. I'm busy with non-Cygwin stuff ATM, but I'll look into it > later. > > Thanks for the report. Thanks a lot for your support! All the best, Mario Emmenlauer -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 München http://www.biodataanalysis.de/ -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Could rm remove files and folders with colon in their name?
Hi Brian, On 10.11.21 17:35, Brian Inglis wrote: > On 2021-11-10 02:45, Mario Emmenlauer wrote: >> PS: These folders are created when I use the Cygwin-based build system >> for ICU (see >> https://unicode-org.github.io/icu/userguide/icu4c/build.html#how-to-build-and-install-on-windows-with-cygwin) >> For me this is in a combination with native Perl for Windows (ActivePerl, >> in my case), and using absolute build paths. After using ICU's build >> system, I can not remove the build tree anymore. It may be possible to >> solve this on the ICU side too. But their automake-and-Perl-based path >> mangling is not easily modified, and I've failed to isolate the root >> cause of the illegal paths. > > Install the Cygwin cygport and libicu-devel source and binary packages and > use only Cygwin exclusively with cygport to avoid problems. I'm not sure that this is what I want. I want to build ICU with Visual Studio and the ClangCl compiler frontend from LLVM 13.0.0. Their build system can use Cygwin for this combination to orchestrate and configure the build - which I very much like :-) Sorry that I did not mention this in my email. The build works fine and I get native ICU shared libraries like I want. The only "downside" is that it creates the unsupported folders :-) Cheers, Mario -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 München http://www.biodataanalysis.de/ -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Could rm remove files and folders with colon in their name?
Dear All,
I've searched if this topic has come up before but could not find it.
Could 'rm' support removing files and folders that have a colon ':' in
their name? I.e. I would like that 'rm -fr' would remove a full directory
tree, including such folders. Currently it will correctly remove anything
inside such folders, but not the folder itself.
As an example, for the following structure:
C:/root/folder/C:/inside/file.txt
When using 'rm -fr root', afterwards I have:
C:/root/folder/C:
To remove everything, I can use 'find root -depth -exec rmdir \{\} \;'
I understand that files and folders with colon in their name are illegal
on Windows and not supported very well. But a number of tools manage to
create (and also remove) such files. I've found that even the native
'del' can support this when using the UNC name (see for example
https://serverfault.com/a/96833). It would be great if Cygwin could
also feature this support.
All the best,
Mario
PS: These folders are created when I use the Cygwin-based build system
for ICU (see
https://unicode-org.github.io/icu/userguide/icu4c/build.html#how-to-build-and-install-on-windows-with-cygwin)
For me this is in a combination with native Perl for Windows (ActivePerl,
in my case), and using absolute build paths. After using ICU's build
system, I can not remove the build tree anymore. It may be possible to
solve this on the ICU side too. But their automake-and-Perl-based path
mangling is not easily modified, and I've failed to isolate the root
cause of the illegal paths.
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
On 25.08.21 19:52, Ken Brown via Cygwin wrote: A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O: https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html I never followed up on it. But if you think it might help with this problem, I could dust it off and try to finish it. This sounds quite interesting! Do I assume correctly that these changes may likely lead to a general improvement also in other situations if the impact of the pipe is so big for this specific case? Pipes are most likely used in quite a lot of scenarios?! All the best, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Help using Cygwin on AppVeyor by Apache thrift, any lucky guesses?
Dear Cygwin community, I know that my report here is not super helpful, but I'm in a slightly tight spot, and hope that someone can help me with a lucky guess?! I'd like to improve the AppVeyor CI build for the Apache thrift project for Cygwin. Apache thrift builds and tests on Cygwin by default, but since a while, the builds have failed with cmake problems. The error message is not too helpful - so I hope that someone here may just have had the same issue before and can help? We use `apt-cyg` to install the packages, could this be related? Here is a link to a failed build log: https://ci.appveyor.com/project/ApacheSoftwareFoundation/thrift/builds/40263513/job/a69xt6m4o0y9x1bw?fullLog=true cmake fails with the (not so helpful) error message: C:/cygwin64/bin/cmake.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory The corresponding Cygwin AppVeyor installation is: https://github.com/apache/thrift/blob/781143e75cb585c0a88dea3f904c77228531d1d6/build/appveyor/CYGW-appveyor-install.bat After the install, there is a call to `refreshenv` in the appveyor.yml, so I think the problem is not due to an incomplete environment setup. And the corresponding Cygwin AppVeyor build script is: https://github.com/apache/thrift/blob/781143e75cb585c0a88dea3f904c77228531d1d6/build/appveyor/CYGW-appveyor-build.bat Any help would be highly appreciated! All the best, Mario Emmenlauer -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 München http://www.biodataanalysis.de/ -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
On 14.10.20 13:50, Corinna Vinschen wrote: > On Oct 14 11:06, Mario Emmenlauer wrote: >> On 14.10.20 10:28, Corinna Vinschen wrote: >>> Actually, not really. It's weird in fact, given ls(1) shows the >>> desired result. That would point to a bug in access(2), but there's >>> no special code in access(2) for NFS. For filesystems not supporting >>> ACLs (FAT, NFS, etc), it calls stat(2) and checks the st_mode bits >>> against the requested access(2) mode based on the uid/gid of the >>> caller, simple as that. >> >> Hmm, now that you mention it, I just coincidentally found an issue >> with the `_stat` call in Microsoft Windows 2004 update. In the Apache > > This is entirely unrelated. We're talking about Cygwin stat(2), > not msvcrt.dll _stat(). Different source, different call. Yes, but Cygwin stat is implemented based on the Win32 posix layer too, or not? At least I got this impression from browsing the sources - albeit admittedly there are far too many indirections and ifdefs for me to really know what's going on... :-) :-) All the best, Mario -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
On 14.10.20 10:28, Corinna Vinschen wrote: On Oct 13 21:00, Mario Emmenlauer wrote: On 13.10.20 20:36, Corinna Vinschen wrote: Everything seems to work quite well, and in `ls -la` I can see the file permissions and user and group entries. But when using `test` to check for read (`test -r`) or execute permissions (`test -x`), it always returns false, even for readable files. `ls` on the other hand shows the permissions correctly, and `cat`ing the files works without problems. There's something fishy in your environment. NFS permissions from NFS shares mounted via Microsoft's NFSv3 are read using some internal function I got hinted at by the MSFT NFS guys at one point. This information is then exported as mode bits by Cygwin's stat(2) and used accordingly by Cygwin's access(2). Having said that, there's *no* magic at all in the user space applications other than using Cygwin's stat(2) and access(2) functions. Consequentially, using Cygwin's ls(1) or test(1) from coreutils, the results are the expected ones in both cases; just tried it myself, just to be sure. So, what's fishy? I don't know, but maybe you're using a non-Cygwin test(1) accidentally? I see your point, but to the best of my knowledge there is nothing fishy. Its a freshly set up computer with an official Windows 10 x64 from Microsoft directly. I've installed all latest updates including the 2004 update. Cygwin was also just installed a few months ago. I did use a script to install Cygwin via its installer in an automated fashion, but I've run the normal, graphical installer a few times since then to make sure everything is nice and clean. Calling "which test" shows "/usr/bin/test", but since I use only bash in all my scripts, I guess it anyways defaults to the builtin. Please check and try again with the stand-alone test(1), too. Sorry, I should have mentioned this: it gives the same result. Is there anything I can do to isolate this further? Its a quite curious case and I'm basically at the end of my wit... Actually, not really. It's weird in fact, given ls(1) shows the desired result. That would point to a bug in access(2), but there's no special code in access(2) for NFS. For filesystems not supporting ACLs (FAT, NFS, etc), it calls stat(2) and checks the st_mode bits against the requested access(2) mode based on the uid/gid of the caller, simple as that. Hmm, now that you mention it, I just coincidentally found an issue with the `_stat` call in Microsoft Windows 2004 update. In the Apache thrift project I found that `_stat` stopped working on domain socket files on Windows. This sounds not directly identical, but could be well related. I did not try `_stat` in other situations, but something must have changed there very recently. The issue is reported upstream, but sadly for the wrong product (Visual Studio), so nobody is following up this report anymore: https://developercommunity.visualstudio.com/content/problem/1180994/-stat-fails-on-existing-domain-socket-file-when-bu.html (Scroll to the bottom to see the full report). Please call ls(1) and test(1) -r (not the bash builtin) on a file exposing the behaviour under strace like this: $ strace -o ls.trace /bin/ls -l $ strace -o test.trace /bin/test -r and send the trace files here, together with the output of the above ls(1) call and the output of id(1). I will to that! Thanks for your help and interest!! All the best, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
Hi Corinna, great to have you back :-) On 13.10.20 20:36, Corinna Vinschen wrote: On Oct 6 18:10, Mario Emmenlauer wrote: Dear Andrey, On 06.10.20 17:46, Andrey Repin wrote: Greetings, Mario Emmenlauer! thanks for the awesome Cygwin, its really great! Everything seems to work quite well, and in `ls -la` I can see the file permissions and user and group entries. But when using `test` to check for read (`test -r`) or execute permissions (`test -x`), it always returns false, even for readable files. `ls` on the other hand shows the permissions correctly, and `cat`ing the files works without problems. This is a known issue. For years known. test only guess -r/-w/-x results based on permissions as it sees them. But it do not actually try to read/write/execute the subject, which, as you can imagine, may lead to all sorts of false positives/negatives on filesystems with less than trivial access control setups. In other words, don't test for rwx if you can avoid it. The results MAY be wrong. Ok, this explains a lot, and I almost guessed as much! But can I ask, do you happen to know why `ls -l` shows the "correct" permissions including 'r' and 'x'? It seems `ls` has some magic that `test` is lacking? And I can not imagine that `ls` would try to open every file, or does it?. So could this "magic" be ported from `ls` to `test`? There's something fishy in your environment. NFS permissions from NFS shares mounted via Microsoft's NFSv3 are read using some internal function I got hinted at by the MSFT NFS guys at one point. This information is then exported as mode bits by Cygwin's stat(2) and used accordingly by Cygwin's access(2). Having said that, there's *no* magic at all in the user space applications other than using Cygwin's stat(2) and access(2) functions. Consequentially, using Cygwin's ls(1) or test(1) from coreutils, the results are the expected ones in both cases; just tried it myself, just to be sure. So, what's fishy? I don't know, but maybe you're using a non-Cygwin test(1) accidentally? I see your point, but to the best of my knowledge there is nothing fishy. Its a freshly set up computer with an official Windows 10 x64 from Microsoft directly. I've installed all latest updates including the 2004 update. Cygwin was also just installed a few months ago. I did use a script to install Cygwin via its installer in an automated fashion, but I've run the normal, graphical installer a few times since then to make sure everything is nice and clean. Calling "which test" shows "/usr/bin/test", but since I use only bash in all my scripts, I guess it anyways defaults to the builtin. The only "weak link" in my setup is the NFS mount. I'm no expert in exporting NFS, nor in mounting NFS from Windows. Maybe something is fishy there, albeit I did not use any special parameters or quirks (again, to the best of my knowledge). What I can say is that I'm limited to NFS v3 since its not a Server-Edition Windows. Also, I tried my best to open all NFS ports in the firewall but possibly I'm not running one of the many extended NFS services like lockd or something. I checked that most are installed, running and use a static port, but its hard to be sure, since NFS seems to support quite many "extras". Is there anything I can do to isolate this further? Its a quite curious case and I'm basically at the end of my wit... All the best, Mario -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
Dear Andrey, On 06.10.20 17:46, Andrey Repin wrote: > Greetings, Mario Emmenlauer! > >> thanks for the awesome Cygwin, its really great! > >> Everything seems to work quite well, and in `ls -la` I can see the >> file permissions and user and group entries. But when using `test` >> to check for read (`test -r`) or execute permissions (`test -x`), it >> always returns false, even for readable files. `ls` on the other hand >> shows the permissions correctly, and `cat`ing the files works without >> problems. > > This is a known issue. For years known. > test only guess -r/-w/-x results based on permissions as it sees them. > But it do not actually try to read/write/execute the subject, which, as you > can imagine, may lead to all sorts of false positives/negatives on filesystems > with less than trivial access control setups. > In other words, don't test for rwx if you can avoid it. The results MAY be > wrong. Ok, this explains a lot, and I almost guessed as much! But can I ask, do you happen to know why `ls -l` shows the "correct" permissions including 'r' and 'x'? It seems `ls` has some magic that `test` is lacking? And I can not imagine that `ls` would try to open every file, or does it?. So could this "magic" be ported from `ls` to `test`? Cheers, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
On 22.09.20 22:14, Mario Emmenlauer wrote: > But since today I met a problem: I mounted a Linux NFSv3 share using > the Windows 10 shipped NFS client. The user and group ID are mapped > via registry settings AnonymousUid and AnonymousGid in the entry > HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default > > Everything seems to work quite well, and in `ls -la` I can see the > file permissions and user and group entries. But when using `test` > to check for read (`test -r`) or execute permissions (`test -x`), it > always returns false, even for readable files. `ls` on the other hand > shows the permissions correctly, and `cat`ing the files works without > problems. > > I've read https://cygwin.com/cygwin-ug-net/using-filemodes.html > about the Cygwin file permissions for NFS, and also the NFS account > mapping at https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-nfs, > but as far as I can see, they are both unrelated. Google turned up no > useful hits for keywords "cygwin" "test" and "nfs", so I'm a bit at the > end of my wit. > > Is this a known issue, and/or are there any workarounds? I'm currently > using `test -e` in place of read or execute checks, but it basically > breaks all my build scripts. Is there something I should do about this issue? I could look into the source code of `test` on Cygwin if someone can point me to the correct repository? Or should I just file an issue? The issue is not a super high priority for me personally, but I guess its quite a limitation of Cygwin if essential scripting functionality is misbehaving on NFS. All the best, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
test -r or -x always return false on an NFS mount?
Dear All, thanks for the awesome Cygwin, its really great! But since today I met a problem: I mounted a Linux NFSv3 share using the Windows 10 shipped NFS client. The user and group ID are mapped via registry settings AnonymousUid and AnonymousGid in the entry HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default Everything seems to work quite well, and in `ls -la` I can see the file permissions and user and group entries. But when using `test` to check for read (`test -r`) or execute permissions (`test -x`), it always returns false, even for readable files. `ls` on the other hand shows the permissions correctly, and `cat`ing the files works without problems. I've read https://cygwin.com/cygwin-ug-net/using-filemodes.html about the Cygwin file permissions for NFS, and also the NFS account mapping at https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-nfs, but as far as I can see, they are both unrelated. Google turned up no useful hits for keywords "cygwin" "test" and "nfs", so I'm a bit at the end of my wit. Is this a known issue, and/or are there any workarounds? I'm currently using `test -e` in place of read or execute checks, but it basically breaks all my build scrips. All the best, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
setup.exe --packages works great for me, thanks!
Hi, just a short: THANK YOU. I followed the discussions about the --packages flag for setup.exe on the mailing lists, and was very much hoping it would make it into the release some time. I just tested it from CVS, and it works great for me. Deploying Cygwin to several very differently installed machines, and no complaints so far. Thanks, and keep up the good work, Mario
