Re: ls/stat on OneDrive causes download of files
On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: Hi Jeffrey, On Mar 6 13:55, Jeffrey Altman via Cygwin wrote: On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: We can add an explicit call to RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); [...] Files and directories that are placeholders should have either the FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file attributes set. When these attributes are set, applications and mini filters are advised not to "read" or "open" the files or directories unless they absolutely need to because doing so will cause the placeholder to be replaced by an object containing the actual data which might take a long time to fetch, Yesterday I stumbled over a certain NtCreateFile flag: FILE_OPEN_NO_RECALL (0x0040) Instructs any filters that perform offline storage or virtualization to not recall the contents of the file as a result of this open. MS-CIFS described it like this: FILE_OPEN_NO_RECALL 0x0040 In a hierarchical storage management environment, this option requests that the file SHOULD NOT be recalled from tertiary storage such as tape. A file recall can take up to several minutes in a hierarchical storage management environment. The clients can specify this option to avoid such delays. This sounds like we could simply add this flag to all NtOpenFile used for path conversion or stat-like calls, without having to care for any file attributes specificially. Does that make sense? Sounds good, without even studying the other details... I speculate some more handling would still be needed to avoid executable detection via magic tags. Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but was not documented by Microsoft relatively recently. Another suggestion would be to try opening the file with FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not required. See https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 Jeffrey Altman -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: We can add an explicit call to RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); and we can recognize the IO_REPARSE_TAG_FILE_PLACEHOLDER and IO_REPARSE_TAG_CLOUD_* tags during symlink evaluation, but even then we still have to know what the reparse point buffer actually contains. Given that the content of reparse points with these reparse tags are undocumented, some people using cloud services should examine these reparse points so we can add some suitable code to Cygwin. Corinna I'm not an expert in this area by any means but here are my recollections from when Microsoft presented in-person on cloud placeholders to filter and filesystem developers many years ago. Files and directories that are placeholders should have either the FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file attributes set. When these attributes are set, applications and mini filters are advised not to "read" or "open" the files or directories unless they absolutely need to because doing so will cause the placeholder to be replaced by an object containing the actual data which might take a long time to fetch, might cost the end user money, or might fail depending upon the network connectivity. In particular, anti-malware should ignore them during scans and only analyze the data when it is fetched locally by an end user application. I believe that IO_REPARSE_TAG_FILE_PLACEHOLDER was replaced by IO_REPARSE_TAG_CLOUD_1 ..IO_REPARSE_TAG_CLOUD_F. Any reparse tag attached to a placeholder object is for the interpretation of the filter associated with the back-end storage and not for the consumption of applications. The content of the reparse tags can be back-end proprietary; different reparse data for onedrive, icloud, dropbox, etc. The default ProcessPlaceholderCompaibilityMode is PHCM_EXPOSE_PLACEHOLDERS which makes the FILE_ATTRIBUTE flags and reparse tags visible. Microsoft maintains a database of processes for which PHCM_DISGUISE_PLACEHOLDER is set which hides that information. Its unclear to me that explicitly setting the placeholder compatibility mode is useful. I'm not sure that exposing the object as a symlink is a good idea. A posix symlink is an object whose type and target information cannot change. In the case of a placeholder, the placeholder is silently replaced by the actual object either when the object is opened or the object's data is accessed. An application that believes it knows that the object is a symlink will be mighty confused when it turns out to be a file or a directory. Perhaps the question that needs to be asked is whether there are opens that can be skipped if an object is known to not be locally present (either of the FILE_ATTRIBUTE flags are set)? Jeffrey Altman -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
On 1/8/2024 1:44 PM, matthew patton via Cygwin wrote: Cygwin does not create symlinks as junctions. No idea where you got that idea. $ echo $CYGWINwinsymlinks:nativestrict $ /usr/bin/ln -s default.GGG6q test1 01/08/2024 01:24 PM test1 [...]Type=File $ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin) 01/08/2024 01:25 PM test2.nocygwin [...]Type=File JUNCTIONS are a type of reparse point tag. Many tools report things as JUNCTIONS when they don't know what else to call it because JUNCTIONS were the only type of reparse tag commonly used. This is the output of JP Software's Take Command for an AFS volume root directory [\\afs\yfs]dir Directory of \\afs\yfs\* 5/26/2021 11:49 . 11/26/2019 14:25 amd64_linux26 [your-file-system.com#amd64_linux26] 11/24/2019 20:29 backups 10/20/2022 10:43 project 12/05/2011 10:14 public [#root.public] 9/06/2020 9:27 service 7/26/2010 20:44 support [#root.support] 6/15/2011 11:40 test [#test.test] 2/15/2012 8:49 u 3/28/2023 12:50 user 11/28/2011 17:01 usr [user] 12/10/2009 0:34 www [#root.www] 12/13/2016 1:29 30 autorun.inf 6/07/2015 21:54 104 Desktop.ini 11/26/2019 14:42 local [@sys\usr\local] 148 bytes in 3 files and 12 dirs 0 bytes free The JUNCTIONS are AFS volume mount points using the Microsoft designated reparse point for OpenAFS. The SYMLINKD is a symlink reparse point to a directory using the NTFS symlink reparse point. The SYMLINK is a symlink reparse point to an object of unknown type (which Windows expects to be a file not a directory) using the NTFS symlink reparse point. Here is the cygwin ls -l output [\\afs\yfs]c:\tools\cygwin\bin\ls -l . total 38 -rw-r--r-- 1 jaltm mkpasswd 104 Jun 7 2015 Desktop.ini drwxr-xr-x 1 jaltm mkpasswd 0 Nov 26 2019 amd64_linux26 -rw-r--r-- 1 jaltm mkpasswd 30 Dec 13 2016 autorun.inf drwxr-xr-x 1 jaltm mkpasswd 0 Nov 24 2019 backups lrwxrwxrwx 1 jaltm mkpasswd 14 Nov 26 2019 local -> @sys/usr/local drwxr-xr-x 1 jaltm mkpasswd 0 Oct 20 2022 project drwxr-xr-x 1 jaltm mkpasswd 0 May 21 2023 public drwxr-xr-x 1 jaltm mkpasswd 0 Sep 6 2020 service drwxr-xr-x 1 jaltm mkpasswd 0 Nov 17 2015 support drwxr-xr-x 1 jaltm mkpasswd 0 May 26 2019 test drwxr-xr-x 1 jaltm mkpasswd 0 Feb 15 2012 u drwxr-xr-x 1 jaltm mkpasswd 0 Mar 28 2023 user lrwxrwxrwx 1 jaltm mkpasswd 4 Nov 28 2011 usr -> user drwxr-xr-x 1 jaltm mkpasswd 0 Jul 10 2017 www As part of this reply I will note that the NTFS symlinks differ from POSIX symlinks in significant ways 1. A pre-existing file system object is required in order to attach a reparse tag 2. The type of the target must be known when the reparse tag is applied to a pre-existing file system object 3. The reparse tag may be removed and replaced any number of times without deleting the pre-existing object to which it is attached. Whereas a POSIX symlink inode target cannot be altered once created. The inode must be deleted and replaced. 4. The Windows file APIs do not behave as many applications expect them to when a symlink reparse tag is present. For example, opening a file handle will traverse the reparse tag and open the target but the file info api when given the same path will return the information belonging to the object on which the reparse tag was applied. This breaks many applications such as the java runtime among others. Jeffrey Altman -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin/Win32 utility function to convert "raw" IPv6 address string into *.ipv6-literal.net string ?
On 9/28/2023 1:56 PM, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote: What do you think that output is - the PTR is resolved to "localhost." You obviously did not get the point that I was making. Using ip6.arpa *is* the standard way to get around with "DNS-like" IPv6 addresses, as it would be "understood". Using the "ipv6-literal.net" domain is not portable and would result in NXDOMAIN anywhere but Windows (where the resolver seems to intercept and convert them internally). The ip6.arpa names are used to lookup PTR records which contain hostnames. The ipv6-literal.net names are used to simulate records and map the name to an IPv6 address (with an optional address scope). For those that are unaware of the history[1][2][3], UNC names support the use of IPv4 addresses as an alternative to SMB server names or DNS host names. End users expect to be able to specify a UNC path such as \\2001:db8:85a3:8d3:1319:8a2e:370:7348\share or \\[2001:db8:85a3:8d3:1319:8a2e:370:7348]\share However, colons are illegal in UNC paths and therefore standard IPv6 representations cannot be used. The .ipv6-literal.net server name is a method of representing an IPv6 address as a UNC server name such that it can be locally translated by getaddrinfo() into an IPv6 address without querying DNS. \\2001-db8-85a3-8d3-1319-8a2e-370-7348.ipv6-literal.net\share If the ipv6.arpa representation\\8.4.3.7.0.7.3.0.e.2.a.8.9.1.3.1.3.d.8.0.3.a.5.8.8.b.d.0.1.0.0.2.ip6.arpa\path was used as a server name and treated as a DNS lookup that would be harder for humans to construct and no more portable. The ip6.arpa representation would also be unable to represent IPv6 address scopes which are supported by ipv6-literal.net names. I am unaware of any Microsoft Windows APIs that can be called to translate from an IPv6 address to an ipv6-literal.net string. [1] https://en.wikipedia.org/wiki/IPv6_address#Literal_IPv6_addresses_in_UNC_path_names [2] https://devblogs.microsoft.com/oldnewthing/20100915-00/?p=12863 [3] https://learn.microsoft.com/en-us/windows/win32/api/winnetwk/nf-winnetwk-wnetaddconnection3a smime.p7s Description: S/MIME Cryptographic Signature -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [EXTERNAL] Re: mkfifo: cannot set permissions of 'x.fifo': Not a directory
On 8/22/2023 10:52 AM, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote: FIFOs which don't make *any* sense ... FWIW, a remote NFS fileystem. I got an impression that the OP is trying to deploy something (maybe the entire Cygwin) onto an NFS share. So the named FIFO "file" is also created in there. It's pointless to assume that the FIFO can be used as a communication device between the hosts that can mount the share, but it can be quite feasible to envision a scenario, in which the same host opens the FIFO located on the share from two processes and establish the communication using that special "file" (which is basically a special data-less i-node). I've been following this thread with quite a bit of curiosity. For those who do not know me, I'm the lead developer of the AFS filesystem on Windows. There have been requests for more than two decades for AFS clients to add support for locally created pipe files because AFS volumes are often used as home directories (even on Windows) and so many applications allocate a pipe in the home directory as a method of inter-process communication or a lock. The problem is that even if the intended usage of the file is entirely local, the directory object, the directory entry and the allocated inode (or equivalent) is network visible. What happens when the user executes two copies of an application such as PyCharm on two separate machines sharing the same home directory? Does the directory entry and inode get reused on startup and/or deleted on exit? How does that impact the process instance on the other machine? The conclusion I came to long ago is that if pipes are to be implemented in a network file system namespace then the pipes must be fully functional network pipes. In just about all cases applications can be configured to use a non-default paths. In my opinion they should not be placed in a shared file system. I've also been following this thread because both the Microsoft NFSv3 and the Citi NFSv4 clients are rather incomplete filesystem implementations from the perspective of the Installable File System interface and the Multiple UNC Provider interface. In my opinion, Microsoft NFSv3 is the bare minimum that must be implemented for Microsoft to advertise that NFSv3 is available on Windows. The NFSv3 symlink interface leveraging extended attributes as a method of setting and reading the target predates the introduction of IO_REPARSE_TAG_SYMLINK for NTFS. Prior to the allocation of IO_REPARSE_TAG_SYMLINK I obtained a proprietary tag allocation from Microsoft for AFS symlinks but switched to IO_REPARSE_TAG_SYMLINK once it was available because then AFS symlinks work the same as NTFS. The Citi NFSv4 implementation is not only incomplete from a Windows interface perspective but Citi never obtained from Microsoft the required filesystem registrations. For example, it doesn't have a Microsoft assigned FLT_FILESYSTEM_TYPE, WNNC network type, name or NodeTypeCode values. FLT_FSTYPE_NFS, WNNC_NET_MS_NFS, and "nfs" are specific to the Microsoft NFSv3 (aka \FileSystem\NfsRdr) implementation. The NodeTypeCode range used by the Citi NFSv4 filesystem (0xFC00) is outside the block allocated to third party filesystems and appears to be from a published sample which could lead to corruption if two filesystems allocating FileControlBlocks with the same NodeTypeCodes are present on the system. The returned WNNC value is also from a sample. Perhaps the original poster knows where development of the Citi NFSv4 client is currently taking place. The source code repos I've been able to find do not have any commits since 2012 (tag v1.0.0) and have been flagged as archival on GitHub. In my opinion, for Cygwin to reliably work with either of these implementations is going to require on-going developer effort and continuous integration testing. Not only of Cygwin but in the case of Citi NFSv4 from the team maintaining the product. Unless something has significantly changed since 2013 I do not expect Microsoft to be willing to invest much effort into enhancing the NFSv3 implementation. The likely recommendation would be to re-export the NFS file shares via Samba and access them via CIFS. Jeffrey Altman smime.p7s Description: S/MIME Cryptographic Signature -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible
On 1/4/2021 10:27 AM, Matt D. via Cygwin (cygwin@cygwin.com) wrote: > I am using symbolic links native to Windows. My CYGWIN environment > variable has been set to "winsymlinks:nativestrict" and my account has > permission to make symbolic links. This is an issue specifically with > Cygwin; I have no problems making links at the windows command line. > Cygwin also does not have a problem making symbolic links-- if the > target already exists. The issue is that I cannot create native > symbolic links with Cygwin for targets that DON'T exist. > > The normal behavior for both Windows and Linux is to create the > symbolic link whether the target exists or not. I don't know why > Cygwin fails to do this only for native Windows symbolic links. It > does not have a problem creating links to any target with the default > Cygwin (non-Windows) symbolic links. Windows native symlinks encode the object type of the target and the encoded type must match that of the target or the link will not work when the target exists. A UNIX symlink does not encode any details of the target. Cygwin doesn't know what type of native symlink to create if the target does not exist. I hope this knowledge helps. Jeffrey Altman smime.p7s Description: S/MIME Cryptographic Signature -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Compiling C-Kermit 9.0.305 Alpha.02 on Cygwin
On 11/30/2020 10:06 AM, Keith Christian via Cygwin (cygwin@cygwin.com) wrote: > I downloaded the .tar.gz of C-Kermit 9.0.305 Alpha.02, 19 September 2020. > Web page: http://www.kermitproject.org/ckdaily.html > Source download: http://www.kermitproject.org/ftp/kermit/test/tar/x.tar.gz > > I realize that C-Kermit has not been in the Cygwin distribution for > awhile. There is no "cygwin" target in the makefile in the x.tar.gz > source distribution. > Is there any guidance for how to compile this modern version of > C-Kermit for present-day Cygwin? > > Thanks. I suspect building C-Kermit with SSH, OpenSSL, Kerberos v5 support is going to be problematic since none of that code has been updated in more than a decade to keep up with the latest upstream packages. I suggest you start by trying to build for a generic linux target without any security features. Jeffrey Altman former Kermit developer smime.p7s Description: S/MIME Cryptographic Signature -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
