Re: ls/stat on OneDrive causes download of files
On Mar 9 15:29, Marcin Wisnicki via Cygwin wrote: > I did more testing and found out that the problem does not happen in > cygwin by default because cygwin mounts with acl which doesn't do > header sniffing while msys uses noacl. > > Testing on an mp4 file in OneDrive, when I use noacl in cygwin it > triggers the read as well. > After upgrading to the test version the read is gone and an mp4 file > is not executable. > > Thank you! Thanks a lot for testing. I backported the changes (minus the lsattr(1)/ chattr(1) changes) to the 3.5 branch so it will be in released with 3.5.2 in the next few weeks. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
I did more testing and found out that the problem does not happen in cygwin by default because cygwin mounts with acl which doesn't do header sniffing while msys uses noacl. Testing on an mp4 file in OneDrive, when I use noacl in cygwin it triggers the read as well. After upgrading to the test version the read is gone and an mp4 file is not executable. Thank you! On Fri, 8 Mar 2024 at 17:26, Marcin Wisnicki wrote: > > On 2024-03-08 17:21, Corinna Vinschen wrote: > > On Mar 8 14:56, Corinna Vinschen via Cygwin wrote: > >> On Mar 8 08:15, Jeffrey Altman via Cygwin wrote: > >>> On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: > Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: > > FILE_OPEN_NO_RECALL (0x0040) > > [...] > > This sounds like we could simply add this flag to all NtOpenFile > > used for path conversion or stat-like calls, without having to care > > for any file attributes specificially. > > > > Does that make sense? > Sounds good, without even studying the other details... > I speculate some more handling would still be needed to avoid executable > detection via magic tags. > > >>> Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but > >>> was > >>> not documented by Microsoft relatively recently. > >> Thanks for the feedback, guys. > >> > >>> Another suggestion would be to try opening the file with > >>> FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not > >>> required. See > >>> > >>> https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 > >> Cygwin uses the minimum of required permissions in NtCreateFile/ > >> NtOpenFile calls anyway. > >> > >> I'm just running a test cygwin DLL locally with a lot of added > >> FILE_OPEN_NO_RECALL bits and a couple of added attribute checks for > >> being offline to allow skipping some code. > >> > >> I think I'll push this change in a bit so we get a test release out > >> so people using OneDrive can test. > > I pushed this change as well as a followup change to make sure we don't > > inadvertently recall an offline file. I also added handling for the > > Pinned and Unpinned attributes to chattr(1) and lsattr(1). > > > > The full set of changes can be tested by installing the Cygwin test > > release 3.6.0-0.77.g06aa5a751682. > > > > Please give it a try. If you encounter a situation which still results > > in recalling an offline file in a situation which doesn't qualify for > > it, please report. We will have to analyze that situation further > > then. > > > > > > Thanks, > > Corinna > > Thanks for doing this work so quickly. I'm not subscribed to this > mailing list so I didn't see previous messages. > > I will try to check this in Cygwin this weekend but should tell you that > I'm not cygwin user and now found a report of another user claiming this > only happens in MSys and not in Cygwin. > > https://github.com/msys2/MSYS2-packages/issues/3049 > -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On 2024-03-08 17:21, Corinna Vinschen wrote: On Mar 8 14:56, Corinna Vinschen via Cygwin wrote: On Mar 8 08:15, Jeffrey Altman via Cygwin wrote: On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: FILE_OPEN_NO_RECALL (0x0040) [...] This sounds like we could simply add this flag to all NtOpenFile used for path conversion or stat-like calls, without having to care for any file attributes specificially. Does that make sense? Sounds good, without even studying the other details... I speculate some more handling would still be needed to avoid executable detection via magic tags. Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but was not documented by Microsoft relatively recently. Thanks for the feedback, guys. Another suggestion would be to try opening the file with FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not required. See https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 Cygwin uses the minimum of required permissions in NtCreateFile/ NtOpenFile calls anyway. I'm just running a test cygwin DLL locally with a lot of added FILE_OPEN_NO_RECALL bits and a couple of added attribute checks for being offline to allow skipping some code. I think I'll push this change in a bit so we get a test release out so people using OneDrive can test. I pushed this change as well as a followup change to make sure we don't inadvertently recall an offline file. I also added handling for the Pinned and Unpinned attributes to chattr(1) and lsattr(1). The full set of changes can be tested by installing the Cygwin test release 3.6.0-0.77.g06aa5a751682. Please give it a try. If you encounter a situation which still results in recalling an offline file in a situation which doesn't qualify for it, please report. We will have to analyze that situation further then. Thanks, Corinna Thanks for doing this work so quickly. I'm not subscribed to this mailing list so I didn't see previous messages. I will try to check this in Cygwin this weekend but should tell you that I'm not cygwin user and now found a report of another user claiming this only happens in MSys and not in Cygwin. https://github.com/msys2/MSYS2-packages/issues/3049 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On Mar 8 14:56, Corinna Vinschen via Cygwin wrote: > On Mar 8 08:15, Jeffrey Altman via Cygwin wrote: > > On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: > > > Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: > > > > FILE_OPEN_NO_RECALL (0x0040) > > > > [...] > > > > This sounds like we could simply add this flag to all NtOpenFile > > > > used for path conversion or stat-like calls, without having to care > > > > for any file attributes specificially. > > > > > > > > Does that make sense? > > > Sounds good, without even studying the other details... > > > I speculate some more handling would still be needed to avoid executable > > > detection via magic tags. > > > > > Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but was > > not documented by Microsoft relatively recently. > > Thanks for the feedback, guys. > > > Another suggestion would be to try opening the file with > > FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not > > required. See > > > > https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 > > Cygwin uses the minimum of required permissions in NtCreateFile/ > NtOpenFile calls anyway. > > I'm just running a test cygwin DLL locally with a lot of added > FILE_OPEN_NO_RECALL bits and a couple of added attribute checks for > being offline to allow skipping some code. > > I think I'll push this change in a bit so we get a test release out > so people using OneDrive can test. I pushed this change as well as a followup change to make sure we don't inadvertently recall an offline file. I also added handling for the Pinned and Unpinned attributes to chattr(1) and lsattr(1). The full set of changes can be tested by installing the Cygwin test release 3.6.0-0.77.g06aa5a751682. Please give it a try. If you encounter a situation which still results in recalling an offline file in a situation which doesn't qualify for it, please report. We will have to analyze that situation further then. Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On Mar 8 08:15, Jeffrey Altman via Cygwin wrote: > On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: > > Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: > > > Yesterday I stumbled over a certain NtCreateFile flag: > > > > > > FILE_OPEN_NO_RECALL (0x0040) > > > > > > Instructs any filters that perform offline storage or > > > virtualization > > > to not recall the contents of the file as a result of this open. > > > > > > MS-CIFS described it like this: > > > > > > FILE_OPEN_NO_RECALL > > > 0x0040 > > > > > > In a hierarchical storage management environment, this option > > > requests that the file SHOULD NOT be recalled from tertiary storage > > > such as tape. A file recall can take up to several minutes in a > > > hierarchical storage management environment. The clients can > > > specify > > > this option to avoid such delays. > > > > > > This sounds like we could simply add this flag to all NtOpenFile > > > used for path conversion or stat-like calls, without having to care > > > for any file attributes specificially. > > > > > > Does that make sense? > > Sounds good, without even studying the other details... > > I speculate some more handling would still be needed to avoid executable > > detection via magic tags. > > > Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but was > not documented by Microsoft relatively recently. Thanks for the feedback, guys. > Another suggestion would be to try opening the file with > FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not > required. See > > https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 Cygwin uses the minimum of required permissions in NtCreateFile/ NtOpenFile calls anyway. I'm just running a test cygwin DLL locally with a lot of added FILE_OPEN_NO_RECALL bits and a couple of added attribute checks for being offline to allow skipping some code. I think I'll push this change in a bit so we get a test release out so people using OneDrive can test. Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: Hi Jeffrey, On Mar 6 13:55, Jeffrey Altman via Cygwin wrote: On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: We can add an explicit call to RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); [...] Files and directories that are placeholders should have either the FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file attributes set. When these attributes are set, applications and mini filters are advised not to "read" or "open" the files or directories unless they absolutely need to because doing so will cause the placeholder to be replaced by an object containing the actual data which might take a long time to fetch, Yesterday I stumbled over a certain NtCreateFile flag: FILE_OPEN_NO_RECALL (0x0040) Instructs any filters that perform offline storage or virtualization to not recall the contents of the file as a result of this open. MS-CIFS described it like this: FILE_OPEN_NO_RECALL 0x0040 In a hierarchical storage management environment, this option requests that the file SHOULD NOT be recalled from tertiary storage such as tape. A file recall can take up to several minutes in a hierarchical storage management environment. The clients can specify this option to avoid such delays. This sounds like we could simply add this flag to all NtOpenFile used for path conversion or stat-like calls, without having to care for any file attributes specificially. Does that make sense? Sounds good, without even studying the other details... I speculate some more handling would still be needed to avoid executable detection via magic tags. Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but was not documented by Microsoft relatively recently. Another suggestion would be to try opening the file with FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not required. See https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 Jeffrey Altman -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: Hi Jeffrey, On Mar 6 13:55, Jeffrey Altman via Cygwin wrote: On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: We can add an explicit call to RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); [...] Files and directories that are placeholders should have either the FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file attributes set. When these attributes are set, applications and mini filters are advised not to "read" or "open" the files or directories unless they absolutely need to because doing so will cause the placeholder to be replaced by an object containing the actual data which might take a long time to fetch, Yesterday I stumbled over a certain NtCreateFile flag: FILE_OPEN_NO_RECALL (0x0040) Instructs any filters that perform offline storage or virtualization to not recall the contents of the file as a result of this open. MS-CIFS described it like this: FILE_OPEN_NO_RECALL 0x0040 In a hierarchical storage management environment, this option requests that the file SHOULD NOT be recalled from tertiary storage such as tape. A file recall can take up to several minutes in a hierarchical storage management environment. The clients can specify this option to avoid such delays. This sounds like we could simply add this flag to all NtOpenFile used for path conversion or stat-like calls, without having to care for any file attributes specificially. Does that make sense? Sounds good, without even studying the other details... I speculate some more handling would still be needed to avoid executable detection via magic tags. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
Hi Jeffrey, On Mar 6 13:55, Jeffrey Altman via Cygwin wrote: > On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: > > We can add an explicit call to > > > >RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); > > [...] > Files and directories that are placeholders should have either the > FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file > attributes set. When these attributes are set, applications and mini filters > are advised not to "read" or "open" the files or directories unless they > absolutely need to because doing so will cause the placeholder to be > replaced by an object containing the actual data which might take a long > time to fetch, Yesterday I stumbled over a certain NtCreateFile flag: FILE_OPEN_NO_RECALL (0x0040) Instructs any filters that perform offline storage or virtualization to not recall the contents of the file as a result of this open. MS-CIFS described it like this: FILE_OPEN_NO_RECALL 0x0040 In a hierarchical storage management environment, this option requests that the file SHOULD NOT be recalled from tertiary storage such as tape. A file recall can take up to several minutes in a hierarchical storage management environment. The clients can specify this option to avoid such delays. This sounds like we could simply add this flag to all NtOpenFile used for path conversion or stat-like calls, without having to care for any file attributes specificially. Does that make sense? Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
Hi Jeffrey, apart from the attribute stuff... On Mar 6 13:55, Jeffrey Altman via Cygwin wrote: > The default ProcessPlaceholderCompaibilityMode is PHCM_EXPOSE_PLACEHOLDERS > which makes the FILE_ATTRIBUTE flags and reparse tags visible. Microsoft > maintains a database of processes for which PHCM_DISGUISE_PLACEHOLDER is set > which hides that information. Its unclear to me that explicitly setting the > placeholder compatibility mode is useful. What I see as a problem here is this: https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-rtlsetprocessplaceholdercompatibilitymode Quote: "Most Windows applications see exposed placeholders by default. For compatibility reasons, Windows may decide that certain applications ^^ see disguised placeholders by default." But then again, in other news from Microsoft: https://learn.microsoft.com/en-us/windows/win32/cfapi/build-a-cloud-file-sync-engine#compatibility-with-applications-that-use-reparse-points Quote: "[...] the cloud files API always hides its reparse points from all applications except for sync engines and processes whose main image ^^^ resides under %systemroot%. Applications that understand reparse ^^^ points correctly can force the platform to expose cloud files API reparse points using RtlSetProcessPlaceholderCompatibilityMode or RtlSetThreadProcessPlaceholderCompatibilityMode. Considering these two statements, it's totally unclear to a process, if it just defaults to "exposed" or "disguised". Fortunately we can ask Windows by calling the RtlQueryProcessPlaceholderCompatibilityMode() function, right? Lets have a look into the documentation at https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-rtlqueryprocessplaceholdercompatibilitymode Quote: "Return value This function returns the process's placeholder compatibily mode (PHCM_xxx), or a negative value on error (PCHM_ERROR_xxx). Contains one of the following values: Compatibility Mode Value PHCM_APPLICATION_DEFAULT 0 PHCM_DISGUISE_PLACEHOLDER 1 PHCM_EXPOSE_PLACEHOLDERS 2 PHCM_MAX 2 PHCM_ERROR_INVALID_PARAMETER -1 PHCM_ERROR_NO_TEB -2" So I called the function right at the start of the Cygwin DLL, and it returns the value 0, i. e., PHCM_APPLICATION_DEFAULT. At this point the process *still* has no idea if placeholders are exposed or disguised. What a great API! \o/ So, from the above, and if we really want to be sure that placeholders will be exposed, I don't see any way around calling RtlSetProcessPlaceholderCompatibilityMode(PHCM_EXPOSE_PLACEHOLDERS) explicitely at DLL startup. What do you think? Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
Hi Jeffrey, looks like writing our mails overlapped: https://cygwin.com/pipermail/cygwin/2024-March/255622.html On Mar 6 13:55, Jeffrey Altman via Cygwin wrote: > On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: > > We can add an explicit call to > > > >RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); > > > > and we can recognize the IO_REPARSE_TAG_FILE_PLACEHOLDER and > > IO_REPARSE_TAG_CLOUD_* tags during symlink evaluation, but even then > > we still have to know what the reparse point buffer actually contains. > > > > Given that the content of reparse points with these reparse tags are > > undocumented, some people using cloud services should examine these > > reparse points so we can add some suitable code to Cygwin. > > > > > > Corinna > I'm not an expert in this area by any means but here are my recollections > from when Microsoft presented in-person on cloud placeholders to filter and > filesystem developers many years ago. > > Files and directories that are placeholders should have either the > FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file > attributes set. When these attributes are set, applications and mini filters > are advised not to "read" or "open" the files or directories unless they > absolutely need to Per https://learn.microsoft.com/en-us/windows/win32/fileio/file-attribute-constants FILE_ATTRIBUTE_RECALL_ON_OPEN only appears in directory listing classes, but not in standard FILE_BASIC_INFORMATION and alike. That's a bit of a problem considering how we check files during path conversion. The MSDN article doesn't state the same for FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS, which is good, I think. But I'm a bit puzzled then in terms of FILE_ATTRIBUTE_OFFLINE. Is it not used for OneDrive files? > [...] > I'm not sure that exposing the object as a symlink is a good idea. Yeah, that's what I realized as well, see my aforementioned mail. > Perhaps the question that needs to be asked is whether there are opens that > can be skipped if an object is known to not be locally present (either of > the FILE_ATTRIBUTE flags are set)? This may be the way to go, see my mail. It wouldn't be much of a problem to check all attribute bits, i.e. FILE_ATTRIBUTE_OFFLINE, FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS and FILE_ATTRIBUTE_RECALL_ON_OPEN. Maybe that's what we should do. Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On Mar 6 18:19, Corinna Vinschen via Cygwin wrote: > On Mar 6 06:54, Brian Inglis via Cygwin wrote: > > On 2024-03-06 06:28, Corinna Vinschen via Cygwin wrote: > > > On Mar 6 14:22, Corinna Vinschen via Cygwin wrote: > > > > Given these placeholder files are actually reparse points of type > > > > IO_REPARSE_TAG_FILE_PLACEHOLDER, we can handle them as symbolic links. > > > > > > > > However, the structure of the IO_REPARSE_TAG_FILE_PLACEHOLDER reparse > > > > data buffer is undocumented. It would be helpful if somebody using > > > > OneDrive would examine the content of the attached REPARSE_DATA_BUFFER. > > > > > > > > > [2] > > > > > https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 > > > > > > > > The NtReadFile call at this point is not the problem. It would be > > > > helpful to point to Cygwin's source instead of MSYS2, btw. > > > > > > Oh, btw., this is from > > > https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4: > > > > > >IO_REPARSE_TAG_FILE_PLACEHOLDER > > >0x8015 > > > > > > Obsolete. > > > - > > > Used by Windows Shell for legacy placeholder files in Windows 8.1. > > > Server-side interpretation only, not meaningful over the wire. > > > > > > So even if we support them, what is their replacement in W10 and later? > > > > May or not help: > > > > https://stackoverflow.com/questions/59152220/cant-get-reparse-point-information-for-the-onedrive-folder > > We can add an explicit call to > > RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); > > and we can recognize the IO_REPARSE_TAG_FILE_PLACEHOLDER and > IO_REPARSE_TAG_CLOUD_* tags during symlink evaluation, but even then > we still have to know what the reparse point buffer actually contains. > > Given that the content of reparse points with these reparse tags are > undocumented, some people using cloud services should examine these > reparse points so we can add some suitable code to Cygwin. Reading further on this it seems that one cannot easily compare these reparse points with symlinks. The tag values are 0x8015 for IO_REPARSE_TAG_FILE_PLACEHOLDER and 0x901AL up to 0x9000F01AL for the IO_REPARSE_TAG_CLOUD_* tags. None of them have the name surrogate bit set, so they don't "represent another named entity in the system". In other words, these reparse points represent themselves rather than pointing to some other file, as symlinks do. Additionally the IO_REPARSE_TAG_CLOUD_* tags all have the directory bit set so it seems they are used on the parent(?) directory of the local OneDrive copy only, but not on the files inside it. Bottom line: I wonder if the real deal is not the reparse tag and the reparse content, but whether or not the file has the FILE_ATTRIBUTE_OFFLINE flag set. If so, we can try to disable any action within path conversion, as well as in our stat(2) and readdir(3) implementation which would trigger onlining an offline file. Can anybody confirm that the idea is right, or if I'm something missing? Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote: We can add an explicit call to RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); and we can recognize the IO_REPARSE_TAG_FILE_PLACEHOLDER and IO_REPARSE_TAG_CLOUD_* tags during symlink evaluation, but even then we still have to know what the reparse point buffer actually contains. Given that the content of reparse points with these reparse tags are undocumented, some people using cloud services should examine these reparse points so we can add some suitable code to Cygwin. Corinna I'm not an expert in this area by any means but here are my recollections from when Microsoft presented in-person on cloud placeholders to filter and filesystem developers many years ago. Files and directories that are placeholders should have either the FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN file attributes set. When these attributes are set, applications and mini filters are advised not to "read" or "open" the files or directories unless they absolutely need to because doing so will cause the placeholder to be replaced by an object containing the actual data which might take a long time to fetch, might cost the end user money, or might fail depending upon the network connectivity. In particular, anti-malware should ignore them during scans and only analyze the data when it is fetched locally by an end user application. I believe that IO_REPARSE_TAG_FILE_PLACEHOLDER was replaced by IO_REPARSE_TAG_CLOUD_1 ..IO_REPARSE_TAG_CLOUD_F. Any reparse tag attached to a placeholder object is for the interpretation of the filter associated with the back-end storage and not for the consumption of applications. The content of the reparse tags can be back-end proprietary; different reparse data for onedrive, icloud, dropbox, etc. The default ProcessPlaceholderCompaibilityMode is PHCM_EXPOSE_PLACEHOLDERS which makes the FILE_ATTRIBUTE flags and reparse tags visible. Microsoft maintains a database of processes for which PHCM_DISGUISE_PLACEHOLDER is set which hides that information. Its unclear to me that explicitly setting the placeholder compatibility mode is useful. I'm not sure that exposing the object as a symlink is a good idea. A posix symlink is an object whose type and target information cannot change. In the case of a placeholder, the placeholder is silently replaced by the actual object either when the object is opened or the object's data is accessed. An application that believes it knows that the object is a symlink will be mighty confused when it turns out to be a file or a directory. Perhaps the question that needs to be asked is whether there are opens that can be skipped if an object is known to not be locally present (either of the FILE_ATTRIBUTE flags are set)? Jeffrey Altman -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On Mar 6 06:54, Brian Inglis via Cygwin wrote: > On 2024-03-06 06:28, Corinna Vinschen via Cygwin wrote: > > On Mar 6 14:22, Corinna Vinschen via Cygwin wrote: > > > Given these placeholder files are actually reparse points of type > > > IO_REPARSE_TAG_FILE_PLACEHOLDER, we can handle them as symbolic links. > > > > > > However, the structure of the IO_REPARSE_TAG_FILE_PLACEHOLDER reparse > > > data buffer is undocumented. It would be helpful if somebody using > > > OneDrive would examine the content of the attached REPARSE_DATA_BUFFER. > > > > > > > [2] > > > > https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 > > > > > > The NtReadFile call at this point is not the problem. It would be > > > helpful to point to Cygwin's source instead of MSYS2, btw. > > > > Oh, btw., this is from > > https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4: > > > >IO_REPARSE_TAG_FILE_PLACEHOLDER > >0x8015 > > > > Obsolete. > > - > > Used by Windows Shell for legacy placeholder files in Windows 8.1. > > Server-side interpretation only, not meaningful over the wire. > > > > So even if we support them, what is their replacement in W10 and later? > > May or not help: > > https://stackoverflow.com/questions/59152220/cant-get-reparse-point-information-for-the-onedrive-folder We can add an explicit call to RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS); and we can recognize the IO_REPARSE_TAG_FILE_PLACEHOLDER and IO_REPARSE_TAG_CLOUD_* tags during symlink evaluation, but even then we still have to know what the reparse point buffer actually contains. Given that the content of reparse points with these reparse tags are undocumented, some people using cloud services should examine these reparse points so we can add some suitable code to Cygwin. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On 2024-03-06 06:28, Corinna Vinschen via Cygwin wrote: On Mar 6 14:22, Corinna Vinschen via Cygwin wrote: On Mar 5 19:54, Marcin Wisnicki via Cygwin wrote: If I invoke ls or anything else that does stat inside OneDrive folder it will trigger download of all files. OneDrive uses placeholder files[1] to represent remote files. I'm guessing reading file content in stat is to support detection of actually executable files as in here[2]? I think this should be disabled on non-hydrated placeholder files. Running `find` or 'ls -R` and having your entire OneDrive downloaded is extremely problematic. I could live without executable scripts in the OneDrive folder and it's easy to mark files as always offline to solve it. Another idea is to skip checking files with extensions known to be non-executable such as jpg (or just any extensions that is not known to be executable). Nothing of this makes sense from a POSIX library POV. The library can either not handle placeholder files specially, as today, or it can handle them all the same way. Given these placeholder files are actually reparse points of type IO_REPARSE_TAG_FILE_PLACEHOLDER, we can handle them as symbolic links. However, the structure of the IO_REPARSE_TAG_FILE_PLACEHOLDER reparse data buffer is undocumented. It would be helpful if somebody using OneDrive would examine the content of the attached REPARSE_DATA_BUFFER. [2] https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 The NtReadFile call at this point is not the problem. It would be helpful to point to Cygwin's source instead of MSYS2, btw. Oh, btw., this is from https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4: IO_REPARSE_TAG_FILE_PLACEHOLDER 0x8015 Obsolete. - Used by Windows Shell for legacy placeholder files in Windows 8.1. Server-side interpretation only, not meaningful over the wire. So even if we support them, what is their replacement in W10 and later? May or not help: https://stackoverflow.com/questions/59152220/cant-get-reparse-point-information-for-the-onedrive-folder -- Take care. Thanks, Brian Inglis Calgary, Alberta, Canada La perfection est atteinte Perfection is achieved non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut -- Antoine de Saint-Exupéry -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On Mar 6 14:22, Corinna Vinschen via Cygwin wrote: > On Mar 5 19:54, Marcin Wisnicki via Cygwin wrote: > > If I invoke ls or anything else that does stat inside OneDrive folder > > it will trigger download of all files. > > > > OneDrive uses placeholder files[1] to represent remote files. > > > > I'm guessing reading file content in stat is to support detection of > > actually executable files as in here[2]? > > > > I think this should be disabled on non-hydrated placeholder files. > > Running `find` or 'ls -R` and having your entire OneDrive downloaded > > is extremely problematic. > > > > I could live without executable scripts in the OneDrive folder and > > it's easy to mark files as always offline to solve it. > > > > Another idea is to skip checking files with extensions known to be > > non-executable such as jpg (or just any extensions that is not known > > to be executable). > > Nothing of this makes sense from a POSIX library POV. The library can > either not handle placeholder files specially, as today, or it can > handle them all the same way. > > Given these placeholder files are actually reparse points of type > IO_REPARSE_TAG_FILE_PLACEHOLDER, we can handle them as symbolic links. > > However, the structure of the IO_REPARSE_TAG_FILE_PLACEHOLDER reparse > data buffer is undocumented. It would be helpful if somebody using > OneDrive would examine the content of the attached REPARSE_DATA_BUFFER. > > > [2] > > https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 > > The NtReadFile call at this point is not the problem. It would be > helpful to point to Cygwin's source instead of MSYS2, btw. Oh, btw., this is from https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4: IO_REPARSE_TAG_FILE_PLACEHOLDER 0x8015 Obsolete. - Used by Windows Shell for legacy placeholder files in Windows 8.1. Server-side interpretation only, not meaningful over the wire. So even if we support them, what is their replacement in W10 and later? Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On Mar 5 19:54, Marcin Wisnicki via Cygwin wrote: > If I invoke ls or anything else that does stat inside OneDrive folder > it will trigger download of all files. > > OneDrive uses placeholder files[1] to represent remote files. > > I'm guessing reading file content in stat is to support detection of > actually executable files as in here[2]? > > I think this should be disabled on non-hydrated placeholder files. > Running `find` or 'ls -R` and having your entire OneDrive downloaded > is extremely problematic. > > I could live without executable scripts in the OneDrive folder and > it's easy to mark files as always offline to solve it. > > Another idea is to skip checking files with extensions known to be > non-executable such as jpg (or just any extensions that is not known > to be executable). Nothing of this makes sense from a POSIX library POV. The library can either not handle placeholder files specially, as today, or it can handle them all the same way. Given these placeholder files are actually reparse points of type IO_REPARSE_TAG_FILE_PLACEHOLDER, we can handle them as symbolic links. However, the structure of the IO_REPARSE_TAG_FILE_PLACEHOLDER reparse data buffer is undocumented. It would be helpful if somebody using OneDrive would examine the content of the attached REPARSE_DATA_BUFFER. > [2] > https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 The NtReadFile call at this point is not the problem. It would be helpful to point to Cygwin's source instead of MSYS2, btw. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
ls/stat on OneDrive causes download of files
If I invoke ls or anything else that does stat inside OneDrive folder it will trigger download of all files. OneDrive uses placeholder files[1] to represent remote files. I'm guessing reading file content in stat is to support detection of actually executable files as in here[2]? I think this should be disabled on non-hydrated placeholder files. Running `find` or 'ls -R` and having your entire OneDrive downloaded is extremely problematic. I could live without executable scripts in the OneDrive folder and it's easy to mark files as always offline to solve it. Another idea is to skip checking files with extensions known to be non-executable such as jpg (or just any extensions that is not known to be executable). This was previously reported in https://github.com/msys2/msys2-runtime/issues/206. [1] https://learn.microsoft.com/en-us/windows/win32/w8cookbook/placeholder-files [2] https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple