[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Eryk Sun added the comment: > As far as I know os.stat() resets d.stat() maybe should be added > some option to d.stat() to force update(). d.stat(nt_force_update=True). It depends on the filesystem. NTFS will update the directory entry as soon as the link is accessed by CreateFileW. But

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: As far as I know os.stat() resets d.stat() maybe should be added some option to d.stat() to force update(). d.stat(nt_force_update=True). I am not sure if os.path.getmtime() can reset d.stat(). os.stat() is 2x times slower than os.path.getmtime() and

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Steve Dower added the comment: Those are all good ideas, but using os.stat(d) instead of d.stat() is shorter, more reliable, more compatible, and already works. There's no middle ground where DirEntry can be faster, because it's already using that middle ground. All the discussion between

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: I think we can assume that NTFS is priority since that is the most used option. I can not discuss what with FAT32 or FAT since I am not the best in this domain (in NTFS I am not the best too now). Whatever I think that system must do allocation for open

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Steve Dower added the comment: > We're faced with the choice between either always calling the real lstat, or > just documenting that files with hard links will have stale information if > the file was updated using another link. That's an easy choice: we document it. The os module comes

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Eryk Sun added the comment: > What it also means is that the "file still in use by another app" > scenario will probably have to manually use os.stat(). We can't > detect it, and it's the same race condition as calling os.stat() > shortly before the update flushes anyway. FAT filesystems

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Steve Dower added the comment: Okay, so it sounds like there's a class of files where we can't rely on the FindFileData having the right values. But we get enough information to be able just suppress the caching behaviour for those, right? Basically, my criteria for fixing this in the

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Eryk Sun added the comment: > Does it make the most sense for us to make .flush() also do an > implicit .fsync() (when it's actually a file object)? Standard I/O in the Windows C runtime supports a "c" commit mode that causes fflush to call _commit() on the underlying fd [1]. Perhaps Python

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: I read some comments os.flush() or os.fsync() can be unrelated to problem. External application can be written in C# or whatever you want. Under Windows (not Linux) - modification dates will be stalled in such sequence. os.scandir() dir_entry.stat() # let it

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: Use case - detection of changes in open files is very important - log scanning - synchronization ... I think that first of all it is need good unit test to detect this problem (rare edge case - probably it is missed because hard to imagine that it can not

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: I do some test on linux all works - changes are detected and os.scandir() works but in Windows not - probably there is not unit test which check if os.scandir() is working on open files for writing. f.flush() no matter since file can be changed in external

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Steve Dower added the comment: Does it make the most sense for us to make .flush() also do an implicit .fsync() (when it's actually a file object)? -- ___ Python tracker ___

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Eryk Sun added the comment: In FSBO [1] section 6 "Time Stamps", note that the LastWriteTime value gets updated when an IRP_MJ_FLUSH_BUFFERS is processed. In the Windows API, this is a FlushFileBuffers [2] call. In the C runtime, it's a _commit [3] call, which is an os.fsync [4] call in

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Steve Dower added the comment: I'm going to have to spend more time to analyse this (later), but it seems like Windows deciding not to update the directory's data structures (containing the st_mtime retrieved by scandir) as long as the file is still open. I suspect the answer for your

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: One hint more. Start of new process os.scandir() give invalid modification date for file open for writing until external tool is not called (like explorer, touch, etc.). So (log open for writing and write is done between 1, 2): 1. Run program with

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

Cezary Wagner added the comment: Extra file for for tests with: DO_STAT = False See not changes but file was writing every second. If os.stat() run all between call os.scandir() all works. C:\root\Python38\python.exe

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

New submission from Cezary Wagner : I have problem with change detection of log during writing under Windows (normal fs and windows share). Probably bad order of Windows API calls - no idea. Test program is attached. You can reproduce it. Try with os.scandir() without os.stats() and