Re: [python-win32] pure python way to open a file with write deny for others
> On Mar 5, 2020, at 5:42 AM, Robin Becker wrote: > > > [EXTERNAL EMAIL] > I want to be able to read a windows file which is being periodically written > by another process. I created a small extension ... > > that seems to work, but I wonder if there's an easier pure python way to do > this. Looking at the docs I see O_EXCL, but the _SH_DENY flags seem to be > absent. If the _fsopen function is present then you might try passing the numeric value of those flags. If the real issue is the lack of a Python wrapper around _fsopen, look at the ctypes module. That offers a simple way to wrap existing shared library (DLL) APIs to make them visible as Python functions, without the need to write your own extension modules. The documentation for ctypes in the Python library manual is quite good, and includes Windows specific examples. paul ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
On 06/03/2020 22:04, Eryk Sun wrote: On 3/6/20, Robin Becker wrote: OK I want to read the (small) file completely. The other process may try to re-write the file while I am reading it. I thought the other process already had the file open for writing, but apparently you just want to deny anyone the right to open the file with write access while you're reading it. If the file is already open with write access, you'll have to periodically retry opening it until you can get read access without sharing write access. yes we poll periodically for late mod time files and try to post these to a web server. If there are misreads we hope that the next poll will find them not being written. After some of the thoughts posted here I suspect we probably ought not to try to prevent the other process writing. What we are really trying to do is get a snapshot of files which have changed since our last poll read. The real problem is to ensure that the content read is not partial ie was not being written at the time of read. Alternatively, you can use regular read-write sharing, but lock the file data via msvcrt.locking. This doesn't deny opening the file for write access, but any attempt to write to a locked region will fail with a locking violation. The locked region can encompass the entire file. (File locking in Windows doesn't prevent deleting or renaming a file, but the default share mode already prevents that.) unfortunately I cannot assume they are appending so partial locking is probably not useful. Of course the other process may adopt a completely orthogonal scheme of opening with a different name and then renaming, Normally, open files in Python cannot be deleted or renamed until they're closed. This is the case as long as you have the file opened with just read or read-write sharing. Within the standard library, only the low-level open flag O_TEMPORARY uses shared delete/rename access. -- Robin Becker ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
On 06/03/2020 21:04, Preston Landers wrote: Is advisory file locking an option? Such as the "portalocker" module for Python? You can have your writer process obtain an exclusive lock (and block until it's obtained), while the readers obtain shared locks for the duration of their read. thanks, but I have no control over (and little knowledge about) the other process; the intention is to poll an output folder for say *.txt files and post these to a web server for processing. Readers don't block other readers, while writers block both writers and readers. . -- Robin Becker ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
On 3/6/20, Robin Becker wrote: > > OK I want to read the (small) file completely. The other process may try to > re-write the file while I am reading it. I thought the other process already had the file open for writing, but apparently you just want to deny anyone the right to open the file with write access while you're reading it. If the file is already open with write access, you'll have to periodically retry opening it until you can get read access without sharing write access. Alternatively, you can use regular read-write sharing, but lock the file data via msvcrt.locking. This doesn't deny opening the file for write access, but any attempt to write to a locked region will fail with a locking violation. The locked region can encompass the entire file. (File locking in Windows doesn't prevent deleting or renaming a file, but the default share mode already prevents that.) > Of course the other process may adopt a completely orthogonal scheme of > opening with a different name and then renaming, Normally, open files in Python cannot be deleted or renamed until they're closed. This is the case as long as you have the file opened with just read or read-write sharing. Within the standard library, only the low-level open flag O_TEMPORARY uses shared delete/rename access. ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
Is advisory file locking an option? Such as the "portalocker" module for Python? You can have your writer process obtain an exclusive lock (and block until it's obtained), while the readers obtain shared locks for the duration of their read. Readers don't block other readers, while writers block both writers and readers. https://github.com/WoLpH/portalocker On Fri, Mar 6, 2020 at 10:51 AM Robin Becker wrote: > > On 05/03/2020 16:04, Eryk Sun wrote: > > On 3/5/20, Robin Becker wrote: > >> I want to be able to read a windows file which is being periodically > >> written > >> by another process. > > > > I'm having difficulty reconciling this sentence with the subject line. > > OK I want to read the (small) file completely. The other process may try to > re-write the file while I am reading it. I > thought that denying them write permission for the short time I have the file > open for reading might make incomplete > files less likely. So far the applications seem to be able to operate in this > fashion and the small files seem to be > complete. > > Of course the other process may adopt a completely orthogonal scheme of > opening with a different name and then renaming, > but I would then try to read the new version as its time stamp would change. > We have no access to the internals of the > writer and are just attempting to push textfile changes from a folder to a > server. Perhaps there's a better way to do that. > > > > If you want to open a file for reading that's already open for > > writing, then the open has to share write access, not deny it (where > > to "deny" access means to not share access) [1]. That's not an issue > > since Python shares read and write access when opening a file. (Note > > that the share mode applies to all opens. It is not related to > > processes, so whether it's another process or the current process > > that's writing to the file is irrelevant to the problem.) > > > > I wouldn't use FILE streams in Python 3. I'd pass an `opener` to > > Python `open` that uses ctypes or PyWin32 to get a handle via > > `CreateFileW`, and then msvcrt.open_osfhandle to wrap the handle with > > a C file descriptor. The opener function signature is opener(filename, > > flags) -> fd. > > > > [1]: > > https://docs.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files > > > > > -- > Robin Becker > ___ > python-win32 mailing list > python-win32@python.org > https://mail.python.org/mailman/listinfo/python-win32 ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
Robin Becker wrote: On 05/03/2020 16:04, Eryk Sun wrote: On 3/5/20, Robin Becker wrote: I want to be able to read a windows file which is being periodically written by another process. I'm having difficulty reconciling this sentence with the subject line. OK I want to read the (small) file completely. The other process may try to re-write the file while I am reading it. I thought that denying them write permission for the short time I have the file open for reading might make incomplete files less likely. So far the applications seem to be able to operate in this fashion and the small files seem to be complete. Remember that the "deny write" permission only applies to opens. And if you have "deny write" set, the other open will fail -- it won't just delay. You can always use win32file.CreateFile directly, and bypass the Python filtering. Alternatively, and perhaps more hacky, you can use subprocess to copy the file to a safe temporary name. -- Tim Roberts, t...@probo.com Providenza & Boekelheide, Inc. smime.p7s Description: S/MIME Cryptographic Signature ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
On 05/03/2020 16:04, Eryk Sun wrote: On 3/5/20, Robin Becker wrote: I want to be able to read a windows file which is being periodically written by another process. I'm having difficulty reconciling this sentence with the subject line. OK I want to read the (small) file completely. The other process may try to re-write the file while I am reading it. I thought that denying them write permission for the short time I have the file open for reading might make incomplete files less likely. So far the applications seem to be able to operate in this fashion and the small files seem to be complete. Of course the other process may adopt a completely orthogonal scheme of opening with a different name and then renaming, but I would then try to read the new version as its time stamp would change. We have no access to the internals of the writer and are just attempting to push textfile changes from a folder to a server. Perhaps there's a better way to do that. If you want to open a file for reading that's already open for writing, then the open has to share write access, not deny it (where to "deny" access means to not share access) [1]. That's not an issue since Python shares read and write access when opening a file. (Note that the share mode applies to all opens. It is not related to processes, so whether it's another process or the current process that's writing to the file is irrelevant to the problem.) I wouldn't use FILE streams in Python 3. I'd pass an `opener` to Python `open` that uses ctypes or PyWin32 to get a handle via `CreateFileW`, and then msvcrt.open_osfhandle to wrap the handle with a C file descriptor. The opener function signature is opener(filename, flags) -> fd. [1]: https://docs.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files -- Robin Becker ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
On 3/5/20, Robin Becker wrote: > I want to be able to read a windows file which is being periodically written > by another process. I'm having difficulty reconciling this sentence with the subject line. If you want to open a file for reading that's already open for writing, then the open has to share write access, not deny it (where to "deny" access means to not share access) [1]. That's not an issue since Python shares read and write access when opening a file. (Note that the share mode applies to all opens. It is not related to processes, so whether it's another process or the current process that's writing to the file is irrelevant to the problem.) I wouldn't use FILE streams in Python 3. I'd pass an `opener` to Python `open` that uses ctypes or PyWin32 to get a handle via `CreateFileW`, and then msvcrt.open_osfhandle to wrap the handle with a C file descriptor. The opener function signature is opener(filename, flags) -> fd. [1]: https://docs.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] pure python way to open a file with write deny for others
On 05/03/2020 14:19, paul.kon...@dell.com wrote: On Mar 5, 2020, at 5:42 AM, Robin Becker wrote: [EXTERNAL EMAIL] I want to be able to read a windows file which is being periodically written by another process. I created a small extension ... that seems to work, but I wonder if there's an easier pure python way to do this. Looking at the docs I see O_EXCL, but the _SH_DENY flags seem to be absent. If the _fsopen function is present then you might try passing the numeric value of those flags. If the real issue is the lack of a Python wrapper around _fsopen, look at the ctypes module. That offers a simple way to wrap existing shared library (DLL) APIs to make them visible as Python functions, without the need to write your own extension modules. The documentation for ctypes in the Python library manual is quite good, and includes Windows specific examples. paul yes I thought about doing this when the app moves to python 3.x; I'm only using the deny write value so passing the integer could work. -- Robin Becker ___ python-win32 mailing list python-win32@python.org https://mail.python.org/mailman/listinfo/python-win32
