[python-win32] file descriptor issues using msvcrt.open_osfhandle()+os.fdopen()
Greetings! Apologies in advance for the length of this post; after many days of scouring Python and Windows news groups, I've seen reports of issues similar to mine, but no cause or resolution other than re-trying an operation. I'm in no way a Windows programmer; I've spent my entire career on Unix/Linux, so the Windows ways are a bit of a mystery to me at times. That being said, I'm hoping that someone might shed some light on something stupid I've done (limited to this topic, please) or that I might actually save someone else's time when they see this post. I have been tracking down random descriptor issues with a multithreaded Windows service written in Python. The issue presents itself as bad file descriptor, attempt to read from something that is not a socket, and various errors closing files and sockets. In my service, I needed to be able to open a file with FILE_FLAG_BACKUP_SEMANTICS, create a Python file object and pass it off to another method to read. Since the Python posix file open() method doesn't provide a way to pass special Windozy flags, and I didn't see a way of setting that post-open, I followed a pattern that I found that looks like this: h = win32file.CreateFile( fname, win32file.GENERIC_READ, win32file.FILE_SHARE_READ, None, win32file.OPEN_EXISTING, win32file.FILE_FLAG_BACKUP_SEMANTICS, None ) fd = msvcrt.open_osfhandle(h, os.O_RDONLY) f = os.fdopen(fd, 'rb') That worked well and I was able to pass that 'f' off to be handled like any other file-like object. However, my multi-threaded service would randomly raise file descriptor errors reading files, logging and writing data on the network. What I found is that the file descriptors in the PyHANDLE and the file object seem to be somewhat out of sync during garbage collection. Closing the PyHANDLE leaves the file object with an invalid descriptor, and closing the file object leaves the PyHANDLE with an invalid descriptor, but only sometimes. If you put the code above into a loop, most of the time it will raise with a bad file descriptor error as the old objects go out of scope and are collected. I tried to remedy that by first calling CloseHandle() on the PyHANDLE and then close() on the file object (wrapped in a a try:..except IOError to deal with the already closed descriptor), and that makes the loop run, but still causes problems with other threads that just happen to open/close a d escriptor at the right/wrong time. So, my completely inexperienced (in Windows) empirical diagnosis is that there is some sort of synchronization issue between the msvcrt library and the CPython runtime that results in descriptors being shared among file and socket objects in a multithreaded application. At any rate, I replaced the CreateFile()/msvcrt.open_osfhandle()/os.fdopen() with a wrapper class that uses CreateFile()/ReadFile() and provides a minimal set of file-like object methods and all of my file descriptor and socket problems went away, so either I'm correct, or I just moved my problem. As I said in the beginning of this epic, I'm no more sure that I've found a problem with the underlying classes that manage descriptors on Windows than I've just replaced my own bad code with other not-as-bad code, but I hope that this rings a bell for someone that can set me straight. Thanks, Scott Leerssen ___ python-win32 mailing list python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] file descriptor issues using msvcrt.open_osfhandle()+os.fdopen()
2011/12/17 Scott Leerssen sleers...@gmail.com Closing the PyHANDLE leaves the file object with an invalid descriptor, and closing the file object leaves the PyHANDLE with an invalid descriptor, but only sometimes After open_osfhandle(), the handle is owned by the file object, and should not be closed when the PyHANDLE object is deallocated. Did you try to use the Detach() method? Something like fd = open_osfhandle(h.Detach(), os.O_RDONLY) Documentation says: You would call this function when you need the underlying win32 handle to exist beyond the lifetime of the handle object which seems adequate here. -- Amaury Forgeot d'Arc ___ python-win32 mailing list python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] file descriptor issues using msvcrt.open_osfhandle()+os.fdopen()
On Dec 17, 2011, at 11:36 AM, Amaury Forgeot d'Arc wrote: 2011/12/17 Scott Leerssen sleers...@gmail.com Closing the PyHANDLE leaves the file object with an invalid descriptor, and closing the file object leaves the PyHANDLE with an invalid descriptor, but only sometimes After open_osfhandle(), the handle is owned by the file object, and should not be closed when the PyHANDLE object is deallocated. Did you try to use the Detach() method? Something like fd = open_osfhandle(h.Detach(), os.O_RDONLY) Documentation says: You would call this function when you need the underlying win32 handle to exist beyond the lifetime of the handle object which seems adequate here. I did see that, but I interpreted that to mean that the PyHANDLE would be dereferenced from the underlying Windows file handle, and I'm not sure what the consequences of that would be. That is, how would I free that resource if I lose it from the Python context? Also, it didn't seem to fit since I didn't want a win32 handle to live past the close. I just wanted all the associated handles and descriptors to close at the same time. ___ python-win32 mailing list python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] file descriptor issues using msvcrt.open_osfhandle()+os.fdopen()
2011/12/17 Scott Leerssen sleers...@gmail.com I did see that, but I interpreted that to mean that the PyHANDLE would be dereferenced from the underlying Windows file handle, It's the other way round: the PyHandle just forgets the win32 handle value, and will not close it in its __del__. -- Amaury Forgeot d'Arc ___ python-win32 mailing list python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] file descriptor issues using msvcrt.open_osfhandle()+os.fdopen()
On Dec 17, 2011, at 12:25 PM, Amaury Forgeot d'Arc wrote: 2011/12/17 Scott Leerssen sleers...@gmail.com I did see that, but I interpreted that to mean that the PyHANDLE would be dereferenced from the underlying Windows file handle, It's the other way round: the PyHandle just forgets the win32 handle value, and will not close it in its __del__. Sorry, I did write that backward. How is the win32 handle freed? ___ python-win32 mailing list python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
Re: [python-win32] send message remote host
Yes, I'm thinking about net send of Windows ___ python-win32 mailing list python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
