Greetings!
Apologies in advance for the length of this post; after many days of scouring
Python and Windows news groups, I've seen reports of issues similar to mine,
but no cause or resolution other than re-trying an operation. I'm in no way a
Windows programmer; I've spent my entire career on Unix/Linux, so the Windows
ways are a bit of a mystery to me at times. That being said, I'm hoping that
someone might shed some light on something stupid I've done (limited to this
topic, please) or that I might actually save someone else's time when they see
this post.
I have been tracking down random descriptor issues with a multithreaded Windows
service written in Python. The issue presents itself as "bad file descriptor",
"attempt to read from something that is not a socket", and various errors
closing files and sockets.
In my service, I needed to be able to open a file with
FILE_FLAG_BACKUP_SEMANTICS, create a Python file object and pass it off to
another method to read. Since the Python posix file open() method doesn't
provide a way to pass special Windozy flags, and I didn't see a way of setting
that post-open, I followed a pattern that I found that looks like this:
h = win32file.CreateFile(
fname,
win32file.GENERIC_READ,
win32file.FILE_SHARE_READ,
None,
win32file.OPEN_EXISTING,
win32file.FILE_FLAG_BACKUP_SEMANTICS,
None
)
fd = msvcrt.open_osfhandle(h, os.O_RDONLY)
f = os.fdopen(fd, 'rb')
That worked well and I was able to pass that 'f' off to be handled like any
other file-like object. However, my multi-threaded service would randomly
raise file descriptor errors reading files, logging and writing data on the
network. What I found is that the file descriptors in the PyHANDLE and the
file object seem to be somewhat out of sync during garbage collection. Closing
the PyHANDLE leaves the file object with an invalid descriptor, and closing the
file object leaves the PyHANDLE with an invalid descriptor, but only sometimes.
If you put the code above into a loop, most of the time it will raise with a
bad file descriptor error as the old objects go out of scope and are collected.
I tried to remedy that by first calling CloseHandle() on the PyHANDLE and then
close() on the file object (wrapped in a a try:..except IOError to deal with
the already closed descriptor), and that makes the loop run, but still causes
problems with other threads that just happen to open/close a d
escriptor at the right/wrong time.
So, my completely inexperienced (in Windows) empirical diagnosis is that there
is some sort of synchronization issue between the msvcrt library and the
CPython runtime that results in descriptors being shared among file and socket
objects in a multithreaded application.
At any rate, I replaced the CreateFile()/msvcrt.open_osfhandle()/os.fdopen()
with a wrapper class that uses CreateFile()/ReadFile() and provides a minimal
set of file-like object methods and all of my file descriptor and socket
problems went away, so either I'm correct, or I just moved my problem.
As I said in the beginning of this epic, I'm no more sure that I've found a
problem with the underlying classes that manage descriptors on Windows than
I've just replaced my own bad code with other not-as-bad code, but I hope that
this rings a bell for someone that can set me straight.
Thanks,
Scott Leerssen
_______________________________________________
python-win32 mailing list
python-win32@python.org
http://mail.python.org/mailman/listinfo/python-win32
