[Michael Hudson]
>> I've been reading the bug report with interest, but unless I can
>> reproduce it it's mighty hard for me to debug, as I'm sure you know.
[Mike Klaas]
> Indeed.
Note that I just attached a much simpler pure-Python script that fails
very quickly, on Windows, using a debug build. Read the new comment
to learn why both "Windows" and "debug build" are essential to it
failing reliably and quickly ;-)
>>> Unfortunately, I've been attempting for hours to reduce the problem to a
>>> completely self-contained script, but it is resisting my efforts
due to timing
>>> problems.
Yes, but you did good! This is still just an educated guess on my
part, but my education here is hard to match ;-): this new business
of generators deciding to "clean up after themselves" if they're left
hanging appears to have made it possible for a generator to hold on to
a frame whose thread state has been free()'d, after the thread that
created the generator has gone away. Then when the generator gets
collected as trash, the new exception-based "clean up abandoned
generator" gimmick tries to access the generator's frame's thread
state, but that's just a raw C struct (not a Python object with
reachability-based lifetime), and the thread free()'d that struct when
the thread went away. The important timing-based vagary here is
whether dead-thread cleanup gets performed before the main thread
tries to clean up the trash generator.
> I've peered at the code, but my knowledge of the python core is
> superficial at best. The fact that it is occuring as a result of a
> long string of garbage collection/dealloc/etc. and involves threading
> lowers my confidence further. That said, I'm beginning to think that
> to reproduce this in a standalone script will require understanding
> the problem in greater depth regardless...
Or upgrade to Windows ;-)
>> Are you absolutely sure that the fault does not lie with any extension
>> modules you may be using? Memory scribbling bugs have been known to
>> cause arbitrarily confusing problems...
Unless I've changed the symptom, it's been reduced to minimal pure
Python. It does require a thread T, and creating a generator in T,
where the generator object's lifetime is controlled by the main
thread, and where T vanishes before the generator has exited of its
own accord.
Offhand I don't know how to repair it. Thread states /aren't/ Python
objects, and there's no provision for a thread state to outlive the
thread it represents.
> I've had sufficient experience being arbitrarily confused to never be
> sure about such things, but I am quite confident. The script I posted
> in the bug report is all stock python save for the operation in <>'s.
> That operation is pickling and unpickling (using pickle, not cPickle)
> a somewhat complicated pure-python instance several times.
FYI, in my whittled script, your `getdocs()` became simply:
def getdocs():
while True:
yield None
and it's called only once, via self.docIter.next(). In fact, the
"while True:" isn't needed there either (given that it's only resumed
once now).
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
