On 4/15/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
Actually, it does, you just have to force GC to clean up stuff (rather than wait for the interpreter-exit to just toss the whole lot out without giving it a second glance.) You do that by making it unreachable and calling gc.collect().
centurion:~/python/python/trunk > cat test_gencrash.py
import gc
class LazyList:
def __init__(self, g):
self.v = None
self.g = g
def __iter__(self):
yield 1
if self.v is None:
self.v = self.g.next()
yield self.v
def loop():
for i in ll:
yield i
ll = LazyList(loop())
g=iter(ll)
g.next ()
g.next()
del g, ll
print gc.collect()
centurion:~/python/python/trunk > ./python test_gencrash.py
Segmentation fault (core dumped)
I'm sure I know where the crash is coming from: one of the generator objects is being dealloced twice. The GC module tp_clears one of the frame objects, which deallocs a generator the frame referenced. That first dealloc calls _Py_ForgetReference and then calls gen_dealloc, which calls its tp_del (as it's a running generator, albeit a simple one), which raises an exception in the frame, which causes it to be cleaned up, which causes another generator object to be cleaned up and its tp_del method to be called, which - for some reason - tries to dealloc the first generator again. The second dealloc calls _Py_ForgetReference again, and _Py_ForgetReference doesn't like that (the ob_prev/ob_next debug ptrs are already wiped when it's called the second time, so it crashes.)
The thing I don't understand is how the cleaning up of the second generator triggers a dealloc of the first generator. The refcount is 0 when the dealloc is called the first time (or _Py_ForgetReference, as well as a few other places, would have bombed out), and it is also 0 when it gets dealloced the second time. Since deallocation is triggered by DECREF'ing, that implies something is INCREF'ing the first generator somewhere -- but without having a valid reference, since the generator has been cleaned up by actually DECREF'ing the reference that a frame object held. Or, somehow, deallocation is triggered by something other than a DECREF. Hmm. No, the second dealloc is triggered by the Py_XDECREF in ceval.c, when clearing the stack after an exception, which is the right thing to do. Time to set some breakpoints and step through the code, I guess.
Interestingly, this code does *not* crash the interpreter on its own, only
when run as a doctest (with or without regrtest).
Actually, it does, you just have to force GC to clean up stuff (rather than wait for the interpreter-exit to just toss the whole lot out without giving it a second glance.) You do that by making it unreachable and calling gc.collect().
centurion:~/python/python/trunk > cat test_gencrash.py
import gc
class LazyList:
def __init__(self, g):
self.v = None
self.g = g
def __iter__(self):
yield 1
if self.v is None:
self.v = self.g.next()
yield self.v
def loop():
for i in ll:
yield i
ll = LazyList(loop())
g=iter(ll)
g.next ()
g.next()
del g, ll
print gc.collect()
centurion:~/python/python/trunk > ./python test_gencrash.py
Segmentation fault (core dumped)
I still haven't figured
out what the actual problem is, but at least this cuts out all the merge()
and times() crud in the case this was derived from, reducing it to a pure
zen essence of non-meaning. :)
I'm sure I know where the crash is coming from: one of the generator objects is being dealloced twice. The GC module tp_clears one of the frame objects, which deallocs a generator the frame referenced. That first dealloc calls _Py_ForgetReference and then calls gen_dealloc, which calls its tp_del (as it's a running generator, albeit a simple one), which raises an exception in the frame, which causes it to be cleaned up, which causes another generator object to be cleaned up and its tp_del method to be called, which - for some reason - tries to dealloc the first generator again. The second dealloc calls _Py_ForgetReference again, and _Py_ForgetReference doesn't like that (the ob_prev/ob_next debug ptrs are already wiped when it's called the second time, so it crashes.)
The thing I don't understand is how the cleaning up of the second generator triggers a dealloc of the first generator. The refcount is 0 when the dealloc is called the first time (or _Py_ForgetReference, as well as a few other places, would have bombed out), and it is also 0 when it gets dealloced the second time. Since deallocation is triggered by DECREF'ing, that implies something is INCREF'ing the first generator somewhere -- but without having a valid reference, since the generator has been cleaned up by actually DECREF'ing the reference that a frame object held. Or, somehow, deallocation is triggered by something other than a DECREF. Hmm. No, the second dealloc is triggered by the Py_XDECREF in ceval.c, when clearing the stack after an exception, which is the right thing to do. Time to set some breakpoints and step through the code, I guess.
(I first thought the problem was caused by gen_dealloc doing 'Py_DECREF(gen->gen_frame)' instead of 'frame = gen->gen_frame; gen->gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should do it that way, I believe, but it's not the cause of this crash.)
--
Thomas Wouters <[EMAIL PROTECTED]>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
