#14471: Exception KeyError: in WeakValueDictionary remove ignored
---------------------------+------------------------------------------------
Reporter: vbraun | Owner: rlm
Type: defect | Status: new
Priority: major | Milestone: sage-5.10
Component: memleak | Resolution:
Keywords: | Work issues:
Report Upstream: N/A | Reviewers:
Authors: | Merged in:
Dependencies: | Stopgaps:
---------------------------+------------------------------------------------
Comment (by nbruin):
Replying to [comment:7 vbraun]:
> Fair enough, but even after I catch the !KeyError the `wr.key in
self.data.keys()` is still `True`. So the `self.data` dict is in a weird
place for sure.
That is weird, and sounds to me like `self.data` is indeed corrupted.
Given that key/value pairs are stored together in a dictionary, it's very
hard for this to be true too. Caching the result from data.keys wouldn't
make sense at all, so I don't think python will be doing that. Perhaps
check that something like `wr.key in self.data` or
`self.data.__contains__(wr.key)` returns true and/or if
`value=self.data[wr.key]` succeeds (and what kind of value you get back:
whether it's `wr`)
That might give you some indication of what exactly is wrong with the
entry and hence where the problem might be coming from.
One thing your traceback is confirming:
- There is no GC going on, so callbacks are happening when you think they
should.
- Hence there's also not a circular data structure directly involved
- Things are happening during cleanup, so these callbacks are probably
indeed happening on a dict that will be deleted too. It's still bad they
don't properly execute, of course, but we've seen before that order in
cleanup can sometimes lead to subtle problems (i.e., whether things are
stored in a cdef attribute or in a conventional dict).
One possible scenario:
- self.data is in the process of being cleared. The dict is actually
emptied then before any entries are decreffed (to prevent dict reshaping
during deallocation).
- One of the weakreffed values also occurs as a key in the dict (and
that's the only strong ref)
- decreffing that key causes the callback, but the callback finds a dict
that is now empty, so of course we get a key error.
Your diagnosis that `wr.key in self.data` is still true is not consistent
with that, but the scenario indeed leads to ignored keyerrors:
{{{
import weakref
class A(object):
def __init__(self,n):
self.n=n
def __repr__(self):
return "A(%d)"%self.n
def mess(n):
D=weakref.WeakValueDictionary()
L=[A(i) for i in range(n)]
for i in range(n-1):
j=(i+10)%n
D[L[i]]=L[j]
return D
D=mess(10000)
D.clear()
}}}
(how many you ignored keyerrors you get depends a bit on memory layout
etc.)
This is definitely a bug in the python library. The problem is that
`remove` does check if the dict still exists (via the weakref, which is
mainly to ensure that presence of callbacks doesn't extend the life of the
dict), but doesn't check if the dict is otherwise in a funny state, such
as "in the process of a `clear`".
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/14471#comment:9>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
