#15156: Pickling with __reduce__() loops creates invalid pickles
-------------------------------------+-------------------------------------
Reporter: vbraun | Owner:
Type: defect | Status: needs_info
Priority: major | Milestone: sage-6.3
Component: misc | Resolution:
Keywords: | Merged in:
Authors: SimonKing, | Reviewers:
jkeitel, novoselt, nbruin | Work issues:
Report Upstream: N/A | Commit:
Branch: | 82ba456b8c828700d28ac36822b300f00d8b1308
u/saraedum/ticket/15156 | Stopgaps:
Dependencies: #15692 |
-------------------------------------+-------------------------------------
Comment (by nbruin):
Replying to [comment:18 saraedum]:
> I don't think that's really what happens. Maybe I misunderstand what you
are saying but I don't think that all objects in the pickle are
instantiated and then all objects' `__setstate__` is called.
Not all objects. Just all objects that are involved making the relevant
`__setstate__` call. And of course this has to happen, otherwise there are
no objects to pass.
> Have a look at the example I posted above. There, to unpickle `b`, `a`
is created, its `__setstate__` is called (which fails), and only then `b`
is created and its `__setstate__` is called.
>
> > > There is of course lots of space for improvement, but what do you
think about this idea?
> > I think it's a bad idea because it doesn't solve the problem where it
needs to be solved. Homsets fundamentally need domain and codomain for
their construction. It's part of their hash. If a parent wants to include
a homset in its construction parameter rather than in the `__setstate__`
then it is introducing an unavoidable cycle.
> This is not what happens. Here is a real-world example:
> {{{
> ...
> sage: pickle.dumps(H)
> AssertionError
> }}}
That is indeed puzzling. Would python `pickle` be unnecessarily strict
compared to `cPickle`? It seems to me that `cPickle` produces a perfectly
workable pickle from this:
{{{
sage: p=dumps(H)
sage: explain_pickle(p)
pg_Hom = unpickle_global('sage.categories.homset', 'Hom')
pg_A = unpickle_global('__main__', 'A')
pg = unpickle_instantiate(pg_A, ())
pg_unreduce = unpickle_global('sage.structure.unique_representation',
'unreduce')
pg_Sets = unpickle_global('sage.categories.sets_cat', 'Sets')
si = pg_unreduce(pg_Sets, (), {})
unpickle_build(pg, pg_Hom(pg, pg, si, False))
pg_Hom(pg, pg, si, False)
}}}
Also for your earlier example, it looks like the pickle produced by dumps
should be just fine:
{{{
sage: explain_pickle(dumps(b))
pg_B = unpickle_global('__main__', 'B')
pg_A = unpickle_global('__main__', 'A')
pg = unpickle_instantiate(pg_A, ())
unpickle_build(pg, {'b':pg_B(pg)})
pg_B(pg)
sage: explain_pickle(dumps(b.a))
pg_A = unpickle_global('__main__', 'A')
pg = unpickle_instantiate(pg_A, ())
pg_B = unpickle_global('__main__', 'B')
unpickle_build(pg, {'b':pg_B(pg)})
pg
}}}
> That's what happens above and does not work. I'm also confused because I
always assumed that it would be enough to break the circularity on the
first parameter in a single place. Somehow this does not seem to work.
Well -- it does not work in the sense that `pickle.dumps` raises an
AssertionError. Since that defaults to protocol 0, while sage has already
settled on protocol 2, it's not an entirely fair test. However, specifying
`protocol=2` doesn't help, so it's really a problem in pickle and possibly
also in cPickle. However, the code produced by `explain_pickle` seems sane
and that is directly interpreted from the produced pickle, so it could
well be that cPickle does perform as we hope.
I did a quick search on whether this is a known problem, but I didn't find
compelling evidence. Closest was
[http://bytes.com/topic/python/answers/37656-assertionerror-pickles-
memoize-function this discussion]
--
Ticket URL: <http://trac.sagemath.org/ticket/15156#comment:19>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/d/optout.
