[issue1062277] Pickle breakage with reduction of recursive structures

[Daniel Diniz]

Change by Daniel Diniz :


--
versions: +Python 3.11 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Davis Herring]

Davis Herring added the comment:

Re msg110868: it's impossible to resolve a __reduce__ loop that involves only 
immutable intermediate objects (including none at all):

class Direct:
  def __reduce__(self): return id,(self,) # obviously impossible
class Indirect:
  # Can't create either the new object or the tuple first!
  def __reduce__(self): return id,((self,),)

With pre-existing mutable objects, the same trick as for tuples certainly could 
be applied:

class Mutable:
  def __init__(self): self.bracketed=[self]
  # Create list, REDUCE, then populate list
  def __reduce__(self): return id,(self.bracketed,)

The way an analog to the tuple implementation would deal with this would be 
something like

id   # the dummy "constructor" in this example
[]   # empty list, memoized immediately as, say, #5
id   # try again to store the Mutable
@5   # fetch from memo
T1   # make singleton tuple
R# reduce (have now succeeded in "making" the Mutable as, say, #20)
APP  # to the list
T1   # finish the first __reduce__ attempt
POP  # abandon it, since the object has been created
POP  # abandon the "id" as well
@20  # fetch the previous answer

If the list were created directly in __reduce__, you'd still recurse infinitely 
trying to create each new list object.  On the other hand, even complicated 
cases like this should work:

class Switch:
  def __reduce__(self): return id,(self.lst,)
a=Switch(); b=Switch()
a.list=[b,a]; b.list=[a,b]

where the pickling of, say, a would look something like

id
[]# -> #17
id# trying b now
[]# -> #42
id# trying a again
@17
T1
R # a done (#88)
APP
id# trying b again
@42
T1
R # b done (#101)
APP   # b's list now populated
POP
POP   # b abandoned
@101  # b fetched
APP   # finally building a's list
@88   # a is long done
APP   # a's list now populated
POP
POP   # a abandoned
@88   # final value: a

Perhaps a technique for distinguishing these cases is to look for new objects 
having been created since the last time we visited an object.  If there are 
none, we're in the immutable case and we lose.  If there are yet more created 
between the second and third visits, on the other hand, we're generating more 
objects from __reduce__ every time and should again abort.

--
nosy: +herring

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

Problem B has been resolved, but problem A is still there.

Python 3.4.1 (default, Jun 29 2014, 15:26:46)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
Type help, copyright, credits or license for more information.
 class C:
... def __init__(self, x=None):
... self.x = x if x is not None else self
... def __reduce__(self):
... return C, (self.x,)
...
 import pickle
 pickle.dumps(C())
Traceback (most recent call last):
  File stdin, line 1, in module
RuntimeError: maximum recursion depth exceeded while calling a Python object
 c = C([])
 c.x.append(c)
 c.x[0] is c
True
 c2 = pickle.loads(pickle.dumps(c))
 c2.x[0] is c2
True

--
assignee: belopolsky - 
versions: +Python 3.5 -Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Eugene Toder]

Eugene Toder added the comment:

To recap, the issue is that pickle doesn't handle recursion via reduce 
arguments (i.e. arguments to the constructor function as returned in 2nd 
element of the tuple from __reduce__). This leads to 2 kind of effects:

class C:
def __init__(self, x=None):
self.x = x if x is not None else self
def __reduce__(self):
return C, (self.x,)

A. Recursion error:
 pickle.dumps(C())
Traceback (most recent call last):
  File pyshell#5, line 1, in module
pickle.dumps(C())
RuntimeError: maximum recursion depth exceeded while calling a Python object

This cannot be helped with the current reduce protocol. The error may be 
improved, but that's about it.

B. Duplication of object when unpickling:
 c = C([])
 c.x.append(c)
 c.x[0] is c
True
 c2 = pickle.loads(pickle.dumps(c))
 c2.x[0] is c2
False

This happens because list (or another recursion-friendly type) inside the 
problematic object handles recursion, but we still get the outer object, 
identical to the inner one.
This can be solved the same way as for tuple:
 t=([],1,2)
 t[0].append(t)
 t2 = pickle.loads(pickle.dumps(t))
 t2[0][0] is t2
True
 pickletools.dis(pickle.dumps(t))
0: \x80 PROTO  3
2: ]EMPTY_LIST
3: qBINPUT 0
5: hBINGET 0
7: KBININT11
9: KBININT12
   11: \x87 TUPLE3
   12: qBINPUT 1
   14: aAPPEND
   15: KBININT11
   17: KBININT12
   19: 0POP
   20: 0POP
   21: 0POP
   22: hBINGET 1
   24: .STOP

After pickling its elements tuple checks if it got into memo. If it did, this 
means it was pickled by one of the elements, so it POPs all elements from the 
stack and fetches itself via GET. This is somewhat inefficient, but probably 
the best it can do.

I suggest we do 3 things:

1. Improve the documentation for __reduce__ function. It should mention that 
all state that a) can potentially point back to the object and b) not strictly 
necessary in the constructor function should be passed via the 3rd element of 
__reduce__ tuple (aka state) instead of the 2nd element, and applied by 
__setstate__. This handles recursion in robust and optimal way.

2. Check that all built-in/standard types follow this advice. I see that Stefan 
Mihaila already fixed sets.

3. To fix case B above add the memo check from save_tuple to save_reduce. While 
at it, we can consider checking after pickling every element instead of after 
pickling all elements, so we reduce the number of POPs and the wastage they 
cause.

--
nosy: +eltoder, pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Eugene Toder]

Changes by Eugene Toder elto...@gmail.com:


--
versions: +Python 3.3, Python 3.4 -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Stefan Mihaila]

Changes by Stefan Mihaila mstefa...@gmail.com:


--
nosy: +mstefanro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[mike bayer]

Changes by mike bayer mike...@zzzcomputing.com:


--
nosy: +zzzeek

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Alex Gaynor]

Changes by Alex Gaynor alex.gay...@gmail.com:


--
nosy: +alex

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[bcroq]

Changes by bcroq bertrand.c...@gmail.com:


--
nosy: +bcroq

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Mark Lawrence]

Changes by Mark Lawrence breamore...@yahoo.co.uk:


--
type: feature request - behavior
versions: +Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
components: +Library (Lib) -Interpreter Core

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Alexander Belopolsky]

Alexander Belopolsky belopol...@users.sourceforge.net added the comment:

As I explained in msg110617 under issue9269, it is possible that we can do 
better than simply detect reduce cycles and bail out.

I am torn between two options:

1. Reject this patch and wait until a proper solution is found.

2. Admit that better is the enemy of good and start with proper error reporting 
on reduce cycles by accepting this patch.

My only concern about this patch is that it is likely to affect performance 
because pickling will have to maintain the reducing_now dictionary that is 
parallel to the memo dictionary.

--
superseder:  - Cannot pickle self-referencing sets

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Alexander Belopolsky]

Alexander Belopolsky belopol...@users.sourceforge.net added the comment:

The issue is still present in py3k.  Attaching an updated patch with tests 
only.  Is this the same as issue998998?

--
assignee:  - belopolsky
nosy: +alexandre.vassalotti, belopolsky
stage:  - needs patch
versions: +Python 3.2
Added file: http://bugs.python.org/file18011/issue1062277-test-py3k.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1062277] Pickle breakage with reduction of recursive structures

[Daniel Diniz]

Daniel Diniz aja...@gmail.com added the comment:

Patch has test, can someone try it? I think it might be fixed already.

--
nosy: +ajaksu2
type:  - feature request
versions: +Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1062277
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com