[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: Nice. Did you considered the possibility to implement the pickling compatible with non-accelerated Python classes with __dict__ for every Cython class? I.e. __getstate__ that returns a dict that maps field names to values and __setstate__ that unpacks such

[issue22995] Restrict default pickleability

Stefan Behnel added the comment: For reference, the bug in Cython is fixed here: https://github.com/cython/cython/commit/ececb3e9473f6aaa65f29467921594c316ec2f06 -- ___ Python tracker

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: I've verified that backing out the typeobject.c change allows kivy and pysam to build. s3ql still fails but for other reasons, though its build does get past the cython failure. This clinches it for me. I am going to remove the typeobject.c clause from

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: On Jan 11, 2016, at 05:50 PM, Serhiy Storchaka wrote: > >Therefore my guess was true. There is a bug in Cython. It creates classes >that can't be correctly unpickled or copied. I don't think that's necessarily true. Clearly the classes *can* be copied and

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset e79eddcdff63 by Barry Warsaw in branch '3.5': Issue #22995: [UPDATE] Comment out the one of the pickleability tests in https://hg.python.org/cpython/rev/e79eddcdff63 New changeset 927fd0e14d49 by Barry Warsaw in branch 'default': Issue #22995:

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: Therefore my guess was true. There is a bug in Cython. It creates classes that can't be correctly unpickled or copied. The question is wherever we should replace TypeError with a warning in maintained releases, or remove this check at all? Here is a patch

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset 9d3ac16b78dc by Barry Warsaw in branch '2.7': Issue #22995: [UPDATE] Comment out the one of the pickleability tests in https://hg.python.org/cpython/rev/9d3ac16b78dc -- ___ Python tracker

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: The classes can't be *correctly* copied and unpickled because the pickle data doesn't contain object's state. "Copied" and "unpickled" objects are not equal to original objects, they are non-initialized instances. You can test attributes of copied

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: On Jan 11, 2016, at 07:20 PM, Serhiy Storchaka wrote: >I think we have to left this restriction in 3.6 (and Cython should fix its >bug by providing either __getnewargs__/ __getnewargs_ex__, __getstate__ or >__reduce__/__reduce_ex__). But in 2.7 and 3.5 we

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: BTW Serhiy, can you please file a bug with upstream Cython? I think you're the right person to do it since you added this restriction. With reverting this in 2.7 and 3.5 it's not urgent, but we don't want to have to reopen this when 3.6 is out. --

[issue22995] Restrict default pickleability

Changes by Serhiy Storchaka : -- nosy: +scoder ___ Python tracker ___ ___

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: So yeah, I guess Cython is doing something magical to boost the instance's type's basicsize, but I can't tell what from the Python prompt. It also clearly doesn't affect the un/picklability of the instance, so I think the entire check is probably misguided.

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: Your explanation still doesn't entirely make sense to me yet because the crash is happening in what seems to be a Cython implementation class, not a generated class based on a .pyx in some other project using Cython. I tried to search the Cython Trac but

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: You asked what reductor(4) returns in Python 3.5.0: > /usr/lib/python3.5/copy.py(176)deepcopy() -> rv = reductor(4) (Pdb) reductor(4) (, (,), None, None, None) And this is completely reasonable: (Pdb) p rv (, (,), None, None, None) (Pdb) p rv[0](*rv[1])

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: I'm not well known with Cython. Can it replace Python class with extension class on fly? There is a declaration of the NameAssignment class in Cython/Compiler/FlowControl.pxd. We need Cython expert. -- ___ Python

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: I added some debugging to the if(required) clause. This is interesting: basicsize 16 slotnames 0 basicsize 16 tp_basicsize 80 basicsize 16 tp_basicsize comes in at 16 bytes. tp_dictoffset and tp_weaklistoffset must both be 0, and while slotnames is not

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: On Jan 06, 2016, at 10:13 PM, Serhiy Storchaka wrote: >What return __reduce_ex__(4) for the NameAssignment instance in 3.5.0? I'm not sure yet (it takes some time to set up the right environment to test this), but I do know which TypeError is getting

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: What return __reduce_ex__(4) for the NameAssignment instance in 3.5.0? -- ___ Python tracker ___

[issue22995] Restrict default pickleability

Barry A. Warsaw added the comment: I'm reopening this issue because we're starting to see a bunch of regressions in Debian and Ubuntu we think is caused by the changes here. We've definitely started seeing them with 3.5.1, where packages that use Cython built just fine in 3.5.0 but now fail

[issue22995] Restrict default pickleability

Changes by Scott Kitterman : -- nosy: +kitterma ___ Python tracker ___ ___

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset 92172d7372dd by Serhiy Storchaka in branch '2.7': Issue #22995: Instances of extension types with a state that aren't https://hg.python.org/cpython/rev/92172d7372dd -- ___ Python tracker

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: That's all with this issue. Analyzing the signature of __new__ is not such easy and perhaps will be done in separate issue. -- assignee: -> serhiy.storchaka resolution: -> fixed stage: patch review -> resolved status: open -> closed

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset 0cd2de69fb66 by Serhiy Storchaka in branch '3.5': Issue #22995: Instances of extension types with a state that aren't https://hg.python.org/cpython/rev/0cd2de69fb66 New changeset b8d108a2a38e by Serhiy Storchaka in branch 'default': Issue #22995:

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: New patch more accurate detects objects with a state. It compares object's size with calculated minimal size, counting __dict__, __weakref__ and slots. -- versions: -Python 3.4 Added file: http://bugs.python.org/file41363/pickle_restrictions_5.patch

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: Was committed the restriction for tp_new == NULL. New patch uses more complex condition for types that doesn't define any reduce-related methods. This case covers _csv.Dialect and memoryview (and buffer in 2.7, issue8323). -- Added file:

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset c8841db9433d by Serhiy Storchaka in branch '3.4': Issue #22995: Default implementation of __reduce__ and __reduce_ex__ now https://hg.python.org/cpython/rev/c8841db9433d New changeset 4c05e7c195ac by Serhiy Storchaka in branch '3.5': Issue #22995:

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset 94664fb4354e by Serhiy Storchaka in branch '2.7': Issue #22995: Backported additional tests for non-pickleable types. https://hg.python.org/cpython/rev/94664fb4354e -- ___ Python tracker

[issue22995] Restrict default pickleability

Roundup Robot added the comment: New changeset 2b950eba9792 by Serhiy Storchaka in branch '2.7': Issue #22995: Default implementation of __reduce__ and __reduce_ex__ now https://hg.python.org/cpython/rev/2b950eba9792 -- ___ Python tracker

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: Could anyone please make a review? -- ___ Python tracker ___ ___

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: See also issue14350 and issue24900. Both will be solved by disabling pickling when tp_new is NULL (a part of proposed patch). So that I think the patch should be applied to all releases. -- type: enhancement -> behavior versions: +Python 2.7, Python

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: Ping. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995 ___ ___ Python-bugs-list mailing list

[issue22995] Restrict default pickleability

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +nadeem.vawda ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995 ___ ___

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: It is hard to write good tests, because every example is temporary, after adding support of pickling it will be not valid. Here is extended patch. It now handles Python subclasses of builtin classes (except list and dict which are pickleable via iterators)

[issue22995] Restrict default pickleability

Antoine Pitrou added the comment: How many cases does the patch catch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995 ___ ___

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: All public classes not designed for pickling explicitly. I tested only operator.methodcaller, mmap.mmap, sqlite3 classes (Connect, Cursor, Row), _socket.socket, select.epoll, _csv.Dialect, but should be more. Instances of these classes can be pickled, but

[issue22995] Restrict default pickleability

Serhiy Storchaka added the comment: Here is sample patch which adds two restrictions. Default reduce method for protocol = 2 will forbid pickling objects: 1) When tp_new == NULL. 2) Builtins without any of pickle-related methods: __getnewargs_ex__, __getnewargs__ or __getstate__. Are there

[issue22995] Restrict default pickleability

Changes by Serhiy Storchaka storch...@gmail.com: -- keywords: +patch Added file: http://bugs.python.org/file37494/pickle_restrictions.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995

[issue22995] Restrict default pickleability

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995 ___

[issue22995] Restrict default pickleability

Changes by Serhiy Storchaka storch...@gmail.com: -- title: Restrict default picleability - Restrict default pickleability ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995 ___

[issue22995] Restrict default pickleability

Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22995 ___ ___ Python-bugs-list mailing

[issue22995] Restrict default pickleability

Josh Rosenberg added the comment: Minor note: Python classes with __slots__ pickle and unpickle just fine under protocol 2 and higher. Only protocols 0 and 1 have this problem, and they are no longer used by default on Py3, and not necessary to communicate with Py2, where protocol 2 is