[issue42360] In the namedtuple documentation, mention that typename should match the variable name for the class to be pickle-able
New submission from Alejandro Gonzalez :
I think the namedtuple documentation should mention that, for classes created
with it to be pickle-able, the typename argument must match the name of the
variable the class is being assigned to. Otherwise you get an error like this:
--
>>> Foo = namedtuple("Bar", "x,y")
>>> pickle.dumps(Foo(1, 2))
Traceback (most recent call last):
File "", line 1, in
_pickle.PicklingError: Can't pickle : attribute lookup
Bar on __main__ failed
--
While it is indeed odd to do such naming in the first place, it should be
admissible in other circumstances not involving pickling, and it is not obvious
that pickling won't work if you do so.
The pickle documentation does mention this, though:
[...]Similarly, classes are pickled by named reference, so the same
restrictions in the unpickling environment apply.[...]
but for someone who encounters this error it might be difficult to figure it
out from that passage.
The added documentation in namedtuple could include a pointer to that section
of pickle's documentation.
--
assignee: docs@python
components: Documentation
messages: 381007
nosy: alegonz, docs@python
priority: normal
severity: normal
status: open
title: In the namedtuple documentation, mention that typename should match the
variable name for the class to be pickle-able
type: enhancement
___
Python tracker
<https://bugs.python.org/issue42360>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Alejandro Gonzalez added the comment: I see, that’s an interesting point. In that case, Serhiy’s approach is indeed better. PR 14166 seems stalled. I’d like to help if there’s anything left to do, if possible :) -- ___ Python tracker <https://bugs.python.org/issue28869> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Alejandro Gonzalez added the comment: Hello, Could anyone take a look at the PR 14126 submitted? Sorry for the trouble and thank you in advance. -- ___ Python tracker <https://bugs.python.org/issue28869> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Alejandro Gonzalez added the comment: >It can be fixed in general if deduce __module__ in type.__call__() instead of >type.__new__(). But wouldn't we have the same problem if a metaclass overrides type.__call__ instead? Also, wouldn't that reset the __module__ value anytime the class is called and an object is instantiated? -- ___ Python tracker <https://bugs.python.org/issue28869> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Alejandro Gonzalez added the comment: >Getting the module name from the caller's frame is an expensive operation. It >is safe to set __module__ to None. In such case the pickle module is able to >find the proper module containing the definition of the class. Wow, indeed it works. I also tried it from another module other than `__main__` and it works. Checking the source, I see the "magic" happens in pickle's `whichmodule` function when looping over `sys.modules` if `__module__` is not found. If that case, why check for `__module__` in the first place? wouldn't it be simpler to always loop over sys.modules? Is it to avoid looping over `sys.modules` whenever possible? -- ___ Python tracker <https://bugs.python.org/issue28869> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Change by Alejandro Gonzalez : -- keywords: +patch pull_requests: +13973 stage: -> patch review pull_request: https://github.com/python/cpython/pull/14126 ___ Python tracker <https://bugs.python.org/issue28869> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Alejandro Gonzalez added the comment: >I think we can proceed with option A, but only if doesn't cause visible >slow-down for creating ABCs (which is already slower that normal classes). >TBH, I don't want to document A as "official" recipe (maybe however mention >the problem in the `pickle` module docs). I'll try using the same approach used in namedtuple, and see if there is any visible slow-down. Regarding the documentation, I see your point. Perhaps adding some section like "Notes on pickling dynamically-defined classes" in the `pickle` module would be more appropriate? That section would mention that you have to make sure to set the `__module__` attribute. >Note that other "class factories" in stdlib (like collections.namedtuple) do >almost exactly option A. Thanks for the tip. Indeed it does. In fact, the API defines a module argument. Its documentation could be improved, though. It should mention why would you want to pass that argument (pickling seems to be the reason that argument was added in the first place). -- ___ Python tracker <https://bugs.python.org/issue28869> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28869] __module__ attribute is not set correctly for a class created by direct metaclass call
Alejandro Gonzalez added the comment:
Hello, I would like to add further on this issue.
(I'm new to bpo. Please advise if this report could be better)
Issue
-
Pickling breaks when attempting to dump an instance of a class that was defined
by calling ABCMeta directly, because it attempts to look for the class in `abc`
rather than the module the class was defined in.
Below is a short snippet to reproduce the issue (my environment is Python 3.7.3
on Ubuntu 16.04)
>>> import pickle
>>> from abc import ABCMeta
>>> class Foo(metaclass=ABCMeta):
>>> pass
>>> Bar = ABCMeta('Bar', (), {})
>>> foo = Foo()
>>> bar = Bar()
>>> foo_p = pickle.dumps(foo) # OK
>>> bar_p = pickle.dumps(bar) # PicklingError: Can't pickle :
>>> attribute lookup Bar on abc failed
I encountered this issue when working on a class factory to define classes
dynamically. You can work around it if you smuggle `{'__module__': __name__}`
when calling the metaclass, e.g.:
>>> Qux = ABCMeta('Qux', (), {'__module__': __name__})
>>> qux = Qux()
>>> qux_p = pickle.dumps(qux) # OK
Apparently others have also stumbled upon this ABCMeta behavior before:
https://stackoverflow.com/q/49457441
Some ideas to solve this
A. Steven's proposal seems that could work (I haven't tested it myself yet
though).
- This, however, would be limited to ABCMeta. Any other metaclass
subclassed from type would have to include that workaround too for it to play
well with pickle, which leads to,
B. Do A. and include in the documentation some recipe that shows this
workaround when subclassing from type.
C. Implement A. in type itself. (Maybe around here?
https://github.com/python/cpython/blob/master/Objects/typeobject.c#L2603)
- I'm not familiar with the internals of CPython, but I guess this would be
the nuclear option.
If any of above ideas (or some other idea that could come up upon discussion)
seems sensible, please allow me to work on it. I'd be happy to contribute :)
--
nosy: +alegonz
___
Python tracker
<https://bugs.python.org/issue28869>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
