[Python-ideas] pickle.reduce and deconstruct
Pickling uses an extensible protocol that lets any class determine how its
instances can be deconstructed and reconstructed. Both `pickle` and `copy` use
this protocol, but it could be useful more generally. Unfortunately, to use it
more generally requires relying on undocumented details. I think we should
expose a couple of helpers to fix that:
# Return the same (shallow) reduction tuple that pickle.py, copy.py, and
_pickle.c would use pickle.reduce(obj) -> (callable, args[, state[, litems[,
ditem[, statefunc)
# Return a callable and arguments to construct a (shallow) equivalent
object # Raise a TypeError when that isn't possible
pickle.deconstruct(obj) -> callable, args, kw
So, why do you want these?
There are many cases where you want to "deconstruct" an object if possible.
Pattern matching depends on being able to deconstruct objects like this.
Auto-generating a `__repr__` as suggested in Chris's thread. Quick REPL
stuff, and deeper reflection stuff using `inspect.Signature` and friends.
Of course not every type tells `pickle` what to do in an appropriate way that
we can use, but a pretty broad range of types do, including (I think; I haven't
double-checked all of them) `@dataclass`, `namedtuple`, `@attr.s`, many builtin
and extension types, almost all reasonable types that use `copyreg`, and any
class that pickles via the simplest customization hook `__getnewargs[_ex]__`.
That's more than enough to be useful. And, just as important, it won't (except
in intentionally pathological cases) give us a false positive, where a type is
correctly pickleable and we think we can deconstruct it but the deconstruction
is wrong. (For some uses, you are going to want to fall back to heuristics that
are often right but sometimes misleadingly wrong, but I don't think the
`pickle` module should offer anything like that. Maybe `inspect` should.)
The way to get the necessary information isn't fully documented, and neither is
the way to interpret it. And I don't think it _should_ be documented, because
it changes every so often, and for good reasons; we don't want anyone writing
third-party code that relies on those details. Plus, a different Python
implementation might conceivably do it differently. Public helpers exposed from
`pickle` itself won't have those problems.
Here's a first take at the code.
def reduce(obj, proto=pickle.DEFAULT_PROTOCOL): """reduce(obj) ->
(callable, args[, state[, litems[, ditem[, statefunc) Return the
same reduction tuple that the pickle and copy modules use """ cls
= type(obj) if reductor := copyreg.dispatch_table.get(cls): return
reductor(obj) # Note that this is not a special method call (not looked up on
the type) if reductor := getattr(obj, "__reduce_ex__"): return
reductor(proto) if reductor := getattr(obj, "__reduce__"): return
reductor() raise TypeError(f"{cls.__name__} objects are not reducible")
def deconstruct(obj): """deconstruct(obj) -> callable, args, kw
callable(*args, **kw) will construct an equivalent object """
reduction = reduce(obj) # If any of the optional members are included,
pickle/copy has to # modify the object after construction, so there is no
useful single # call we can deconstruct to. if any(reduction[2:]):
raise TypeError(f"{type(obj).__name__} objects are not deconstrutable")
func, args, *_ = reduction # Most types (including @dataclass,
namedtuple, and many builtins) # use copyreg.__newobj__ as the constructor
func. The args tuple is # the type (or, when appropriate, some other registered
# constructor) followed by the actual args. However, any function # with the
same name will be treated the same way (because under the # covers, this is
optimized to a special opcode). if func.__name__ == "__newobj__":
return args[0], args[1:], {} # Mainly only used by types that implement
__getnewargs_ex__ use # copyreg.__newobj_ex__ as the constructor func. The args
tuple # holds the type, *args tuple, and **kwargs dict. Again, this is #
special-cased by name. if func.__name__ == "__newobj_ex__": return
args # If any other special copyreg functions are added in the future, # this
code won't know how to handle them, so bail. if func.__module__ ==
'copyreg': raise TypeError(f"{type(obj).__name__} objects are not
deconstrutable") # Otherwise, the type implements a custom __reduce__ or
__reduce_ex__, # and whatever it specifies as the constructor is the real
constructor. return func, args, {}
Actually looking at that code, I think it makes a better argument for why we
don't want to make all the internal details public. :)
Here are some quick (completely untested) examples of other things we could
build on it.
# in inspect def deconstruct(obj): """deconstruct(obj) ->
callable, bound_args Calling the
[Python-ideas] Re: addition of "nameof" operator
Thanks everyone for your responses. I would like to take all the information you gave me and do some research on it. I'll try to get it done this weekend or beginning of next week. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BGYKWTLGPQR34DTOVF52HPF5NIZ5OCQN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Suggestion for language addition
On Tuesday, December 3, 2019 at 7:54:03 PM UTC-5, Jan Bakuwel wrote: > > Hi Guido, > > On 4/12/19 1:06 pm, Guido van Rossum wrote: > > I was playing the devil's advocate in jest. There is no way this will > > be added to Python, for a large variety of sociological and software > > engineering reasons. While some of the responses may to you seem like > > they come from inexperienced people who have not carefully read your > > argument, consider that to us, your proposal appears to come from yet > > another person who is relatively new to Python asking for their > > favorite feature from language X without understanding how that > > feature would interact with the rest of Python. > > I was not implying at all that any response was from inexperienced > people. However no-one, at least not in a way clear to me, responded to > the value of the suggestion to get early warnings (ie. at parse time) > when an "end while" is written where an "end if" should have been written. > I can see how you would find it useful, and I can also see why you're getting a lot of opposition. One compromise suggestion is to take Andrew's advice and add your own # end for's. Then take Serhiy's advice and add the check to a linter. After you've been coding like this for a while, start your movement to popularize "# end for" :) > > While I like Python a lot, I do miss the support of advanced compilers > that tell me at compile time where I made a typo or logic error so I > find myself spending relatively more time debugging at runtime. Not a > worry as Python easily makes up for the lost time for 1000+ reasons. I'm > just thinking aloud how it could, in my eyes, be even better while also > considering the sociological and software engineering impact on the > Python community to the best of my abilities. > > I appreciate that I have much less experience with Python than many of > the folks on the list here. So please help me understand how the > suggested feature would, in any way, unfavorably interact with the rest > of Python as you say. > > cheers, > > Jan > > ___ > Python-ideas mailing list -- python...@python.org > To unsubscribe send an email to python-id...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/WH4TOP57AZSMRQG26YRWHDMDBEXV5WQV/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4LMK2RQMFIG6FJPWNWZPW3KM53CC5RWV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: __repr__ helper(s) in reprlib
I really like this feature suggestion and I would definitely use it if it
were available.
It might be easiest to have the repr helper do as much as it can, but give
the user the flexibility to override things with varying levels of
precision. For example:
def kwargs(obj,
keys: Optional[Collection[str]] = None,
keys_values: Optional[Mapping[str, Any]] = None,
*,
qualified=True):
if keys is None and keys_values is None:
pass # Fallback?
keys_strings = (
[]
if keys is None
else [(k, repr(getattr(obj, k))) for k in keys])
keys_strings += (
[]
if keys_values is None
else [(k, repr(v)) for k, v in keys_values.items()])
class_name = (obj.__class__.__qualname__
if qualified
else obj.__class__.__name__)
parameters = ", ".join(
f"{k}={v}" for k, v in keys_strings)
return f"{class_name}({parameters})"
On Tuesday, January 21, 2020 at 10:33:43 PM UTC-5, Andrew Barnert via
Python-ideas wrote:
>
> On Jan 21, 2020, at 14:30, Chris Angelico >
> wrote:
> >
> > On Wed, Jan 22, 2020 at 9:17 AM Andrew Barnert > wrote:
> >>
> >>> On Jan 21, 2020, at 12:29, Chris Angelico > wrote:
> >>>
> >>> For non-dataclass classes, it would be extremely helpful to have an
> >>> easy helper function available:
> >>>
> >>> class Spam:
> >>> def __repr__(self):
> >>> return reprlib.kwargs(self, ["quality", "recipe", "ham"])
> >>>
> >>> The implementation for this function would be very similar to what
> >>> dataclasses already do:
> >>>
> >>> # in reprlib.py
> >>> def kwargs(obj, attrs):
> >>> attrs = [f"{a}={getattr(obj, a)!r}" for a in attrs]
> >>> return f"{obj.__class__.__qualname__}({", ".join(attrs)})"
> >>>
> >>> A similar function for positional args would be equally easy.
> >>
> >> I like this, but I think it’s still more complex than it needs to be
> for 80% of the cases (see below)
> >
> > IMO that's not a problem. The implementation of reprlib.kwargs() is
> > allowed to be complex, since it's buried away as, well, implementation
> > details. As long as it's easy to call, that's all that matters.
>
> Sorry, I didn’t mean the implementation, I meant the interface. You
> shouldn’t have to specify the names in simple cases, and there are no
> correct names to specify in more complicated cases, and the range in which
> names are useful but also necessary seems pretty narrow.
>
> Even this example can’t work with your kwargs function:
>
> class Spam
> def __init__(self, x):
> self._x = x
> def __repr__(self):
> return kwargs(self, '_x'.split()) # or 'x'.split()
>
> You want the repr to be `Spam(x=3)`, but there’s no way to get that. If
> you use _x you get `Spam(_x=3)`, which is wrong; if you use x you get an
> AttributeError, which nobody wants from repr.
>
> >> while for the other 20%, I think it might make it too easy to get
> things wrong.
> >
> > Hmm. I kept it completely explicit - it will generate a repr that
> > shows the exact attributes listed (and personally, I'd often write it
> > as "quality recipe ham".split()),
>
> Right, but that’s exactly what makes it easy to get wrong. If I add a new
> param and attribute with a default value and don’t remember to also add it
> to the repr, I’m now silently generating reprs that look right but aren’t.
> If I have an attribute that’s computed and include it in repr by accident,
> likewise.
>
> >>> Bikeshedding opportunity: Should it be legal to omit the attrs
> >>> parameter, and have it use __slots__ or fall back to dir(obj) ?
> >>
> >> This makes things even more dangerous. It’s very common for classes to
> have attributes that aren’t part of the constructor call, or constructor
> params that aren’t attributes, and this would give you the wrong answer.
> >>
> >> It would be nice if there were a safe way to get the
> constructor-call-style repr. And I think there might be for 80% of the
> types—and the rest can specify it manually and take the risk of getting it
> wrong, probably.
> >>
> >> One option is the pickle/copy protocol. If the type uses one of the
> newargs methods, you can use that to get the constructor arguments; if it
> uses one of the other pickling methods (or can’t be pickled), this just
> doesn’t work.
> >>
> >> You could also look at the inspect.signature of __init__ and/or
> __new__. If every param has an attribute with the same name, use that;
> otherwise, this doesn’t work.
> >>
> >> And if none of the automatic ways worked and you tried to use them
> anyway, you get an error.
> >
> > This is definitely getting into the realm of magic.
>
> I don’t think getnewargs is any more magical than dir is—and it’s correct,
> and ties into a protocol already used for two other things in Python.
>
> > The question is,
> > is it worth the convenience? I'd be +0.25
