Re: [Python-Dev] Class decorators can't be pickled, which breaks multiprocessing and concurrent.futures. Any plans of improving this?
11.08.18 23:08, Santiago Basulto пише:
Hello folks! I'm using the `concurrent.futures.ProcessPoolExecutor` with
a couple of functions that have been decorated with a class decorator.
Both `concurrent.futures` and `multiprocessing` breaks because "the
object's can't be pickled". There's a really simple fix for this, which
is just, instead of "decorating" the function (with the @), instantiate
the decorator and use it directly.
Example. This is my (very simple, for demonstration purposes) decorator:
class CheckOnlyIntegers:
def __init__(self, fn):
self.fn = fn
def __call__(self, *args):
if not all([type(arg) == int for arg in args]):
raise ValueError("Invalid param is not an integer")
return self.fn(*args)
If I define a simple `add` function and decorate it using the
`CheckOnlyIntegers` decorator:
@CheckOnlyIntegers
def add(x, y):
return x + y
and try using a regular `ProcessPoolExecutor().submit(add, 2, 3)`, it
fails with:
```
Can't pickle : it's not the same object as
__main__.add.
```
By default instances of Python classes are pickled by pickling a set of
their attributes. Functions are pickled by name. But since the name of
self.fn corresponds to the decorated function, not the function itself,
it can't be pickled.
You can implement the explicit pickle support for your decorator that
bypass this limitation.
def __reduce__(self):
return self.fn.__qualname__
Now the decorated function will be pickled by name.
It may help to set also self.__module__ = self.fn.__module__ in the
constructor.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Class decorators can't be pickled, which breaks multiprocessing and concurrent.futures. Any plans of improving this?
Hello folks! I'm using the `concurrent.futures.ProcessPoolExecutor` with a
couple of functions that have been decorated with a class decorator. Both
`concurrent.futures` and `multiprocessing` breaks because "the object's
can't be pickled". There's a really simple fix for this, which is just,
instead of "decorating" the function (with the @), instantiate the
decorator and use it directly.
Example. This is my (very simple, for demonstration purposes) decorator:
class CheckOnlyIntegers:
def __init__(self, fn):
self.fn = fn
def __call__(self, *args):
if not all([type(arg) == int for arg in args]):
raise ValueError("Invalid param is not an integer")
return self.fn(*args)
If I define a simple `add` function and decorate it using the
`CheckOnlyIntegers` decorator:
@CheckOnlyIntegers
def add(x, y):
return x + y
and try using a regular `ProcessPoolExecutor().submit(add, 2, 3)`, it fails
with:
```
Can't pickle : it's not the same object as
__main__.add.
```
The fix for this is simple, instead of "decorating" the function,
instantiate the class and use a different name:
def add(x, y):
return x + y
add_2 = CheckOnlyIntegers(add)
In this case `ProcessPoolExecutor().submit(add_2, 2, 3)` works correctly.
(here's the full sample code
)
I know this is an issue with the pickle module (not concurrent.futures or
multiprocessing). But are there any ways of improving this in future
versions? Not being able to pickle functions decorated with Class
Decorators seems like an unnecessary limitation.
Thanks for your feedback!
--
Santiago Basulto.-
Up!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
