[
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=301584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-301584
]
ASF GitHub Bot logged work on BEAM-5878:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Aug/19 23:11
Start Date: 26/Aug/19 23:11
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9237: [BEAM-5878]
support DoFns with Keyword-only arguments
URL: https://github.com/apache/beam/pull/9237#discussion_r317834907
##########
File path: sdks/python/apache_beam/internal/pickler.py
##########
@@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj):
dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators
+# TODO: Remove this once uqfoundation/dill#313 is fixed
+if sys.version_info[0] > 2:
+ # Monkey patch for dill._dill.Pickler to pickle functions
+ # with keyword-only args
+ _create_function = dill.dill._create_function
+
+ def _create_function_has_kwdefaults(fcode, fglobals, fname=None,
+ fdefaults=None, fclosure=None,
fdict=None,
+ fkwdefaults=None):
+ func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict)
+ func.__kwdefaults__ = fkwdefaults
+ return func
+
+ def new_save_reduce(self, func, args, state=None, listitems=None,
Review comment:
Can we use a more generic signature of new_save_reduce, for example `def
new_save_reduce(self, func, args, **kwargs)`? The problem is that we assume a
particular version of the API for pickle.save_reduce here, and we can see that
it will change in Python 3.8, see
https://github.com/python/cpython/blob/c75f0e5bdee3cfaba9fd5b3a8549dec0aba01ebe/Lib/pickle.py#L619.
I think with a generic definition of `new_save_reduce` we can still update
`args` list , and pass `**kwargs` to `picker.save_reduce`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 301584)
Time Spent: 9h 10m (was: 9h)
> Support DoFns with Keyword-only arguments in Python 3.
> ------------------------------------------------------
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: yoshiki obata
> Priority: Minor
> Fix For: 2.16.0
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to
> define functions with keyword-only arguments.
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
> in our codebase that we should fix: in Python in 3.0 inspect.getargspec()
> will fail on functions with keyword-only arguments, but a new method
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
> supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only
> arguments once Beam Python 3 tests are in a good shape.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
