[
https://issues.apache.org/jira/browse/BEAM-8326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik closed BEAM-8326.
---------------------------
Fix Version/s: Not applicable
Resolution: Duplicate
> Run Portable_Python PreCommit fails with pickling
> -------------------------------------------------
>
> Key: BEAM-8326
> URL: https://issues.apache.org/jira/browse/BEAM-8326
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Ahmet Altay
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Fix For: Not applicable
>
>
> Test log:
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/434/console
> I see the following error:
> 09:31:34 Traceback (most recent call last):
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py",
> line 261, in loads
> 09:31:34 return dill.loads(s)
> 09:31:34 File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line
> 317, in loads
> 09:31:34 return load(file, ignore)
> 09:31:34 File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line
> 305, in load
> 09:31:34 obj = pik.load()
> 09:31:34 TypeError: _create_function() takes from 2 to 6 positional arguments
> but 7 were given
> 09:31:34
> 09:31:34 During handling of the above exception, another exception occurred:
> 09:31:34
> 09:31:34 Traceback (most recent call last):
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 158, in _execute
> 09:31:34 response = task()
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 191, in <lambda>
> 09:31:34 self._execute(lambda: worker.do_instruction(work), work)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 343, in do_instruction
> 09:31:34 request.instruction_id)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 363, in process_bundle
> 09:31:34 instruction_id, request.process_bundle_descriptor_id)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 306, in get
> 09:31:34 self.data_channel_factory)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 580, in __init__
> 09:31:34 self.ops =
> self.create_execution_tree(self.process_bundle_descriptor)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 624, in create_execution_tree
> 09:31:34 descriptor.transforms, key=topological_height, reverse=True)])
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 623, in <listcomp>
> 09:31:34 for transform_id in sorted(
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 548, in wrapper
> 09:31:34 result = cache[args] = func(*args)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 607, in get_operation
> 09:31:34 in descriptor.transforms[transform_id].outputs.items()
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 606, in <dictcomp>
> 09:31:34 for tag, pcoll_id
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 605, in <listcomp>
> 09:31:34 tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 548, in wrapper
> 09:31:34 result = cache[args] = func(*args)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 610, in get_operation
> 09:31:34 transform_id, transform_consumers)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 869, in create_operation
> 09:31:34 return creator(self, transform_id, transform_proto, payload,
> consumers)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1112, in create
> 09:31:34 serialized_fn, parameter)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1150, in _create_pardo_operation
> 09:31:34 dofn_data = pickler.loads(serialized_fn)
> 09:31:34 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py",
> line 265, in loads
> 09:31:34 return dill.loads(s)
> 09:31:34 File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line
> 317, in loads
> 09:31:34 return load(file, ignore)
> 09:31:34 File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line
> 305, in load
> 09:31:34 obj = pik.load()
> 09:31:34 TypeError: _create_function() takes from 2 to 6 positional arguments
> but 7 were given
> First exception `IndexError` is expected and handled here
> (https://github.com/apache/beam/blob/ee45cb03f6e6f3685ae0dccf75a8af4d259b3b1a/sdks/python/apache_beam/runners/worker/sdk_worker.py#L301)
> The issue here is with pickling, might be a dill issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
