Pawel Urbanowicz created BEAM-10020:
---------------------------------------
Summary: Java transform executed as Python transform?
Key: BEAM-10020
URL: https://issues.apache.org/jira/browse/BEAM-10020
Project: Beam
Issue Type: Bug
Components: jobserver, runner-flink
Affects Versions: 2.21.0
Reporter: Pawel Urbanowicz
Hey, I created a transform method in Java and now I want to use it in Python
using Cross-language. I got pretty stuck with the following problem:
*p*
*| GenerateSequence(...)*
*|ExternalTransform(...) => is working like a charm*
*p*
*| Create(...)*
*| ExternalTransform(...) => getting assert pardo_payload.do_fn.urn ==
python_urns.PICKLED_DOFN_INFO*
Based on [https://www.mail-archive.com/[email protected]/msg04887.html]
it seems like a Create transform is being registered as a Java transform but
executed as Python transform.
Traceback (most recent call last): File
"/Users/urban/projects/beam/sdks/python/apache_beam/io/external/snowflake_test.py",
line 92, in test_snowflake_write_read self.run_write() File
"/Users/urban/projects/beam/sdks/python/apache_beam/io/external/snowflake_test.py",
line 129, in run_write expansion_service=self.expansion_service, File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 528, in
__exit__ self.run().wait_until_finish() File
"/Users/urban/projects/beam/sdks/python/apache_beam/testing/test_pipeline.py",
line 112, in run False if self.not_use_test_runner_api else test_runner_api))
File "/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line
501, in run self._options).run(False) File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 858, in
from_runner_api p.transforms_stack =
[context.transforms.get_by_id(root_transform_id)] File
"/Users/urban/projects/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id self._id_to_proto[id], self._pipeline_context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 1231, in
from_runner_api part = context.transforms.get_by_id(transform_id) File
"/Users/urban/projects/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id self._id_to_proto[id], self._pipeline_context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 1231, in
from_runner_api part = context.transforms.get_by_id(transform_id) File
"/Users/urban/projects/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id self._id_to_proto[id], self._pipeline_context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 1231, in
from_runner_api part = context.transforms.get_by_id(transform_id) File
"/Users/urban/projects/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id self._id_to_proto[id], self._pipeline_context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 1231, in
from_runner_api part = context.transforms.get_by_id(transform_id) File
"/Users/urban/projects/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id self._id_to_proto[id], self._pipeline_context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/pipeline.py", line 1170, in
from_runner_api transform = ptransform.PTransform.from_runner_api(proto,
context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/transforms/ptransform.py",
line 685, in from_runner_api context) File
"/Users/urban/projects/beam/sdks/python/apache_beam/transforms/core.py", line
1380, in from_runner_api_parameter assert pardo_payload.do_fn.urn ==
python_urns.PICKLED_DOFN_INFO AssertionError
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
