[
https://issues.apache.org/jira/browse/BEAM-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentyn Tymofieiev updated BEAM-8418:
--------------------------------------
Description:
Following pipeline fails on Dataflow runner unless we use beam_fn_api
experiment.
{noformat}
class NoOpDoFn(beam.DoFn):
def process(self, element):
return element
p = beam.Pipeline(options=pipeline_options)
_ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
result = p.run()
{noformat}
The reason is that we encode Impluse payload using url-escaping in [1], while
Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF
runner expects URL escaping.
We should fix or reconcile the encoding in non-FnAPI path, and add a
ValidatesRunner test that catches this error.
[1]
https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633
was:
Following pipeline fails on Dataflow runner unless we use beam_fn_api
experiment.
{noformat}
class NoOpDoFn(beam.DoFn):
def process(self, element):
return element
p = beam.Pipeline(options=pipeline_options)
_ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
result = p.run()
{noformat}
The reason is that we encode Impluse payload using url-escaping in [1], while
Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF
runner expects URL escaping.
We should fix or otherwise the encoding in non-FnAPI path, and add a
ValidatesRunner test that catches this error.
[1]
https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633
> Fix handling of Impulse transform in Dataflow runner.
> ------------------------------------------------------
>
> Key: BEAM-8418
> URL: https://issues.apache.org/jira/browse/BEAM-8418
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Robert Bradshaw
> Priority: Major
>
> Following pipeline fails on Dataflow runner unless we use beam_fn_api
> experiment.
> {noformat}
> class NoOpDoFn(beam.DoFn):
> def process(self, element):
> return element
> p = beam.Pipeline(options=pipeline_options)
> _ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
> result = p.run()
> {noformat}
> The reason is that we encode Impluse payload using url-escaping in [1], while
> Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF
> runner expects URL escaping.
> We should fix or reconcile the encoding in non-FnAPI path, and add a
> ValidatesRunner test that catches this error.
> [1]
> https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
