[
https://issues.apache.org/jira/browse/BEAM-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Moravek updated BEAM-8608:
--------------------------------
Description:
Right now, in Batch runner, DoFn is executed using MapPartition operator
(_FlinkDoFnFunction_), which doesn't have chained driver implementation.
We need to reimplement DoFnFunction with FlatMap to allow chaining.
Attached is the execution graph for the same pipeline, before and after the
patch.
was:
Right now, in Batch runner, DoFn is executed using MapPartition operator
(_FlinkDoFnFunction_), which doesn't have chained driver implementation.
We need to reimplement DoFnFunction with FlatMap to allow chaining.
> Chain DoFns in Flink batch runner when possible.
> ------------------------------------------------
>
> Key: BEAM-8608
> URL: https://issues.apache.org/jira/browse/BEAM-8608
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Affects Versions: 2.16.0
> Reporter: David Moravek
> Assignee: David Moravek
> Priority: Major
> Attachments: Screen Shot 2019-11-07 at 10.35.07.png, Screen Shot
> 2019-11-07 at 10.35.22.png
>
>
> Right now, in Batch runner, DoFn is executed using MapPartition operator
> (_FlinkDoFnFunction_), which doesn't have chained driver implementation.
> We need to reimplement DoFnFunction with FlatMap to allow chaining.
> Attached is the execution graph for the same pipeline, before and after the
> patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
