Boyuan Zhang created BEAM-12335:
-----------------------------------
Summary: Apply basic fusion to Java DirectRunner to avoid keeping
all intermittent results in memory
Key: BEAM-12335
URL: https://issues.apache.org/jira/browse/BEAM-12335
Project: Beam
Issue Type: Improvement
Components: runner-direct
Reporter: Boyuan Zhang
Current java direct runner doesn't fuse transforms into steps. Instead, it
almost executes each transform one by one. It results in memory pressure when
any transform is high-fanout.
We already have a simple fusion logic in Java
SDK(https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java).
Work remaining here might be:
* Apply such fusion into DirectRunner
* Change the DirectRunner to be able run the fused steps.
I understand that DirectRunner doesn't expect processing large volume data and
changing DirectRunner execution might be a fair amount of work.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
