Eugene Kirpichov created BEAM-1234:
--------------------------------------
Summary: Consider a hint ParDo.withHighFanout()
Key: BEAM-1234
URL: https://issues.apache.org/jira/browse/BEAM-1234
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci
Priority: Minor
I'm finding myself again and again suggesting users on StackOverflow to insert
fusion breaks after high-fanout ParDo's.
I think we should just implement this as a hint on ParDo and MapElements
transforms, like we have on GroupByKey.fewKeys() or Combine.withHotKeyFanout().
E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner
that implements fusion could decide to insert a runner-specific fusion break.
This somewhat sidesteps the issues in
https://issues.apache.org/jira/browse/BEAM-730 and
https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
because every runner can decide how to do the right thing, or is free to
ignore the hint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
