[
https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897173#comment-15897173
]
Eugene Kirpichov commented on BEAM-68:
--------------------------------------
This feels more runner-specific to me, because e.g. Dataflow runner is able to
dynamically choose parallelism and allocate more or fewer resources depending
on how much parallelism is needed for the currently running steps (see
https://cloud.google.com/blog/big-data/2016/03/comparing-cloud-dataflow-autoscaling-to-spark-and-hadoop).
I don't know what controls Flink in particular provides for this, though.
> Support for limiting parallelism of a step
> ------------------------------------------
>
> Key: BEAM-68
> URL: https://issues.apache.org/jira/browse/BEAM-68
> Project: Beam
> Issue Type: New Feature
> Components: beam-model
> Reporter: Daniel Halperin
>
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets
> TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected
> QPS/step) on the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model.
> A GroupByKey with exactly k keys will guarantee that only k elements are
> produced, but runners are free to break fusion in ways that each element may
> be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the
> Beam Model.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
