Maximilian Michels created BEAM-7128:
----------------------------------------
Summary: When unset parallelism is unavailable when applying
ReplacementTransforms
Key: BEAM-7128
URL: https://issues.apache.org/jira/browse/BEAM-7128
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Fix For: 2.13.0
In streaming mode, the Flink Runner applies a ReplacementTransform to set a
sharding strategy for Writes. This requires the parallelism of the pipeline to
be available up front. The replacement transforms are applied before the final
parallelism has been determined which leads to an error if the parallelism has
not been manually set:
{noformat}
Exception in thread "main" java.lang.IllegalArgumentException: Parallelism of a
job should be greater than 0. Currently set: {} [-1]
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:161)
at
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator$StreamingShardedWriteFactory.getReplacementTransform(FlinkStreamingPipelineTranslator.java:197)
at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:554)
at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:290)
at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:208)
at
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:93)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:108)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at com.maximilianmichels.WordCount.main(WordCount.java:64)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
