Alexey Romanenko created BEAM-4798:
--------------------------------------
Summary: IndexOutOfBoundsException when Flink parallelism > 1
Key: BEAM-4798
URL: https://issues.apache.org/jira/browse/BEAM-4798
Project: Beam
Issue Type: Bug
Components: runner-flink
Affects Versions: 2.5.0
Reporter: Alexey Romanenko
Assignee: Alexey Romanenko
Running job on Flink in streaming mode and get data from a Kafka topic with
parallelism > 1 causes an exception:
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:277)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at
org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It happens when number of Kafka topic partitions is less than value of
parallelism (number of task slots).
So, workaround for now can be to set parallelism <= number of topic partitions,
thus if parallelism=2 then number_partitions >= 2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
