Elphas Toringepi created FLINK-36637:
----------------------------------------
Summary: Fix ListShards for Kinesis streams with large number of
shards
Key: FLINK-36637
URL: https://issues.apache.org/jira/browse/FLINK-36637
Project: Flink
Issue Type: Bug
Components: Connectors / AWS
Affects Versions: 2.0.0
Reporter: Elphas Toringepi
Fix For: 2.0.0
The new Kinesis source fails to consume from streams with a large number of
shards because it cannot successfully list the shards. This issue was
discovered on a stream with 1024 shards.
The following exception is thrown:
{{org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'Source: Kinesis Stream Source' (operator
bc764cd8ddf7a0cff126f51c16239658).}}
{{ at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:651)}}
{{ ...}}
{{Caused by:
org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException:
Failed to list shards.}}
{{ at
org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.processDiscoveredSplits(KinesisStreamsSourceEnumerator.java:375)}}
{{ at
org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$1(ExecutorNotifier.java:83)}}
{{ at
org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)}}
{{ ... 6 more}}
{{Caused by:
org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException:
Unable to list shards}}
{{ at
org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.initialDiscoverSplits(KinesisStreamsSourceEnumerator.java:260)}}
{{ at
org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)}}
{{ ... 6 more}}
{{Caused by:
org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.InvalidArgumentException:
NextToken and StreamARN cannot be provided together. (Service: Kinesis, Status
Code: 400, Request ID: d0ca3ae9-975e-6f46-8f56-70aa948cfa30, Extended Request
ID:
+4y3p3zgJ9MOZ7FOL4C1LnhrVm69QLZXbwvCyZ7f3bfArn66oKcYz5J0xAGEKMA7JBGa6wzI7aOTwdHbPofuhfOY+LpjTSUk)}}
{{ ...}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
