[
https://issues.apache.org/jira/browse/FLINK-36637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh updated FLINK-36637:
------------------------------------
Fix Version/s: aws-connector-4.4.0
(was: 2.0.0)
> Fix ListShards for Kinesis streams with large number of shards
> --------------------------------------------------------------
>
> Key: FLINK-36637
> URL: https://issues.apache.org/jira/browse/FLINK-36637
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
> Affects Versions: aws-connector-4.4.0
> Reporter: Elphas Toringepi
> Assignee: Elphas Toringepi
> Priority: Critical
> Fix For: aws-connector-4.4.0
>
>
> The new Kinesis source fails to consume from streams with a large number of
> shards because it cannot successfully list the shards. This issue was
> discovered on a stream with 1024 shards.
> The following exception is thrown:
> {{org.apache.flink.util.FlinkException: Global failure triggered by
> OperatorCoordinator for 'Source: Kinesis Stream Source' (operator
> bc764cd8ddf7a0cff126f51c16239658).}}
> {{ at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:651)}}
> {{ ...}}
> {{Caused by:
> org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException:
> Failed to list shards.}}
> {{ at
> org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.processDiscoveredSplits(KinesisStreamsSourceEnumerator.java:375)}}
> {{ at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$1(ExecutorNotifier.java:83)}}
> {{ at
> org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)}}
> {{ ... 6 more}}
> {{Caused by:
> org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException:
> Unable to list shards}}
> {{ at
> org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.initialDiscoverSplits(KinesisStreamsSourceEnumerator.java:260)}}
> {{ at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)}}
> {{ ... 6 more}}
> {{Caused by:
> org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.InvalidArgumentException:
> NextToken and StreamARN cannot be provided together. (Service: Kinesis,
> Status Code: 400, Request ID: d0ca3ae9-975e-6f46-8f56-70aa948cfa30, Extended
> Request ID:
> +4y3p3zgJ9MOZ7FOL4C1LnhrVm69QLZXbwvCyZ7f3bfArn66oKcYz5J0xAGEKMA7JBGa6wzI7aOTwdHbPofuhfOY+LpjTSUk)}}
> {{ ...}}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
