[
https://issues.apache.org/jira/browse/FLINK-37004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh updated FLINK-37004:
------------------------------------
Fix Version/s: aws-connector-5.1.0
> Implement Multithreaded SourceReader
> ------------------------------------
>
> Key: FLINK-37004
> URL: https://issues.apache.org/jira/browse/FLINK-37004
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
> Reporter: Hong Liang Teoh
> Priority: Major
> Fix For: aws-connector-5.1.0
>
>
> Currently the *KinesisStreamsSourceReader* extends
> {*}SingleThreadMultiplexSourceReaderBase{*}, which has a single thread per
> subtask.
>
> Since multiple shards can be assigned to the same subtask, this can limit the
> maximum throughput of the source, unless users increase the parallelism of
> the source.
>
> However, this solution doesn't work when Kinesis streams have on-demand mode.
> Since the stream can shard up / down dynamically.
>
> In this case, it might make sense to have a multithread source reader, and
> some form of setting to limit the maximum number of shards handled by each
> thread.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
