Hong Liang Teoh created FLINK-37004:
---------------------------------------
Summary: Implement Multithreaded SourceReader
Key: FLINK-37004
URL: https://issues.apache.org/jira/browse/FLINK-37004
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Reporter: Hong Liang Teoh
Currently the *KinesisStreamsSourceReader* extends
{*}SingleThreadMultiplexSourceReaderBase{*}, which has a single thread per
subtask.
Since multiple shards can be assigned to the same subtask, this can limit the
maximum throughput of the source, unless users increase the parallelism of the
source.
However, this solution doesn't work when Kinesis streams have on-demand mode.
Since the stream can shard up / down dynamically.
In this case, it might make sense to have a multithread source reader, and some
form of setting to limit the maximum number of shards handled by each thread.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
