Tzu-Li (Gordon) Tai created FLINK-8542:
------------------------------------------
Summary: Do not indefinitely store closed shard's state in the
FlinkKinesisConsumer
Key: FLINK-8542
URL: https://issues.apache.org/jira/browse/FLINK-8542
Project: Flink
Issue Type: Improvement
Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
See original discussion here:
[https://github.com/apache/flink/pull/5337|https://github.com/apache/flink/pull/5337#issuecomment-362227711]
Currently, the Kinesis consumer keeps a list of {{(StreamShardMetadata,
SequenceNumber)}} as its state. That list also contains all shards that have
been closed already, and is kept in the state indefinitely so that on restore,
we know that a closed shard is already fully consumed,
The downside of this, is that the state size of the Kinesis consumer can
basically grow without bounds, as the consumed Kinesis streams are resharded
and more and more closed shards are present.
Some possible solutions have been discussed in the linked PR comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
