Hong Liang Teoh created FLINK-35299:
---------------------------------------
Summary: FlinkKinesisConsumer does not respect
StreamInitialPosition for new Kinesis Stream when restoring from snapshot
Key: FLINK-35299
URL: https://issues.apache.org/jira/browse/FLINK-35299
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: aws-connector-4.2.0
Reporter: Hong Liang Teoh
Fix For: aws-connector-4.4.0
h3. What
The FlinkKinesisConsumer allows users to read from [multiple Kinesis
Streams|https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L224].
Users can also specify a STREAM_INITIAL_POSITION, which configures if the
consumer starts reading the stream from TRIM_HORIZON / LATEST / AT_TIMESTAMP.
When restoring the Kinesis Consumer from an existing snapshot, users can
configure the consumer to read from additional Kinesis Streams. The expected
behavior would be for the FlinkKinesisConsumer to start reading from the
additional Kinesis Streams respecting the STREAM_INITIAL_POSITION
configuration. However, we find that it currently reads from TRIM_HORIZON.
This is surprising behavior and should be corrected.
h3. Why
Principle of Least Astonishment
h3. How
We recommend that we reconstruct the previously seen streams by iterating
through the [sequenceNumsStateForCheckpoint in
FlinkKinesisConsumer#initializeState()|https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L454].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
