[
https://issues.apache.org/jira/browse/FLINK-34342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh reassigned FLINK-34342:
---------------------------------------
Assignee: Abhishek Kumar
> Address Shard Consistency Issue for DDB Streams Source
> ------------------------------------------------------
>
> Key: FLINK-34342
> URL: https://issues.apache.org/jira/browse/FLINK-34342
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / DynamoDB
> Reporter: Danny Cranmer
> Assignee: Abhishek Kumar
> Priority: Major
>
> *Problem*
> We call
> [DescribeStream|https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_DescribeStream.html]
> with the ExclusiveStartShardId parameter set to the last seen shard ID. The
> issue is that the API is eventually consistent, meaning if we call the API
> multiple times we might get different results, for example:
> * Call 1: [A, C]
> * Call 2: [A, B, C]
> Since we would set ExclusiveStartShardId to {{{}C{}}}, in the above example
> the connector would miss shard {{B}}
> *Solution*
> We need to find a solution to support this gap. This could be to periodically
> list all shards to find gaps and not start processing new shards until their
> parents are complete. This feature does not need to be applied to the KDS
> source.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
