[
https://issues.apache.org/jira/browse/FLINK-34342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-34342:
----------------------------------
Description:
*Problem*
We call
[DescribeStream|https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_DescribeStream.html]
with the ExclusiveStartShardId parameter set to the last seen shard ID. The
issue is that the API is eventually consistent, meaning if we call the API
multiple times we might get different results, for example:
* Call 1: [A, C]
* Call 2: [A, B, C]
Since we would set ExclusiveStartShardId to {{{}C{}}}, in the above example the
connector would miss shard {{B}}
*Solution*
We need to find a solution to support this gap. This could be to periodically
list all shards to find gaps and not start processing new shards until their
parents are complete. This feature does not need to be applied to the KDS
source.
was:
*Problem*
We call
[DescribeStream|https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_DescribeStream.html]
with the ExclusiveStartShardId parameter set to the last seen shard ID.
> Address Shard Consistency Issue for DDB Streams Source
> ------------------------------------------------------
>
> Key: FLINK-34342
> URL: https://issues.apache.org/jira/browse/FLINK-34342
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / DynamoDB
> Reporter: Danny Cranmer
> Priority: Major
>
> *Problem*
> We call
> [DescribeStream|https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_DescribeStream.html]
> with the ExclusiveStartShardId parameter set to the last seen shard ID. The
> issue is that the API is eventually consistent, meaning if we call the API
> multiple times we might get different results, for example:
> * Call 1: [A, C]
> * Call 2: [A, B, C]
> Since we would set ExclusiveStartShardId to {{{}C{}}}, in the above example
> the connector would miss shard {{B}}
> *Solution*
> We need to find a solution to support this gap. This could be to periodically
> list all shards to find gaps and not start processing new shards until their
> parents are complete. This feature does not need to be applied to the KDS
> source.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
