Danny Cranmer created FLINK-26890:
-------------------------------------
Summary: [Kinesis][Consumer] DynamoDB consumer error consuming
partitions close to retention
Key: FLINK-26890
URL: https://issues.apache.org/jira/browse/FLINK-26890
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.15.0
Reporter: Danny Cranmer
Fix For: 1.15.0, 1.16.0, 1.13.7, 1.14.5
*Background*
The Amazon Kinesis Data Streams consumer supports consuming from Amazon
DynamoDB via the [DynamoDB Streams Kinesis
Adapter|https://github.com/awslabs/dynamodb-streams-kinesis-adapter].
*Problem*
We have seen instances of consumer throwing {{ResouceNotFoundException}} when
attempting to invoke {{GetShardIterator}}.
{code}
com.amazonaws.services.kinesis.model.ResourceNotFoundException: Requested
resource not found: Shard does not exist
{code}
According to the DynamoDB team, the {{DescribeStream}} call may return shard
IDs that are no longer valid, and this exception needs to be handled by the
client.
*Solution*
Modify the DynamoDB consumer to treat {{ResourceNotFoundException}} as a shard
closed signal.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
