Pranav Wadekar created FLINK-37273:
--------------------------------------
Summary: Dynamodb Streams getRecords call adaptive delay
Key: FLINK-37273
URL: https://issues.apache.org/jira/browse/FLINK-37273
Project: Flink
Issue Type: Improvement
Components: Connectors / DynamoDB
Affects Versions: aws-connector-5.0.0
Reporter: Pranav Wadekar
Fix For: 2.0.0
I am trying to read the data from DynamoDBStreams which is introduced in Flink
v18.0. It is working fine for large tables (10K RPM), and the efficiency is
also good.
However, the dynamodb stream cost for {{getRecords}} AWS requests. Although
it's not a problem for big tables, for small tables even if the records are not
committed Flink is continuously polling the records.
With the above behaviour, we are getting an unnecessary cost of 130$ per day.
Ideally, we need to scan the stream at specific intervals with some backoff
time.
I did not find anything regarding this.
Can we please implement the adaptive delay if there is any idleness in incoming
records? I can also contribute to the code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
