[
https://issues.apache.org/jira/browse/FLINK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh updated FLINK-37273:
------------------------------------
Fix Version/s: aws-connector-5.2.0
(was: aws-connector-5.1.0)
(was: 2.1.0)
> Dynamodb Streams getRecords call adaptive delay
> -----------------------------------------------
>
> Key: FLINK-37273
> URL: https://issues.apache.org/jira/browse/FLINK-37273
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / DynamoDB
> Affects Versions: 2.1.0
> Reporter: Pranav Wadekar
> Priority: Major
> Fix For: aws-connector-5.2.0
>
>
> I am trying to read the data from DynamoDBStreams which is introduced in
> Flink v18.0. It is working fine for large tables (10K RPM), and the
> efficiency is also good.
> However, the dynamodb stream cost for {{getRecords}} AWS requests. Although
> it's not a problem for big tables, for small tables even if the records are
> not committed Flink is continuously polling the records.
> With the above behaviour, we are getting an unnecessary cost of 130$ per day.
> Ideally, we need to scan the stream at specific intervals with some backoff
> time.
> I did not find anything regarding this.
> Can we please implement the adaptive delay if there is any idleness in
> incoming records? I can also contribute to the code.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
