Abhi Gupta created FLINK-36270:
----------------------------------
Summary: DDB Streams Connector performance issue due to
splitsAvailableForAssignment function
Key: FLINK-36270
URL: https://issues.apache.org/jira/browse/FLINK-36270
Project: Flink
Issue Type: Bug
Components: Connectors / DynamoDB
Reporter: Abhi Gupta
In DDB Streams connector, while testing we found out that when we are spending
a lot of time in markAsFinished function because we are calling
splitsAvailableForAssignment which is O(n), and given n shards can be marked as
finished concurrently, the algorithm becomes O(n^2). Change the algo to assign
only child shards when a parent is finished. We can start tracking child shards
of a shard in SplitTracker
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
