gguptp commented on code in PR #261:
URL:
https://github.com/apache/flink-connector-aws/pull/261#discussion_r4515666688
##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java:
##########
@@ -160,19 +168,43 @@ public SourceReader<T, DynamoDbStreamsShardSplit>
createReader(
final Duration getRecordsIdlePollingTimeBetweenEmptyPolls =
sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS);
+ // If true, the split reader calls GetRecords on one shard at a time
and waits for the
+ // response. Default - false
+ final boolean blockingGetRecords =
sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_BLOCKING);
+
+ // Maximum number of non-blocking calls in progress per source reader.
Default - 100
+ final int maxCallsInFlight =
sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_MAX_IN_FLIGHT);
+
Map<String, List<Shard>> childShardMap = new ConcurrentHashMap<>();
- // We create a new stream proxy for each split reader since they have
their own independent
- // lifecycle.
+ // All split readers of this source reader share one stream proxy. The
fetcher manager
+ // creates a new split reader each time it starts a fetcher (for
example, after all splits
+ // of the previous fetcher finished), and a proxy per split reader
would create a new AWS
+ // SDK client, HTTP client and, for non-blocking calls, Netty event
loop each time. The
+ // proxy is created with the first split reader, so that a source
reader without splits
+ // opens no clients. The source reader closes the proxy.
+ //
+ // The circuit breaker of the retry strategy is off. The clients of
one proxy share the
+ // retry token bucket of the strategy, and with many non-blocking
calls in progress, one
+ // burst of errors would spend the whole bucket and fail the calls
without retries.
+ RetryStrategy retryStrategy =
+ SdkDefaultRetryStrategy.legacyRetryStrategyBuilder()
Review Comment:
Agreed with your thought there. Let me update the PR for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
