Samrat002 commented on code in PR #28984:
URL: https://github.com/apache/flink/pull/28984#discussion_r3801135184
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystemFactory.java:
##########
@@ -282,6 +282,21 @@ public class NativeS3FileSystemFactory implements
FileSystemFactory {
"Maximum delay cap for exponential backoff,
applied to both "
+ "normal and throttle retry paths.");
+ public static final ConfigOption<Boolean> RETRY_CIRCUIT_BREAKER_ENABLED =
+ ConfigOptions.key("s3.retry.circuit-breaker.enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "Whether the AWS SDK's retry circuit breaker
(token bucket) is enabled. "
+ + "The SDK stops retrying once a shared
per-client token bucket is "
+ + "drained by recent failures, regardless
of the retry/backoff settings "
+ + "above. Under a high volume of
concurrent requests hitting S3 "
+ + "throttling (e.g. large incremental
checkpoints), this bucket can drain "
+ + "within seconds, causing retries to be
abandoned well before the "
+ + "configured backoff and retry count are
exhausted. Disabled by default "
+ + "so the retry/backoff settings above
fully govern retry behavior, "
+ + "matching the S3A plugin's behavior.");
Review Comment:
[NIT] We should remove S3A plugin names from the doc.
Description write-up can be simple, stating to switch off the circuit
breaker, which the user cares.
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java:
##########
@@ -270,6 +273,11 @@ Duration getRetryMaxBackoff() {
return retryMaxBackoff;
}
+ @VisibleForTesting
+ boolean isRetryCircuitBreakerEnabled() {
+ return retryCircuitBreakerEnabled;
+ }
Review Comment:
Method only for test ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
