yuqi1129 commented on code in PR #9282:
URL: https://github.com/apache/gravitino/pull/9282#discussion_r2584011512
##########
bundles/aws/src/main/java/org/apache/gravitino/s3/fs/S3FileSystemProvider.java:
##########
@@ -129,6 +131,33 @@ private void checkAndSetCredentialProvider(Map<String,
String> configs) {
}
}
+ private Map<String, String> additionalS3Config(Map<String, String> configs) {
+ Map<String, String> additionalConfigs = Maps.newHashMap(configs);
+
+ // Avoid multiple retries to speed up failure in test cases.
+ // Use hard code instead of Constants.MAX_ERROR_RETRIES to avoid
dependency on a specific Hadoop
+ // version
+ if (!configs.containsKey("fs.s3a.attempts.maximum")) {
+ additionalConfigs.put("fs.s3a.attempts.maximum", "2");
+ }
+
+ if (!configs.containsKey("fs.s3a.connection.establish.timeout")) {
+ additionalConfigs.put("fs.s3a.connection.establish.timeout", "5000");
+ }
+
+ if (!configs.containsKey("fs.s3a.retry.limit")) {
+ additionalConfigs.put("fs.s3a.retry.limit", "2");
+ }
+
+ if (!configs.containsKey("fs.s3a.retry.throttle.limit")) {
+ additionalConfigs.put("fs.s3a.retry.throttle.limit", "2");
+ }
+
+ // More tuning can be added here.
+
+ return ImmutableMap.copyOf(additionalConfigs);
+ }
Review Comment:
I have tested it locally with a real s3 account. To verify it in test, we
need to trigger the test in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
