mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way
to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377517531
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration
originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
verifyBucketExists() and verifyBucketExistsV2() are the methods which are
getting called from the doBucketProbing() method and I see they are already
using invoked which has a retry policy set to TRY_ONCE_THEN_FAIL. Do we need to
put explicit retry in this method?
Also both these methods are annotated with RetryTranslated and the
documentation of Retried say that if RetryTranslated is used, the called
shouldn't perform another layer of retries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
