steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way
to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377166880
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration
originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
+ int bucketProbe = this.getConf()
+ .getInt(S3A_BUCKET_PROBE, S3A_BUCKET_PROBE_DEFAULT);
+ Preconditions.checkArgument(bucketProbe >= 0 && bucketProbe <= 2,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2");
+ switch (bucketProbe) {
+ case 0:
+ break;
+ case 1:
+ verifyBucketExists();
+ break;
+ case 2:
+ verifyBucketExistsV2();
+ break;
+ default:
+ break;
Review comment:
Add a comment here saying this won't be reached because of the checks above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
