mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init() URL: https://github.com/apache/hadoop/pull/1838#discussion_r377593684
########## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md ########## @@ -608,3 +608,27 @@ with HADOOP-15669. Other options may be added to `fs.s3a.ssl.channel.mode` in the future as further SSL optimizations are made. + +## Tuning S3AFileSystem Initialization. +Any client using S3AFileSystem has to initialize it by providing a S3 bucket +and configuration. The init method checks if the bucket provided is valid +or not which is a slow operation leading poor performance. We can ignore +bucket validation by configuring `fs.s3a.bucket.probe` as follows: + +```xml +<property> + <name>fs.s3a.bucket.probe</name> + <value>0</value> + <description> + The value can be 0, 1 or 2(default). When set to 0, bucket existence + check won't be done during initialization thus making it faster. + Though it should be noted that if bucket is not available in S3, Review comment: Doc changes already present similar to you said. Do you want me to tweak this here? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
