steveloughran commented on issue #1619: HADOOP-16478. S3Guard bucket-info fails if the caller lacks s3:GetBucketLocation URL: https://github.com/apache/hadoop/pull/1619#issuecomment-541132228 Also just ran the CLI against a public bucket which blocks this operation ``` Filesystem s3a://tpcds10g 2019-10-11 17:24:14,361 [main] DEBUG s3a.Invoker (DurationInfo.java:<init>(74)) - Starting: getBucketLocation() 2019-10-11 17:24:14,472 [main] DEBUG s3a.Invoker (DurationInfo.java:close(89)) - getBucketLocation(): duration 0:00.110s 2019-10-11 17:24:14,473 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:run(1232)) - failed to get bucket location java.nio.file.AccessDeniedException: tpcds10g: getBucketLocation() on tpcds10g: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CE32462FD451F00D; S3 Extended Request ID: /pM+yWUtyByovVFTzOHPDDEQhzQAuF9zVrimxhbzaX6b8iYv6pgGO9cNbhL30eZ9wOTBcGpyvIY=), S3 Extended Request ID: /pM+yWUtyByovVFTzOHPDDEQhzQAuF9zVrimxhbzaX6b8iYv6pgGO9cNbhL30eZ9wOTBcGpyvIY=:AccessDenied at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:244) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:315) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:311) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:286) at org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:741) at org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:724) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:1227) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:429) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1816) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1825) Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CE32462FD451F00D; S3 Extended Request ID: /pM+yWUtyByovVFTzOHPDDEQhzQAuF9zVrimxhbzaX6b8iYv6pgGO9cNbhL30eZ9wOTBcGpyvIY=), S3 Extended Request ID: /pM+yWUtyByovVFTzOHPDDEQhzQAuF9zVrimxhbzaX6b8iYv6pgGO9cNbhL30eZ9wOTBcGpyvIY= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860) at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:999) at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:1005) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getBucketLocation$3(S3AFileSystem.java:742) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110) ... 11 more Location unknown -caller lacks s3:GetBucketLocation permission Filesystem s3a://tpcds10g is not using S3Guard The "magic" committer is supported S3A Client Signing Algorithm: fs.s3a.signing-algorithm=(unset) Endpoint: fs.s3a.endpoint=(unset) Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal Change Detection Source: fs.s3a.change.detection.source=etag Change Detection Mode: fs.s3a.change.detection.mode=server Delegation token support is disabled ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
