[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-10-16 Thread Hudson (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952627#comment-16952627
 ] 

Hudson commented on HADOOP-16478:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17538 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/17538/])
HADOOP-16478. S3Guard bucket-info fails if the caller lacks (stevel: rev 
bbcf0b91d6f5eb697d09e45505b0e72e193c3d75)
* (edit) 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
* (edit) 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
* (add) 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestNeworkBinding.java
* (edit) 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/NetworkBinding.java
* (edit) 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
* (edit) 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (edit) 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ContextAccessors.java
* (edit) 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java


> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-10-10 Thread Steve Loughran (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948416#comment-16948416
 ] 

Steve Loughran commented on HADOOP-16478:
-

While I go near this command, I'd like to also list the auth directories for a 
bucket, as it is now a bit trickier to work out what is going on.

This command is proving to be step one in understanding s3guard related issues 
-it needs to be complete.

> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-10-08 Thread Steve Loughran (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946990#comment-16946990
 ] 

Steve Loughran commented on HADOOP-16478:
-

metastore already does this, but reviewed the message and tuned both it and the 
exception.

> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-08-19 Thread Steve Loughran (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910621#comment-16910621
 ] 

Steve Loughran commented on HADOOP-16478:
-

+have the DDB metastore print this fact out on the error

> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-08-19 Thread Steve Loughran (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910620#comment-16910620
 ] 

Steve Loughran commented on HADOOP-16478:
-

also doc for s3guard that you need this for DDB if you dont set 
fs.s3a.s3guard.ddb.region

> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-08-01 Thread Steve Loughran (JIRA) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897925#comment-16897925
 ] 

Steve Loughran commented on HADOOP-16478:
-

Two options

* S3AFileSystem.getBucketLocation (and the StoreContext) return "" if there is 
no location, document that and have the clients deal with that. Good: no need 
for try/catch. Bad: still need error handling, and if the goal is raise an 
exception, the stack is lost (example: Dynamo)
* Make clear in the javadocs for the API this error must be handled
* And do that in the bucket info call
* +review other places

> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16478) S3Guard bucket-info fails if the bucket location is denied to the caller

2019-07-31 Thread Steve Loughran (JIRA) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897291#comment-16897291
 ] 

Steve Loughran commented on HADOOP-16478:
-

{code}
java.nio.file.AccessDeniedException: mow-dev-istio-west-demo: 
getBucketLocation() on s3a://restricted: 
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: 
Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 
030653A1119B53A7; S3 Extended Request ID: 
lmr6jNHSrfpvjcuyJP4D0wovmqnfFVrnHOQNQD9SXV6ZVTF7eF5IHddEXnUtp2STMvxc7PySzkw=), 
S3 Extended Request ID: 
lmr6jNHSrfpvjcuyJP4D0wovmqnfFVrnHOQNQD9SXV6ZVTF7eF5IHddEXnUtp2STMvxc7PySzkw=:AccessDenied
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:243)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:314)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:406)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:310)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:285)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:716)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:703)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:1185)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:401)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1672)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1681)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied 
(Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 
030653A1119B53A7; S3 Extended Request ID: 
lmr6jNHSrfpvjcuyJP4D0wovmqnfFVrnHOQNQD9SXV6ZVTF7eF5IHddEXnUtp2STMvxc7PySzkw=), 
S3 Extended Request ID: 
lmr6jNHSrfpvjcuyJP4D0wovmqnfFVrnHOQNQD9SXV6ZVTF7eF5IHddEXnUtp2STMvxc7PySzkw=
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)
at 
com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:999)
at 
com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:1005)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getBucketLocation$3(S3AFileSystem.java:717)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
... 11 more


> S3Guard bucket-info fails if the bucket location is denied to the caller
> 
>
> Key: HADOOP-16478
> URL: https://issues.apache.org/jira/browse/HADOOP-16478
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> IF you call "Hadoop s3guard bucket info" on a bucket and you don't have 
> permission to list the bucket location, then you get a stack trace, with all 
> other diagnostics being missing.
> Preferred: catch the exception, warn its unknown and only log@ debug



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org