[
https://issues.apache.org/jira/browse/HADOOP-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582973#comment-17582973
]
ASF GitHub Bot commented on HADOOP-16806:
-----------------------------------------
jmahonin commented on PR #4753:
URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1222353654
Tried with and without an `fs.s3a.assumed.role.arn` setting. I'm not sure if
I'm expected to set up this dataset manually, or if there's some magic behind
the scenes that should make it available?
```
[ERROR]
testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)
Time elapsed: 10.84 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on
s3a://landsat-pds/scene_list.gz:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon
S3; Status Code: 400; Error Code: 400 Bad Request; Request ID:
16AXE4CXSVZBN8BH; S3 Extended Request ID:
3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=;
Proxy: null), S3 Extended Request ID:
3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=:400
Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code:
400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID:
3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID:
16AXE4CXSVZBN8BH; S3 Extended Request ID:
3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
[ERROR]
testJobSubmissionCollectsTokens[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)
Time elapsed: 7.758 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on
s3a://landsat-pds/scene_list.gz:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon
S3; Status Code: 400; Error Code: 400 Bad Request; Request ID:
W01GKMAXC95KHVPR; S3 Extended Request ID:
OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=;
Proxy: null), S3 Extended Request ID:
OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=:400
Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code:
400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID:
OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID:
W01GKMAXC95KHVPR; S3 Extended Request ID:
OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
```
> AWS AssumedRoleCredentialProvider needs ExternalId add
> ------------------------------------------------------
>
> Key: HADOOP-16806
> URL: https://issues.apache.org/jira/browse/HADOOP-16806
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.1
> Reporter: Jon Hartlaub
> Priority: Minor
> Labels: pull-request-available
>
> AWS has added a security feature to the assume-role function in the form of
> the "ExternalId" key in the AWS Java SDK
> {{STSAssumeRoleSessionCredentialsProvider.Builder}} class. To support this
> security feature, the hadoop aws {{AssumedRoleCredentialProvider}} needs a
> patch to include this value from the configuration as well as an added
> Constant to the {{org.apache.hadoop.fs.s3a.Constants}} file.
> The ExternalId is not a required security feature, it is an augmentation of
> the current assume role configuration.
> Proposed:
> * Get the assume-role ExternalId token from the configuration for the
> configuration key {{fs.s3a.assumed.role.externalid}}
> * Use the configured ExternalId value in the
> {{STSAssumeRoleSessionCredentialsProvider.Builder}}
> e.g.
> {{if (StringUtils.isNotEmpty(externalId)) {}}
> {{ builder.withExternalId(externalId); // include the token for
> cross-account assume role}}
> {{}}}
> Tests:
> * +Unit test+ which verifies the ExternalId state value of the
> {{AssumedRoleCredentialProvider}} is consistent with the configured value -
> either empty or populated
> * Question: not sure about how to write the +integration test+ for this
> feature. We have an account configured for this use-case that verifies this
> feature but I don't have much context on the Hadoop project AWS S3
> integration tests, perhaps a pointer could help.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
