[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650831#comment-16650831
]
Steve Loughran commented on HADOOP-14556:
-----------------------------------------
HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn
cluster to verify that MR job submission collects DTs for source and
destination paths.
To do this the MockJob class had to go into
hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java and
job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but
that only works if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't)
improved/extended in assumed_roles.md as well as delegation_tokens.md.
All AWS error messages related to STS/session and role requests are now in
assumed_roles.md to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward
any session creds it gets from its own auth chain, rather than ask for new ones
(which it can't do with session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the
AssumeRole and delegation tests which were clearing or overwriting the
fs.s3a.{auth, secret, session} options, as those in the creds file were still
being picked up. Fix: explicitly reset hadoop.security.credential.provider.path
for all the tests which were now failing.
* minor checkstyle fixup
tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of
{{ITestS3GuardToolLocal\#testDestroyNoBucket }} *even when I was running with
dynamodb*. I think that test suite is running when it shouldn't. More research
needed there
> S3A to support Delegation Tokens
> --------------------------------
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch,
> HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
