[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700653#comment-16700653
]
Steve Loughran commented on HADOOP-14556:
-----------------------------------------
Hi [~elgoiri]: thanks for this review; not had a chance to reply until now.
bq. The unit tests cover the basic cases well.
I'd have liked to have a real mini-yarn cluster with distcp, but couldn't get
kerberos to work with miniyarn and minihdfs to the extent the cluster would
come up. If/when someone can do that. I'd revisit it.
bq. Very long patch and even though there are a bunch of interfaces which are
pretty verbose, there is a lot here. I'm not sure if there are ways to split
it. For example the utilities to fetch the DT.
I know, and I always worry about adding more complexity for the following
reason: other people have to maintain it, and if they can't either the code is
neglected or I'm expected to be the maintainer indefinitely.
I've tried to keep all DT support out in its own home, with not that much in
the S3A FS -but as I changed the encryption stuff there may be too much of a
diff there. I could perhaps revert some of that. Less elegant but a smaller
diff for that file, and so less risk of merge conflict.
And because I was going near session credential management, I also tried to
coalesce stuff that the credential providers were doing. Again, I could look to
pull that for now
Otherwise: I've needed to do all 3 including the role stuff, to make sure I
hadn't blocked out those. I even believe that I've done enough to support more
advanced bindings. We could strip out the full credentials as it doesn't reduce
risk, and so only support session and role secrets? that'd work well for
locking down AWS, but I would also like to support third party stores which
don't have sessions
regarding the docs, [~lmccay] has suggested I could actually do a video of this
at work. Would people be interested? That'd be a real demo of role-base-DT =>
live cluster for distcp.
> S3A to support Delegation Tokens
> --------------------------------
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
