[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567625#comment-16567625
]
Steve Loughran commented on HADOOP-14237:
-----------------------------------------
I'm looking at this again, going to move to 3.3 along with most of the other
outstainding s3 for 3.2 features.
* I don't like saving the full secrets (unencrypted) to HDFS
* session secrets could work, though of course they'll expire within 24h.
once HADOOP-15883 is in I'm going to revisit HADOOP-14556, which lets the s3a
client to serialize its secrets as a filesystem delegation token, something
apps (hive, spark, MR) know to ask for -and which YARN knows how to securely
marshall to launched apps. With this feature you could launch things into a
pool of VMs with reduced privilege IAM roles, sending in higher privilege
credentials with the request. Would that work?
I've also created HADOOP-15650 to cover the issue of better retry logic on
credential retrieval. I see there's an async option, which might be more
responsive, but could put even more load on the service unless managed
carefully. What it could do though, is handle retries much better (though it'd
also be a more more complicated....)
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> -----------------------------------------------------------------------
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
> Reporter: Kazuyuki Tanimura
> Assignee: Kazuyuki Tanimura
> Priority: Major
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
