[
https://issues.apache.org/jira/browse/HADOOP-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867640#comment-15867640
]
Steve Loughran commented on HADOOP-12806:
-----------------------------------------
Given Hadoop 2.8 lets you declare whatever provider you want, it should be
possible to wire things up, even if we aren't doing it out the box in a way
which works with AWS Lambda.
Nikolaos: could you build the latest 2.8 branch and see if it works now?
> Hadoop fs s3a lib not working with temporary credentials in AWS Lambda
> -----------------------------------------------------------------------
>
> Key: HADOOP-12806
> URL: https://issues.apache.org/jira/browse/HADOOP-12806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 2.7.2
> Reporter: Nikolaos Tsipas
> Labels: aws-s3
>
> Trying to use the hadoop fs s3a library in AWS lambda with temporary
> credentials but it's not possible because of the way the
> {{AWSCredentialsProviderChain}} is defined under
> https://github.com/apache/hadoop/blob/29ae25801380b94442253c4202dee782dc4713f5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> Specifically the following code is used to initialise the credentials chain
> {code}
> AWSCredentialsProviderChain credentials = new AWSCredentialsProviderChain(
> new BasicAWSCredentialsProvider(accessKey, secretKey),
> new InstanceProfileCredentialsProvider(),
> new AnonymousAWSCredentialsProvider()
> );
> {code}
> The above works fine when the EC2 metadata endpoint is available (i.e.
> running on an EC2 instance) however it doesn't work properly when the
> environment variables are used to define credentials as it happens in AWS
> Lambda. Amazon suggests to use the {{EnvironmentVariableCredentialsProvider}}
> in AWS Lambda.
> To summarise and suggest an alternative I think that the
> {{DefaultAWSCredentialsProviderChain}} could be used instead of the
> {{InstanceProfileCredentialsProvider}} and that would cover the following
> cases:
> {panel}
> * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
> (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for
> .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)
> * Java System Properties - aws.accessKeyId and aws.secretKey
> * Credential profiles file at the default location (~/.aws/credentials)
> shared by all AWS SDKs and the AWS CLI
> * Instance profile credentials delivered through the Amazon EC2 metadata
> service
> {panel}
> If you think that the above change would be useful I could investigate more
> about what the required changes would be and submit a patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
