Steve Loughran resolved HADOOP-12806.
       Resolution: Fixed
    Fix Version/s: 2.8.0

> Hadoop fs s3a lib not working with temporary credentials in AWS Lambda 
> -----------------------------------------------------------------------
>                 Key: HADOOP-12806
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12806
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 2.7.2
>            Reporter: Nikolaos Tsipas
>            Priority: Major
>              Labels: aws-s3
>             Fix For: 2.8.0
> Trying to use the hadoop fs s3a library in AWS lambda with temporary 
> credentials but it's not possible because of the way the 
> {{AWSCredentialsProviderChain}} is defined under 
> https://github.com/apache/hadoop/blob/29ae25801380b94442253c4202dee782dc4713f5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> Specifically the following code is used to initialise the credentials chain
> {code}
> AWSCredentialsProviderChain credentials = new AWSCredentialsProviderChain(
>         new BasicAWSCredentialsProvider(accessKey, secretKey),
>         new InstanceProfileCredentialsProvider(),
>         new AnonymousAWSCredentialsProvider()
>     );
> {code}
> The above works fine when the EC2 metadata endpoint is available (i.e. 
> running on an EC2 instance) however it doesn't work properly when the 
> environment variables  are used to define credentials as it happens in AWS 
> Lambda. Amazon suggests to use the {{EnvironmentVariableCredentialsProvider}} 
> in AWS Lambda. 
> To summarise and suggest an alternative I think that the 
> {{DefaultAWSCredentialsProviderChain}} could be used instead of  the 
> {{InstanceProfileCredentialsProvider}}  and that would cover the following 
> cases: 
> {panel}
> * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY 
> (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for 
> .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)
> * Java System Properties - aws.accessKeyId and aws.secretKey
> * Credential profiles file at the default location (~/.aws/credentials) 
> shared by all AWS SDKs and the AWS CLI
> * Instance profile credentials delivered through the Amazon EC2 metadata 
> service
> {panel}
> If you think that the above change would be useful I could investigate more 
> about what the required changes would be and submit a patch.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to