[ https://issues.apache.org/jira/browse/HADOOP-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369338#comment-16369338 ]
Steve Loughran commented on HADOOP-12806: ----------------------------------------- Given the auth chain is configurable, I'm, closing this as fixed against Hadoop 2.8 > Hadoop fs s3a lib not working with temporary credentials in AWS Lambda > ----------------------------------------------------------------------- > > Key: HADOOP-12806 > URL: https://issues.apache.org/jira/browse/HADOOP-12806 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 > Affects Versions: 2.7.2 > Reporter: Nikolaos Tsipas > Priority: Major > Labels: aws-s3 > Fix For: 2.8.0 > > > Trying to use the hadoop fs s3a library in AWS lambda with temporary > credentials but it's not possible because of the way the > {{AWSCredentialsProviderChain}} is defined under > https://github.com/apache/hadoop/blob/29ae25801380b94442253c4202dee782dc4713f5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java > Specifically the following code is used to initialise the credentials chain > {code} > AWSCredentialsProviderChain credentials = new AWSCredentialsProviderChain( > new BasicAWSCredentialsProvider(accessKey, secretKey), > new InstanceProfileCredentialsProvider(), > new AnonymousAWSCredentialsProvider() > ); > {code} > The above works fine when the EC2 metadata endpoint is available (i.e. > running on an EC2 instance) however it doesn't work properly when the > environment variables are used to define credentials as it happens in AWS > Lambda. Amazon suggests to use the {{EnvironmentVariableCredentialsProvider}} > in AWS Lambda. > To summarise and suggest an alternative I think that the > {{DefaultAWSCredentialsProviderChain}} could be used instead of the > {{InstanceProfileCredentialsProvider}} and that would cover the following > cases: > {panel} > * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY > (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for > .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK) > * Java System Properties - aws.accessKeyId and aws.secretKey > * Credential profiles file at the default location (~/.aws/credentials) > shared by all AWS SDKs and the AWS CLI > * Instance profile credentials delivered through the Amazon EC2 metadata > service > {panel} > If you think that the above change would be useful I could investigate more > about what the required changes would be and submit a patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org