[
https://issues.apache.org/jira/browse/HADOOP-9384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597027#comment-13597027
]
D. Granit commented on HADOOP-9384:
-----------------------------------
Supplied a patched version of the {{NativeFileSystemStore}} called
{{AmazonS3ClientNativeFileSystemStore}}. Integrated it into the existing test
suite in {{NativeS3FileSystemContractBaseTest}} and
{{FileSystemContractBaseTest}} via
{{AmazonS3ClientNativeFileSystemContractTest}}.
Passes all local tests, with the credentials set in
{{src/test/resources/core-site}}:
{code:xml}
<property>
<name>test.fs.s3n.name</name>
<value>s3n://accessId:secret@bucketName</value>
<description>The name of the s3n file system for testing.</description>
</property>
{code}
Tested this as a patched distribution of hadoop-commons-3.0.0 on an EC2
instance having role with an acces policy for S3. The test was run as part of a
Single agent Flume setup, where flume was configured to write to an HDFS sink,
which in turn uses the HDFS to S3 implementation to log to S3. This was done
both with credentials set explicitly and no credentials set at all, such that
they are retrieved from the instance meta data service.
The patch only affects the native implementation of the S3 fs.
> Update S3 native fs implementation to use AWS SDK to support authorization
> through roles
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-9384
> URL: https://issues.apache.org/jira/browse/HADOOP-9384
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Environment: Locally: RHEL 6, AWS S3
> Remotely: AWS EC2 (RHEL 6), AWS S3
> Reporter: D. Granit
> Fix For: 3.0.0
>
> Attachments: HADOOP-9384.patch
>
>
> Currently the S3 native implementation
> {{org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore}} requires
> credentials to be set explicitly. Amazon allows setting credentials for
> instances instead of users, via roles. Such are rotated frequently and kept
> in a local cache all of which is handled by the AWS SDK in this case the
> {{AmazonS3Client}}. The SDK follows a specific order to establish whether
> credentials are set explicitly or via a role:
> - Environment Variables: AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
> - Java System Properties: aws.accessKeyId and aws.secretKey
> - Instance Metadata Service, which provides the credentials associated with
> the IAM role for the EC2 instance
> as seen in
> http://docs.aws.amazon.com/IAM/latest/UserGuide/role-usecase-ec2app.html
> To support this feature the current {{NativeFileSystemStore}} implementation
> needs to be altered to use the AWS SDK instead of the JetS3t S3 libraries.
> A request for this feature has previously been raised as part of the Flume
> project (FLUME-1691) where the HDFS on top of S3 implementation is used as a
> manner of logging into S3 via an HDFS Sink.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
