[
https://issues.apache.org/jira/browse/HADOOP-9384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597156#comment-13597156
]
Steve Loughran commented on HADOOP-9384:
----------------------------------------
I can see the value in this, however, I don't want to add a new JAR dependency
to the core, as it has downstream effects.
We've been discussing having a bit of the source tree set up explictly for
filesystem (especially blobstore) clients -a new S3 client library is a change
that would justify moving it into this (not yet created) location.
# The S3:// filesystem needs to be change in sync, because it doesn't make
sense to have different behaviour.
# At that point, maybe jets3t could be dropped entirely.
# HADOOP-9360 proposes some more changes to the FS contract, these can be used
to extend the regression tests.
# HADOOP-9258 adds coverage tests for s3; again, these could be used for
regression testing
# The HADOOP-9258 tests look at what happens when credentials aren't supplied.
Those need to be extended with tests that set the system properties and not the
conf.xml ones, to verify that the desired behaviour -sysprops- works. Verifying
env variable support is trickier. There is a [ugly test-only hack for
this|http://stackoverflow.com/questions/318239/how-do-i-set-environment-variables-from-java]
-not something I'm sure anyone would encourage the use of
> Update S3 native fs implementation to use AWS SDK to support authorization
> through roles
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-9384
> URL: https://issues.apache.org/jira/browse/HADOOP-9384
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Environment: Locally: RHEL 6, AWS S3
> Remotely: AWS EC2 (RHEL 6), AWS S3
> Reporter: D. Granit
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-9384.patch, HADOOP-9384-v2.patch
>
>
> Currently the S3 native implementation
> {{org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore}} requires
> credentials to be set explicitly. Amazon allows setting credentials for
> instances instead of users, via roles. Such are rotated frequently and kept
> in a local cache all of which is handled by the AWS SDK in this case the
> {{AmazonS3Client}}. The SDK follows a specific order to establish whether
> credentials are set explicitly or via a role:
> - Environment Variables: AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
> - Java System Properties: aws.accessKeyId and aws.secretKey
> - Instance Metadata Service, which provides the credentials associated with
> the IAM role for the EC2 instance
> as seen in
> http://docs.aws.amazon.com/IAM/latest/UserGuide/role-usecase-ec2app.html
> To support this feature the current {{NativeFileSystemStore}} implementation
> needs to be altered to use the AWS SDK instead of the JetS3t S3 libraries.
> A request for this feature has previously been raised as part of the Flume
> project (FLUME-1691) where the HDFS on top of S3 implementation is used as a
> manner of logging into S3 via an HDFS Sink.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
