[
https://issues.apache.org/jira/browse/FLUME-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597057#comment-13597057
]
D. Granit commented on FLUME-1691:
----------------------------------
A patch was committed to HADOOP-9384 that allows using the HDFS sink with the
latest Amazon SDK to support the use of the instance meta data service to
establish credentials provided via a role applied to an instance.
if you leave your {{hdfs-site.xml}} without any configuration and set your
flume config to
{code}
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = s3n://yourBucket
{code}
without credentials provided (It's important to use {{s3n://}} and not
{{s3://}} as the patch only patches the native s3 implementation and not the
block fs one) then it'll use the {{AmazonS3Client}} constructor following the
order you have described above and as such will pick up the credentials
provided by the meta data service.
> Allow use of EC2 roles with S3 sink
> -----------------------------------
>
> Key: FLUME-1691
> URL: https://issues.apache.org/jira/browse/FLUME-1691
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Reporter: Steve Stogner
> Priority: Minor
>
> If you assign an IAM role to an EC2 instance, then AWS exposes role
> credentials through the metadata interface. These credentials are temporary
> credentials that AWS rolls periodically. When making calls to AWS with
> temporary credentials, you have to use a token in addition to the access ID
> and secret key. Flume would impress if it would default to the EC2 role
> credentials when using an S3 sink with no credentials configuration required.
> Flume would either refresh the credentials from the metadata with every call
> to S3 or when it detects that the credentials have expired. Users could
> still override the use of role credentials with user credentials via the
> current configuration method (fs.s3.awsAccessKeyId, fs.s3.awsSecretAccessKey,
> fs.s3n.awsAccessKeyId, fs.s3n.awsSecretAccessKey).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
