[
https://issues.apache.org/jira/browse/SPARK-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204334#comment-14204334
]
Chris Fregly commented on SPARK-3640:
-------------------------------------
quick update:
Aniket and I spoke off-line about using AWS IAM Instance Profiles for EC2
instances. These work similar to IAM User Profiles - you can apply
fine-grained IAM Policies to EC2 instances.
The DefaultCredentialsProvider handles all of these sources of AWS credentials.
I am adding all of this to the Kinesis Spark Streaming Guide, btw.
Summary: we may be able to close this jira without a change. just waiting for
Aniket to confirm that this AWS Instance Profile approach satisfies his need.
it seems to be a safer approach than passing credentials between Spark Driver
and Worker nodes.
> KinesisUtils should accept a credentials object instead of forcing
> DefaultCredentialsProvider
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-3640
> URL: https://issues.apache.org/jira/browse/SPARK-3640
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.1.0
> Reporter: Aniket Bhatnagar
> Labels: kinesis
>
> KinesisUtils should accept AWS Credentials as a parameter and should default
> to DefaultCredentialsProvider if no credentials are provided. Currently, the
> implementation forces usage of DefaultCredentialsProvider which can be a pain
> especially when jobs are run by multiple unix users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
