[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266430#comment-16266430
]
Christoph Pirkl commented on SPARK-10969:
-
Thanks for the information, this solves my problem. You can close the ticket.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148366#comment-15148366
]
Christoph Pirkl commented on SPARK-10969:
-
I created PR https://github.com/apache/spark/pull/11215 that implements this.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036418#comment-15036418
]
Christoph Pirkl commented on SPARK-10969:
-
This issue is about adding additional (optional) credentials for dynamodb and
cloudwatch. In my opinion it would be better to introduce a parameter object
containing all credentials than to have four additional String parameters.
I don't think it is possible to use the same mechanism as in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L98
(i.e. reading the credentials from System.getenv()) as this would mean
defining environment variables on each node.
A generic configuration using a map would mean a potential error source when
you mistype a parameter name.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035873#comment-15035873
]
Christoph Pirkl commented on SPARK-10969:
-
While this commit is useful it does not fix this issue. In order to fix this,
it would be best to introduce a serializable parameter object that contains aws
credentials for kinesis, dynamodb and cloudwatch (and maybe other values). This
would make it easier to add more parameters later.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
Christoph Pirkl created SPARK-10969:
---
Summary: Spark Streaming Kinesis: Allow specifying separate
credentials for Kinesis and DynamoDB
Key: SPARK-10969
URL: https://issues.apache.org/jira/browse/SPARK-10969
Project: Spark
Issue Type: Improvement
Components: Streaming
Affects Versions: 1.5.1
Reporter: Christoph Pirkl
Priority: Critical
{{KinesisUtils.createStream()}} allows specifying only one set of AWS
credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB and
CloudWatch.
h5. Motivation
In a scenario where one needs to read from a Kinesis Stream owned by a
different AWS account the user usually has minimal rights (i.e. only read from
the stream). In this case creating the DynamoDB table in KCL will fail.
h5. Proposal
My proposed solution would be to allow specifying multiple credentials in
{{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
additional credentials could then be passed to the constructor of
{{KinesisClientLibConfiguration}} or method
{{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
