[
https://issues.apache.org/jira/browse/SPARK-27950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-27950:
----------------------------------
Affects Version/s: (was: 2.4.3)
3.0.0
> Additional DynamoDB and CloudWatch config
> -----------------------------------------
>
> Key: SPARK-27950
> URL: https://issues.apache.org/jira/browse/SPARK-27950
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Eric S Meisel
> Priority: Minor
>
> I was researching getting Spark’s Kinesis integration running locally against
> {{localstack}}. We found this issue, and it creates a complication:
> [localstack/localstack#677|https://github.com/localstack/localstack/issues/677]
> Effectively, we need to be able to redirect calls for Kinesis, DynamoDB and
> Cloudwatch in order for the KCL to properly use the {{localstack}}
> infrastructure. We have successfully done this with the KCL (both 1.x and
> 2.x), but with Spark’s integration we are unable to configure DynamoDB and
> Cloudwatch’s endpoints:
> [https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
