etspaceman opened a new pull request #24801: URL: https://github.com/apache/spark/pull/24801
## What changes were proposed in this pull request? I was researching getting Spark’s Kinesis integration running locally against `localstack`. We found this issue, and it creates a complication: https://github.com/localstack/localstack/issues/677 Effectively, we need to be able to redirect calls for Kinesis, DynamoDB and Cloudwatch in order for the KCL to properly use the `localstack` infrastructure. We have successfully done this with the KCL (both 1.x and 2.x), but with Spark’s integration we are unable to configure DynamoDB and Cloudwatch’s endpoints: https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162 This PR adds optional configuration values to the interfaces for dynamoDBEndpointUrl and cloudWatchMetricsLevel. Why cloudWatchMetricsLevel instead of cloudWatchEndpointUrl? Because the 1.x version of the KCL does not expose a means of configuring the cloudWatchEndpointUrl. Localstack users can instead disable metrics entirely by setting the cloudWatchMetricsLevel to Some(MetricsLevel.NONE) ## How was this patch tested? Existing unit tests were expanded to check that these values were set. Please review https://spark.apache.org/contributing.html before opening a pull request. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
