murrelljenna opened a new pull request, #45619: URL: https://github.com/apache/spark/pull/45619
This PR is a rebased and updated version of a [previous pr](https://github.com/apache/spark/pull/24801) by @etspaceman from 2019. This PR allows the spark kinesis library to work with localstack instances of kinesis. I have not added any changes to @etspaceman 's original code, except to remove the `cloudwatchMetricsLevel` value in the original PR, as subsequent PRs have added that same value in since. ### What changes were proposed in this pull request? Currently the spark kinesis library allows you to override the kinesis endpoint, but not the dynamodb endpoint the kinesis consumer will use to manage leases. This PR adds that configurable url into the kinesis input stream builder. ### Why are the changes needed? Localstack has only become more popular for integration testing since the original PR. Currently, one cannot use spark streaming and localstack kinesis together, as the kinesis consumer will only attempt to manage leases through a remote aws dynamo instance. ### Does this PR introduce _any_ user-facing change? This PR will allow the user to specify their own dynamo endpoint, much like how they can currently specify their own kinesis endpoint. ### How was this patch tested? This PR adds unit tests to cover this new value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
