mkman84 opened a new pull request, #36166: URL: https://github.com/apache/spark/pull/36166
…nsuming from kinesis streams JIRA: https://issues.apache.org/jira/browse/SPARK-38881 ### What changes were proposed in this pull request? Exposing a configuration option (metricsLevel) used for CloudWatch metrics reporting when consuming from an AWS Kinesis Stream, which is already available in Scala/Java Spark APIs This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was merged as part of Spark 3.0.0 ### Why are the changes needed? This change is desirable as it further exposes the metricsLevel config parameter that was added for the Scala/Java Spark APIs when working with the Kinesis Streaming integration, and makes it available to the PySpark API as well. ### Does this PR introduce _any_ user-facing change? No. Default behavior of MetricsLevel.DETAILED is maintained. ### How was this patch tested? This change passes all tests, and local testing was done with a development Kinesis stream in AWS, in order to confirm that metrics were no longer being reported to CloudWatch after specifying MetricsLevel.NONE in the PySpark Kinesis streaming context creation, and also worked as it does today when leaving the MetricsLevel parameter out, which would result in a default of DETAILED, with CloudWatch metrics appearing again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
