sekikn commented on a change in pull request #24651:
[SPARK-27420][DSTREAMS][Kinesis] KinesisInputDStream should expose a way to
configure CloudWatch metrics
URL: https://github.com/apache/spark/pull/24651#discussion_r317070472
##########
File path:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
##########
@@ -262,12 +271,45 @@ object KinesisInputDStream {
* endpoint. Will use the same credentials used for AWS Kinesis if no
custom value is set.
*
* @param credentials [[SparkAWSCredentials]] to use for CloudWatch
authentication
+ * @return Reference to this [[KinesisInputDStream.Builder]]
*/
def cloudWatchCredentials(credentials: SparkAWSCredentials): Builder = {
cloudWatchCredsProvider = Option(credentials)
this
}
+ // scalastyle:off line.size.limit
+ /**
+ * Sets the CloudWatch metrics level. Defaults to
+ * [[KinesisClientLibConfiguration.DEFAULT_METRICS_LEVEL]] if no custom
value is specified.
+ *
+ * @param metricsLevel [[MetricsLevel]] to specify the CloudWatch metrics
level
+ * @return Reference to this [[KinesisInputDStream.Builder]]
+ * @see
[[https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#metric-levels]]
+ */
+ // scalastyle:on line.size.limit
+ def metricsLevel(metricsLevel: MetricsLevel): Builder = {
+ this.metricsLevel = Option(metricsLevel)
+ this
+ }
+
+ // scalastyle:off line.size.limit
+ /**
+ * Sets the enabled CloudWatch metrics dimensions. Defaults to
+ * [[KinesisClientLibConfiguration.DEFAULT_METRICS_ENABLED_DIMENSIONS]]
+ * if no custom value is specified.
+ *
+ * @param metricsEnabledDimensions [[Set[String]]] to specify
+ * the enabled CloudWatch metrics
dimensions
+ * @return Reference to this [[KinesisInputDStream.Builder]]
+ * @see
[[https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#metric-levels]]
+ */
+ // scalastyle:off line.size.limit
Review comment:
Thank you for reviewing @sarutak, just updated the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
