[
https://issues.apache.org/jira/browse/SPARK-27420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-27420.
-------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 24651
[https://github.com/apache/spark/pull/24651]
> KinesisInputDStream should expose a way to configure CloudWatch metrics
> -----------------------------------------------------------------------
>
> Key: SPARK-27420
> URL: https://issues.apache.org/jira/browse/SPARK-27420
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Input/Output
> Affects Versions: 2.3.3
> Reporter: Jerome Gagnon
> Assignee: Kengo Seki
> Priority: Major
> Fix For: 3.0.0
>
>
> KinesisInputDStream currently does not provide a way to disable CloudWatch
> metrics push. Kinesis client library (KCL) which is used under the hood
> provide the ability through `withMetrics` methods.
> To make things worse the default level is "DETAILED" which pushes 10s of
> metrics every 10 seconds. When dealing with multiple streaming jobs this add
> up pretty quickly, leading to thousands of dollar in cost.
> Exposing a way to disable/set the proper level of monitoring is critical to
> us. We had to send invalid credentials and suppress log as a less-than-ideal
> workaround : see
> [https://stackoverflow.com/questions/41811039/disable-cloudwatch-for-aws-kinesis-at-spark-streaming/55599002#55599002]
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
