Gerard Maas created SPARK-4537:
----------------------------------
Summary: Add 'processing delay' and 'totalDelay' to the metrics
reported by the Spark Streaming subsystem
Key: SPARK-4537
URL: https://issues.apache.org/jira/browse/SPARK-4537
Project: Spark
Issue Type: Improvement
Components: Streaming
Affects Versions: 1.1.0
Reporter: Gerard Maas
As the Spark Streaming tuning guide indicates, the key indicators of a healthy
streaming job are:
- Processing Time
- Total Delay
The Spark UI page for the Streaming job [1] shows these two indicators but the
metrics source for Spark Streaming (StreamingSource.scala) [2] does not.
Adding these metrics will allow external monitoring systems that consume the
Spark metrics interface to track these two critical pieces of information on a
streaming job performance.
[1]
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L127
[2]
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
