[
https://issues.apache.org/jira/browse/SPARK-40506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
王俊博 updated SPARK-40506:
Description:
Spark StreamingSource Metrics sourceName is inappropriate.The label now looks
like
`application_x__driver_NetworkWordCount_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime
`, instead of
`application_x__driver_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime`,
the Spark app name is not need.
This makes it hard to use metrics for different Spark applications over time.
And this makes the metrics sourceName standard inconsistent.
{code:java}
//代码占位符
private[streaming] class StreamingSource(ssc: StreamingContext) extends Source {
override val metricRegistry = new MetricRegistry
override val sourceName =
"%s.StreamingMetrics".format(ssc.sparkContext.appName)
}{code}
And for example, other metrics sourceName don't have appName.
{code:java}
//代码占位符
private[spark] class LiveListenerBusMetrics(conf: SparkConf)
extends Source with Logging {
override val sourceName: String = "LiveListenerBus"
override val metricRegistry: MetricRegistry = new MetricRegistry
...
}
{code}
was:
Spark StreamingSource Metrics sourceName is inappropriate.The label now looks
like
`application_x__driver_NetworkWordCount_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime
`, instead of
`application_x__driver_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime`,
the Spark app name is not need.
This makes it hard to use metrics for different Spark applications over time.
And this makes the metrics sourceName standard inconsistent
{code:java}
//代码占位符
private[streaming] class StreamingSource(ssc: StreamingContext) extends Source {
override val metricRegistry = new MetricRegistry
override val sourceName =
"%s.StreamingMetrics".format(ssc.sparkContext.appName)
}{code}
And for example, other metrics sourceName don't have appName.
{code:java}
//代码占位符
private[spark] class LiveListenerBusMetrics(conf: SparkConf)
extends Source with Logging {
override val sourceName: String = "LiveListenerBus"
override val metricRegistry: MetricRegistry = new MetricRegistry
...
}
{code}
> Spark Streaming metrics name don't need application name
>
>
> Key: SPARK-40506
> URL: https://issues.apache.org/jira/browse/SPARK-40506
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
>Affects Versions: 3.2.2
>Reporter: 王俊博
>Priority: Trivial
>
> Spark StreamingSource Metrics sourceName is inappropriate.The label now
> looks like
> `application_x__driver_NetworkWordCount_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime
> `, instead of
> `application_x__driver_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime`,
> the Spark app name is not need.
> This makes it hard to use metrics for different Spark applications over time.
> And this makes the metrics sourceName standard inconsistent.
> {code:java}
> //代码占位符
> private[streaming] class StreamingSource(ssc: StreamingContext) extends
> Source {
> override val metricRegistry = new MetricRegistry
> override val sourceName =
> "%s.StreamingMetrics".format(ssc.sparkContext.appName)
>
> }{code}
> And for example, other metrics sourceName don't have appName.
> {code:java}
> //代码占位符
> private[spark] class LiveListenerBusMetrics(conf: SparkConf)
> extends Source with Logging {
> override val sourceName: String = "LiveListenerBus"
> override val metricRegistry: MetricRegistry = new MetricRegistry
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
