dongjoon-hyun commented on a change in pull request #26060: [SPARK-29400][CORE]
Improve PrometheusResource to use labels
URL: https://github.com/apache/spark/pull/26060#discussion_r333270955
##########
File path:
core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
##########
@@ -40,30 +40,35 @@ private[v1] class PrometheusResource extends
ApiRequestContext {
def executors(): String = {
val sb = new StringBuilder
val store = uiRoot.asInstanceOf[SparkUI].store
- val appId = store.applicationInfo.id.replaceAll("[^a-zA-Z0-9]", "_")
store.executorList(true).foreach { executor =>
- val prefix = s"metrics_${appId}_${executor.id}_executor_"
- sb.append(s"${prefix}rddBlocks_Count ${executor.rddBlocks}\n")
- sb.append(s"${prefix}memoryUsed_Count ${executor.memoryUsed}\n")
- sb.append(s"${prefix}diskUsed_Count ${executor.diskUsed}\n")
- sb.append(s"${prefix}totalCores_Count ${executor.totalCores}\n")
- sb.append(s"${prefix}maxTasks_Count ${executor.maxTasks}\n")
- sb.append(s"${prefix}activeTasks_Count ${executor.activeTasks}\n")
- sb.append(s"${prefix}failedTasks_Count ${executor.failedTasks}\n")
- sb.append(s"${prefix}completedTasks_Count ${executor.completedTasks}\n")
- sb.append(s"${prefix}totalTasks_Count ${executor.totalTasks}\n")
- sb.append(s"${prefix}totalDuration_Value ${executor.totalDuration}\n")
- sb.append(s"${prefix}totalGCTime_Value ${executor.totalGCTime}\n")
- sb.append(s"${prefix}totalInputBytes_Count
${executor.totalInputBytes}\n")
- sb.append(s"${prefix}totalShuffleRead_Count
${executor.totalShuffleRead}\n")
- sb.append(s"${prefix}totalShuffleWrite_Count
${executor.totalShuffleWrite}\n")
- sb.append(s"${prefix}maxMemory_Count ${executor.maxMemory}\n")
+ val prefix = "metrics_executor_"
+ val labels = Seq(
+ "application_id" -> store.applicationInfo.id,
+ "application_name" -> store.applicationInfo.name,
+ "executor_id" -> executor.id
+ ).map { case (k, v) => s"""$k="$v"""" }.mkString("{", ", ", "}")
+ sb.append(s"${prefix}rddBlocks_Count$labels ${executor.rddBlocks}\n")
Review comment:
Right. The redundant information is moved to labels.
> Actually prefix is fixed? So they are now the same metrics. And
application id, executor id now are labels on them?
No. Prometheus query language support to handle them individually.
> Does it have bad impact on the metrics usage later? Because now all
applications are recorded under the same metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
