viirya commented on a change in pull request #26060: [SPARK-29400][CORE]
Improve PrometheusResource to use labels
URL: https://github.com/apache/spark/pull/26060#discussion_r333281562
##########
File path:
core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
##########
@@ -40,30 +40,35 @@ private[v1] class PrometheusResource extends
ApiRequestContext {
def executors(): String = {
val sb = new StringBuilder
val store = uiRoot.asInstanceOf[SparkUI].store
- val appId = store.applicationInfo.id.replaceAll("[^a-zA-Z0-9]", "_")
store.executorList(true).foreach { executor =>
- val prefix = s"metrics_${appId}_${executor.id}_executor_"
- sb.append(s"${prefix}rddBlocks_Count ${executor.rddBlocks}\n")
- sb.append(s"${prefix}memoryUsed_Count ${executor.memoryUsed}\n")
- sb.append(s"${prefix}diskUsed_Count ${executor.diskUsed}\n")
- sb.append(s"${prefix}totalCores_Count ${executor.totalCores}\n")
- sb.append(s"${prefix}maxTasks_Count ${executor.maxTasks}\n")
- sb.append(s"${prefix}activeTasks_Count ${executor.activeTasks}\n")
- sb.append(s"${prefix}failedTasks_Count ${executor.failedTasks}\n")
- sb.append(s"${prefix}completedTasks_Count ${executor.completedTasks}\n")
- sb.append(s"${prefix}totalTasks_Count ${executor.totalTasks}\n")
- sb.append(s"${prefix}totalDuration_Value ${executor.totalDuration}\n")
- sb.append(s"${prefix}totalGCTime_Value ${executor.totalGCTime}\n")
- sb.append(s"${prefix}totalInputBytes_Count
${executor.totalInputBytes}\n")
- sb.append(s"${prefix}totalShuffleRead_Count
${executor.totalShuffleRead}\n")
- sb.append(s"${prefix}totalShuffleWrite_Count
${executor.totalShuffleWrite}\n")
- sb.append(s"${prefix}maxMemory_Count ${executor.maxMemory}\n")
+ val prefix = "metrics_executor_"
+ val labels = Seq(
+ "application_id" -> store.applicationInfo.id,
+ "application_name" -> store.applicationInfo.name,
+ "executor_id" -> executor.id
+ ).map { case (k, v) => s"""$k="$v"""" }.mkString("{", ", ", "}")
+ sb.append(s"${prefix}rddBlocks_Count$labels ${executor.rddBlocks}\n")
Review comment:
> No. Prometheus query language support to handle them individually.
Yes. But I am wondering is, now all numbers from all applications are
recorded under same metric. To retrieve number for specified application, does
not Prometheus need to search it among all applications' metric numbers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
