Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2432#discussion_r18381560
--- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
---
@@ -98,10 +96,45 @@ private[spark] class MetricsSystem private (
sinks.foreach(_.report())
}
+ /**
+ * Build a unique name for each combination of application,
executor/driver and metric source.
+ * The name is built like <Application ID>.<Executor ID( or "driver" for
Driver)>.<Metric name>
+ *
+ * @param source Metric source to be named by this method.
+ * @return An unique metric name for each combination of
+ * application, executor/driver and metric source.
+ */
+ def buildRegistryName(source: Source): String =
+ instance match {
+ /**
+ * Only Driver and Executor are set spark.app.id and
spark.executor.id.
+ * For instance, Master and Worker are not related to a specific
application.
+ */
+ case "driver" | "executor" =>
+ val appIdOpt = conf.getOption("spark.app.id")
+ val executorIdOpt = conf.getOption("spark.executor.id")
+ (for {
+ appId <- appIdOpt
+ executorId <- executorIdOpt
+ } yield {
+ MetricRegistry.name(appId, executorId, source.sourceName)
+ }).getOrElse {
+ if (appIdOpt == None) {
+ logWarning("spark.app.id is not set.")
+ }
+ if (executorIdOpt == None) {
+ logWarning("spark.executor.id is not set")
+ }
+ MetricRegistry.name(source.sourceName)
+ }
+ case _ => MetricRegistry.name(source.sourceName)
+ }
--- End diff --
Almost agree with you but we need to check whether the instance is
(driver|executor) or not.
Metrics for master/worker/application for Spark standalone mode should not
be set spark.app.id and spark.executor.id.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
