squito commented on a change in pull request #24132: [SPARK-27189][CORE] Add
Executor metrics and memory usage instrumentation to the metrics system
URL: https://github.com/apache/spark/pull/24132#discussion_r353992410
##########
File path:
core/src/main/scala/org/apache/spark/executor/ExecutorMetricsSource.scala
##########
@@ -19,9 +19,18 @@ package org.apache.spark.executor
import com.codahale.metrics.{Gauge, MetricRegistry}
-import org.apache.spark.metrics.ExecutorMetricType
+import org.apache.spark.metrics.{ExecutorMetricType, MetricsSystem}
import org.apache.spark.metrics.source.Source
+// Expose executor metrics from [[ExecutorMetricsType]] using the Dropwizard
metrics system
+// Metrics related the memory system can be expensive to gather, we implement
some optimizations:
+// (1) Metrics values are cached, updated at each heartbeat (default period is
10 seconds)
+// An alternative faster polling mechanism is used only if activated, by
setting
+// spark.executor.metrics.pollingInterval=<interval in ms>
+// (2) procfs metrics are gathered all in one-go and only conditionally:
+// if the /proc filesystem exists
+// and spark.eventLog.logStageExecutorProcessTreeMetrics.enabled=true
+// and spark.eventLog.logStageExecutorMetrics.enabled=true
Review comment:
can you change to javadoc style? (`/**`) just helps in IDEs
also some nits:
blank line after the first summary line, end sentence with period.
missing "to" and "so" in "Metrics related to the memory system can be
expensive to gather, so ..."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
