wypoon commented on a change in pull request #23767: [SPARK-26329][CORE] Faster
polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r270285004
##########
File path: core/src/main/scala/org/apache/spark/executor/Executor.scala
##########
@@ -191,9 +191,20 @@ private[spark] class Executor(
*/
private val HEARTBEAT_INTERVAL_MS = conf.get(EXECUTOR_HEARTBEAT_INTERVAL)
+ /**
+ * Interval to poll for executor metrics, in milliseconds
+ */
+ private val METRICS_POLLING_INTERVAL_MS =
conf.get(EXECUTOR_METRICS_POLLING_INTERVAL)
Review comment:
When executor metrics were introduced, there was a config introduced,
spark.eventLog.logStageExecutorMetrics.enabled. However, that only controlled
whether aggregated metrics (peaks per stage per executor) were written to event
logs; it did not control polling. Regardless of the value of
spark.eventLog.logStageExecutorMetrics.enabled, executor metrics were polled
and sent in the executor heartbeat. The polling was done in the reportHeartBeat
function, so was done at the frequency of the heartbeat.
In order to have faster polling, which is the whole point of this change, it
is necessary to have a config specifying how frequently to poll, so there is no
redundant config, if that is what you mean. It should not be tied to the
executor heartbeat frequency, since the use case is we don't want faster
heartbeats, only faster polling.
And there is no redundant polling for metrics, if that is what you mean. If
spark.executor.metrics.pollingInterval is not set, then we continue to poll in
reportHeartBeat, as before. There are not two separate polling threads; there
is only ever one thread per executor that polls executor metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
