edwinalu commented on a change in pull request #23767: [SPARK-26329][CORE][WIP]
Faster polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r256956538
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -2380,10 +2381,14 @@ class SparkContext(config: SparkConf) extends Logging {
/** Reports heartbeat metrics for the driver. */
private def reportHeartBeat(): Unit = {
- val driverUpdates = _heartbeater.getCurrentMetrics()
+ val currentMetrics = ExecutorMetrics.getCurrentMetrics(env.memoryManager)
+ val driverUpdates = new HashMap[(Int, Int), ExecutorMetrics]
+ // In the driver, we do not track per-stage metrics, so use a dummy stage
+ // for the key
+ driverUpdates.put((-1, -1), new ExecutorMetrics(currentMetrics))
Review comment:
Per stage metrics could still be useful for the driver, for an application
that is long-running, and if the driver memory is changing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
