edwinalu commented on a change in pull request #23767: [SPARK-26329][CORE][WIP]
Faster polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r257536532
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -2380,10 +2381,14 @@ class SparkContext(config: SparkConf) extends Logging {
/** Reports heartbeat metrics for the driver. */
private def reportHeartBeat(): Unit = {
- val driverUpdates = _heartbeater.getCurrentMetrics()
Review comment:
We've seen some issues with OOM on the driver, but it is much more common
for executors, so agreed that it is not as important for the driver. For
accurate stage information, one option could be to pass in a pointer to the
DAGScheduler, and add a method to DAGScheduler to return the running stages.
Thanks for your work improving the accuracy of the metrics. Have a great
vacation! Please let me know if you have any questions about the comments, or
want to discuss anything when you get back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
