wypoon commented on a change in pull request #23767: [SPARK-26329][CORE][WIP]
Faster polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r257489061
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -2380,10 +2381,14 @@ class SparkContext(config: SparkConf) extends Logging {
/** Reports heartbeat metrics for the driver. */
private def reportHeartBeat(): Unit = {
- val driverUpdates = _heartbeater.getCurrentMetrics()
Review comment:
My thinking, as I commented in the JIRA, was that most memory issues are
with executors rather than with the driver. So I thought it was much less
important to poll frequently in the driver.
That said, we can poll more frequently in the driver as well and keep track
of peaks since the last heartbeat. The only issue though is that in the driver
we don't know what stages are active each time we poll; we rely on the
EventLoggingListener, which knows when a stage has started or ended, to
attribute the driver's update to the stage. Peaks since the last heartbeat
could cross stage boundaries. But maybe we don't care?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
