wypoon commented on a change in pull request #23767: [SPARK-26329][CORE] Faster
polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r299204611
##########
File path: core/src/main/scala/org/apache/spark/executor/Executor.scala
##########
@@ -412,6 +432,11 @@ private[spark] class Executor(
env.mapOutputTracker.asInstanceOf[MapOutputTrackerWorker].updateEpoch(task.epoch)
}
+ val stageId = task.stageId
+ val stageAttemptId = task.stageAttemptId
+ metricsPoller.onTaskStart(taskId, stageId, stageAttemptId)
+ taskStarted = true
Review comment:
This boolean is needed because we call metricsPoller.onTaskStart in this try
block here and in the finally block we want to call
metricsPoller.onTaskCompletion to do cleanup. However, we do not want to call
metricsPoller.onTaskCompletion if we exited the try block before
metricsPoller.onTaskStart has been called. The contract for
ExecutorMetricsPoller is that onTaskCompletion should only be called if
onTaskStart has been called first with the same arguments.
There is a comment in the finally block explaining the significance
taskStarted. I didn't think I need a comment here as well. But let me see if I
can add a suitable comment before the finally block.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
