edwinalu commented on a change in pull request #23767: [SPARK-26329][CORE][WIP]
Faster polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r257385711
##########
File path: core/src/main/scala/org/apache/spark/executor/Executor.scala
##########
@@ -524,10 +605,19 @@ private[spark] class Executor(
executorSource.METRIC_DISK_BYTES_SPILLED.inc(task.metrics.diskBytesSpilled)
executorSource.METRIC_MEMORY_BYTES_SPILLED.inc(task.metrics.memoryBytesSpilled)
+ def getMetricPeaks(): Array[Long] = {
+ val currentPeaks = taskMetricPeaks.get(taskId)
Review comment:
It's possible for very short tasks that polling may not have occurred while
the task was running. I think it's OK for values to be zero, since it won't
affect peak calculations on the other end. One option would be to get the
metric values when the task is starting and/or ending, but there would be extra
overhead especially for stages with a lot of very short tasks, which does not
seem worthwhile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
