Github user edwinalu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187507940
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -93,6 +94,10 @@ private[spark] class EventLoggingListener(
// Visible for tests only.
private[scheduler] val logPath = getLogPath(logBaseDir, appId,
appAttemptId, compressionCodecName)
+ // map of live stages, to peak executor metrics for the stage
+ private val liveStageExecutorMetrics = mutable.HashMap[(Int, Int),
--- End diff --
This is tracking peak metric values for executors for each stage, so that
the peak values for the stage can be dumped at stage end. The purpose is to
reduce the amount of logging, to only number of stages * number of executors
ExecutorMetricsUpdate events.
I originally tried logging for new peak values, resetting when a new stage
begins -- this is simpler, but can lead to more events being logged.
Having stage level information is useful for users trying to identify which
stages are more memory intensive. This information could be useful they are
trying to reduce the amount of memory used, since they would know which stages
(and the relevant code) to focus on.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
