[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-23429.
--------------------------------
Resolution: Fixed
> Add executor memory metrics to heartbeat and expose in executors REST API
> -------------------------------------------------------------------------
>
> Key: SPARK-23429
> URL: https://issues.apache.org/jira/browse/SPARK-23429
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 2.2.1
> Reporter: Edwina Lu
> Priority: Major
>
> Add new executor level memory metrics ( jvmUsedMemory, onHeapExecutionMemory,
> offHeapExecutionMemory, onHeapStorageMemory, offHeapStorageMemory,
> onHeapUnifiedMemory, and offHeapUnifiedMemory), and expose these via the
> executors REST API. This information will help provide insight into how
> executor and driver JVM memory is used, and for the different memory regions.
> It can be used to help determine good values for spark.executor.memory,
> spark.driver.memory, spark.memory.fraction, and spark.memory.storageFraction.
> Add an ExecutorMetrics class, with jvmUsedMemory, onHeapExecutionMemory,
> offHeapExecutionMemory, onHeapStorageMemory, and offHeapStorageMemory. This
> will track the memory usage at the executor level. The new ExecutorMetrics
> will be sent by executors to the driver as part of the Heartbeat. A heartbeat
> will be added for the driver as well, to collect these metrics for the driver.
> Modify the EventLoggingListener to log ExecutorMetricsUpdate events if there
> is a new peak value for one of the memory metrics for an executor and stage.
> Only the ExecutorMetrics will be logged, and not the TaskMetrics, to minimize
> additional logging. Analysis on a set of sample applications showed an
> increase of 0.25% in the size of the Spark history log, with this approach.
> Modify the AppStatusListener to collect snapshots of peak values for each
> memory metric. Each snapshot has the time, jvmUsedMemory, executionMemory and
> storageMemory, and list of active stages.
> Add the new memory metrics (snapshots of peak values for each memory metric)
> to the executors REST API.
> This is a subtask for SPARK-23206. Please refer to the design doc for that
> ticket for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
