[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-23429:
------------------------------------
Assignee: (was: Apache Spark)
> Add executor memory metrics to heartbeat and expose in executors REST API
> -------------------------------------------------------------------------
>
> Key: SPARK-23429
> URL: https://issues.apache.org/jira/browse/SPARK-23429
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 2.2.1
> Reporter: Edwina Lu
> Priority: Major
>
> Add new executor level memory metrics ( jvmUsedMemory, executionMemory,
> storageMemory, and unifiedMemory), and expose these via the executors REST
> API. This information will help provide insight into how executor and driver
> JVM memory is used, and for the different memory regions. It can be used to
> help determine good values for spark.executor.memory, spark.driver.memory,
> spark.memory.fraction, and spark.memory.storageFraction.
> Add an ExecutorMetrics class, with jvmUsedMemory, executionMemory, and
> storageMemory. This will track the memory usage at the executor level. The
> new ExecutorMetrics will be sent by executors to the driver as part of the
> Heartbeat. A heartbeat will be added for the driver as well, to collect these
> metrics for the driver.
> Modify the EventLoggingListener to log ExecutorMetricsUpdate events if there
> is a new peak value for one of the memory metrics for an executor and stage.
> Only the ExecutorMetrics will be logged, and not the TaskMetrics, to minimize
> additional logging. Analysis on a set of sample applications showed an
> increase of 0.25% in the size of the Spark history log, with this approach.
> Modify the AppStatusListener to collect snapshots of peak values for each
> memory metric. Each snapshot has the time, jvmUsedMemory, executionMemory and
> storageMemory, and list of active stages.
> Add the new memory metrics (snapshots of peak values for each memory metric)
> to the executors REST API.
> This is a subtask for SPARK-23206. Please refer to the design doc for that
> ticket for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
