GitHub user edwinalu opened a pull request:
https://github.com/apache/spark/pull/20940
[SPARK-23429][CORE] Add executor memory metrics to heartbeat and expose in
executors REST API
Add new executor level memory metrics (JVM used memory, on/off heap
execution memory, on/off heap storage memory), and expose via the executors
REST API. This information will help provide insight into how executor and
driver JVM memory is used, and for the different memory regions. It can be used
to help determine good values for spark.executor.memory, spark.driver.memory,
spark.memory.fraction, and spark.memory.storageFraction.
## What changes were proposed in this pull request?
An ExecutorMetrics class is added, with jvmUsedMemory,
onHeapExecutionMemory, offHeapExecutionMemory, onHeapStorageMemory, and
offHeapStorageMemory. The new ExecutorMetrics is sent by executors to the
driver as part of Heartbeat. A heartbeat is added for the driver as well, to
collect these metrics for the driver.
The EventLoggingListener logs ExecutorMetricsUpdate events if there is a
new peak value for any of the memory metrics for an executor and stage. Only
the ExecutorMetrics will be logged, and not the TaskMetrics, to minimize
additional logging.
The AppStatusListener records the peak values for each memory metric.
The new memory metrics are added to the executors REST API.
## How was this patch tested?
New unit tests have been added. This was also tested on our cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/edwinalu/spark SPARK-23429
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20940.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20940
----
commit 48eab1d23fcebb315bbe787bed8e468c7fc69233
Author: Edwina Lu <edlu@...>
Date: 2018-03-09T23:39:36Z
SPARK-23429: Add executor memory metrics to heartbeat and expose in
executors REST API
Add new executor level memory metrics (JVM used memory, on/off heap
execution memory, on/off heap storage
memory), and expose via the executors REST API. This information will help
provide insight into how executor
and driver JVM memory is used, and for the different memory regions. It can
be used to help determine good
values for spark.executor.memory, spark.driver.memory,
spark.memory.fraction, and spark.memory.storageFraction.
Add an ExecutorMetrics class, with jvmUsedMemory, onHeapExecutionMemory,
offHeapExecutionMemory,
onHeapStorageMemory, and offHeapStorageMemory. The new ExecutorMetrics will
be sent by executors to the
driver as part of Heartbeat. A heartbeat will be added for the driver as
well, to collect these metrics
for the driver.
Modify the EventLoggingListener to log ExecutorMetricsUpdate events if
there is a new peak value for any
of the memory metrics for an executor and stage. Only the ExecutorMetrics
will be logged, and not the
TaskMetrics, to minimize additional logging.
Modify the AppStatusListener to record the peak values for each memory
metric.
Add the new memory metrics to the executors REST API.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
