[
https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23431:
------------------------------
Description:
Collect and show the new executor memory metrics for each stage, to provide
more information on how memory is used per stage.
Modify the AppStatusListener to track the peak values for JVM used memory,
execution memory, storage memory, and unified memory for each executor for each
stage.
This is a subtask for SPARK-23206. Please refer to the design doc for that
ticket for more details.
was:
Collect and show the new executor memory metrics for each stage, to provide
more information on how memory is used per stage.
Modify the AppStatusListener to track the peak values for JVM used memory,
execution memory, storage memory, and unified memory for each executor for each
stage.
Add the peak values for the metrics to the stages REST API. Also add a new
stageSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://<spark history server>:18080/api/v1/applications/<application
id>/<application attempt/stages/<stage id>/<stage attempt>/executorSummary{code}
This is a subtask for SPARK-23206. Please refer to the design doc for that
ticket for more details.
> Expose the new executor memory metrics at the stage level
> ---------------------------------------------------------
>
> Key: SPARK-23431
> URL: https://issues.apache.org/jira/browse/SPARK-23431
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 2.2.1
> Reporter: Edwina Lu
> Priority: Major
>
> Collect and show the new executor memory metrics for each stage, to provide
> more information on how memory is used per stage.
> Modify the AppStatusListener to track the peak values for JVM used memory,
> execution memory, storage memory, and unified memory for each executor for
> each stage.
> This is a subtask for SPARK-23206. Please refer to the design doc for that
> ticket for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
