Edwina Lu created SPARK-23431:

             Summary: Expose the new executor memory metrics at the stage level
                 Key: SPARK-23431
                 URL: https://issues.apache.org/jira/browse/SPARK-23431
             Project: Spark
          Issue Type: Sub-task
          Components: Spark Core
    Affects Versions: 2.2.1
            Reporter: Edwina Lu

Collect and show the new executor memory metrics for each stage, to provide 
more information on how memory is used per stage.

Modify the AppStatusListener to track the peak values for JVM used memory, 
execution memory, storage memory, and unified memory for each executor for each 

Add the peak values for the metrics to the stages REST API. Also add a new 
stageSummary REST API, which will return executor summary metrics for a 
specified stage:
curl http://<spark history server>:18080/api/v1/applications/<application 
id>/<application attempt/stages/<stage id>/<stage attempt>/executorSummary{code}
This is a subtask for SPARK-23206. Please refer to the design doc for that 
ticket for more details.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to