[
https://issues.apache.org/jira/browse/SPARK-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hovell reassigned SPARK-27071:
-----------------------------------------
Assignee: Tom van Bussel
> Expose additional metrics in status.api.v1.StageData
> ----------------------------------------------------
>
> Key: SPARK-27071
> URL: https://issues.apache.org/jira/browse/SPARK-27071
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Tom van Bussel
> Assignee: Tom van Bussel
> Priority: Major
>
> Currently StageData exposes the following metrics:
> * executorRunTime
> * executorCpuTime
> * inputBytes
> * inputRecords
> * outputBytes
> * outputRecords
> * shuffleReadBytes
> * shuffleReadRecords
> * shuffleWriteBytes
> * shuffleWriteRecords
> * memoryBytesSpilled
> * diskBytesSpilled
> These metrics are computed by aggregating the metrics of the tasks in the
> stage. For the task metrics however we keep track of a lot more metrics.
> Currently these metrics are also computed for stages (such shuffle read fetch
> wait time), but these are not exposed through the api. It would be very
> useful if these were also exposed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
