[
https://issues.apache.org/jira/browse/SPARK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-19069.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0
Issue resolved by pull request 16473
[https://github.com/apache/spark/pull/16473]
> Expose task 'status' and 'duration' in spark history server REST API.
> ---------------------------------------------------------------------
>
> Key: SPARK-19069
> URL: https://issues.apache.org/jira/browse/SPARK-19069
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Parag Chaudhari
> Fix For: 2.2.0
>
> Attachments: screenshot-1.png
>
>
> Although Spark history server UI shows task ‘status’ and ‘duration’ fields,
> it does not expose these fields in the REST API response. For the Spark
> history server API users, it is not possible to determine task status and
> duration. Spark history server has access to task status and duration from
> event log, but it is not exposing these in API. This patch is proposed to
> expose task ‘status’ and ‘duration’ fields in Spark history server REST API.
> e.g. Spark history server UI: PFA
> e.g. Spark history sever REST API response with no ‘status’ and ‘duration’:
> {noformat}
> {
> "taskId" : 7,
> "index" : 0,
> "attempt" : 0,
> "launchTime" : "2017-01-02T17:32:43.037GMT",
> "executorId" : "2",
> "host" : "ip-10-171-154-17.ec2.internal",
> "taskLocality" : "NODE_LOCAL",
> "speculative" : false,
> "accumulatorUpdates" : [ ],
> "taskMetrics" : {
> "executorDeserializeTime" : 138,
> "executorRunTime" : 10524,
> "resultSize" : 2078,
> "jvmGcTime" : 240,
> "resultSerializationTime" : 0,
> "memoryBytesSpilled" : 0,
> "diskBytesSpilled" : 0,
> "inputMetrics" : {
> "bytesRead" : 0,
> "recordsRead" : 0
> },
> "outputMetrics" : {
> "bytesWritten" : 7474953,
> "recordsWritten" : 287254
> },
> "shuffleReadMetrics" : {
> "remoteBlocksFetched" : 4,
> "localBlocksFetched" : 3,
> "fetchWaitTime" : 203,
> "remoteBytesRead" : 4740801,
> "localBytesRead" : 2011044,
> "recordsRead" : 134
> },
> "shuffleWriteMetrics" : {
> "bytesWritten" : 0,
> "writeTime" : 0,
> "recordsWritten" : 0
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
