[
https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512643#comment-16512643
]
Thomas Graves commented on SPARK-24539:
---------------------------------------
Its possible, I thought when I checked the history server I was actually seeing
them aggregated properly but I don't know if I checked the specific task
events. Probably all related.
> HistoryServer does not display metrics from tasks that complete after stage
> failure
> -----------------------------------------------------------------------------------
>
> Key: SPARK-24539
> URL: https://issues.apache.org/jira/browse/SPARK-24539
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.3.1
> Reporter: Imran Rashid
> Priority: Major
>
> I noticed that task metrics for completed tasks with a stage failure do not
> show up in the new history server. I have a feeling this is because all of
> the tasks succeeded *after* the stage had been failed (so they were
> completions from a "zombie" taskset). The task metrics (eg. the shuffle read
> size & shuffle write size) do not show up at all, either in the task table,
> the executor table, or the overall stage summary metrics. (they might not
> show up in the job summary page either, but in the event logs I have, there
> is another successful stage attempt after this one, and that is the only
> thing which shows up in the jobs page.) If you get task details from the api
> endpoint (eg.
> http://[host]:[port]/api/v1/applications/[app-id]/stages/[stage-id]/[stage-attempt])
> then you can see the successful tasks and all the metrics
> Unfortunately the event logs I have are huge and I don't have a small repro
> handy, but I hope that description is enough to go on.
> I loaded the event logs I have in the SHS from spark 2.2 and they appear fine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
