[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511365#comment-16511365 ]
Marcelo Vanzin commented on SPARK-24539: ---------------------------------------- >From a previous chat with [~tgraves] this sounds like SPARK-24415. > HistoryServer does not display metrics from tasks that complete after stage > failure > ----------------------------------------------------------------------------------- > > Key: SPARK-24539 > URL: https://issues.apache.org/jira/browse/SPARK-24539 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 2.3.1 > Reporter: Imran Rashid > Priority: Major > > I noticed that task metrics for completed tasks with a stage failure do not > show up in the new history server. I have a feeling this is because all of > the tasks succeeded *after* the stage had been failed (so they were > completions from a "zombie" taskset). The task metrics (eg. the shuffle read > size & shuffle write size) do not show up at all, either in the task table, > the executor table, or the overall stage summary metrics. (they might not > show up in the job summary page either, but in the event logs I have, there > is another successful stage attempt after this one, and that is the only > thing which shows up in the jobs page.) If you get task details from the api > endpoint (eg. > http://[host]:[port]/api/v1/applications/[app-id]/stages/[stage-id]/[stage-attempt]) > then you can see the successful tasks and all the metrics > Unfortunately the event logs I have are huge and I don't have a small repro > handy, but I hope that description is enough to go on. > I loaded the event logs I have in the SHS from spark 2.2 and they appear fine. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org