GitHub user ankuriitg opened a pull request:
https://github.com/apache/spark/pull/22209
[SPARK-24415][Core] HistoryServer does not display metrics from tasksâ¦
⦠that
complete after stage failure
## What changes were proposed in this pull request?
The problem occurs because stage object is removed from liveStages in
AppStatusListener onStageCompletion. Because of this any onTaskEnd event
received after onStageCompletion event do not update stage metrics.
The fix is to retain stage objects in liveStages until all tasks are
complete.
## How was this patch tested?
1. Fixed the reproducible example posted in the JIRA
2. Added unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ankuriitg/spark ankurgupta/SPARK-24415
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22209.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22209
----
commit cd1e8564cd04ae6d94857608b87ace3fef975136
Author: ankurgupta <ankur.gupta@...>
Date: 2018-08-23T18:48:40Z
[SPARK-24415][Core] HistoryServer does not display metrics from tasks that
complete after stage failure
The problem occurs because stage object is removed from liveStages in
AppStatusListener onStageCompletion. Because of this any onTaskEnd event
received after onStageCompletion event do not update stage metrics.
The fix is to retain stage objects in liveStages until all tasks are
complete.
Testing Done:
1. Fixed the reproducible example posted in the JIRA
2. Added unit test
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
