Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23038#discussion_r234400684
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -565,7 +567,11 @@ private[spark] class AppStatusListener(
if (metricsDelta != null) {
esummary.metrics = LiveEntityHelpers.addMetrics(esummary.metrics,
metricsDelta)
}
- conditionalLiveUpdate(esummary, now, removeStage)
+
+ val isLastTask =
(stage.activeTasksPerExecutor(event.taskInfo.executorId) == 0) &&
+ ((stage.status == v1.StageStatus.COMPLETE) || (stage.status ==
v1.StageStatus.FAILED))
--- End diff --
This issue occurs, when the taskEvent comes after stageEnd. Because during
'OnStageCompletd' event, we are writing all the esummary to the store. So,
'OnTaskEnd' method, we just need to force write only if the stageCompleted
event already have happened.
Yes. the stageEnd check isn't really required, as we always update on the
last task of each executors of the particular stage. I updated
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
