Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/20408#discussion_r164292079
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -594,12 +606,24 @@ private[spark] class AppStatusListener(
stage.executorSummaries.values.foreach(update(_, now))
update(stage, now, last = true)
+
+ val executorIdsForStage = stage.executorSummaries.keySet
+ executorIdsForStage.foreach { executorId =>
+ liveExecutors.get(executorId).foreach { exec =>
+ removeBlackListedStageFrom(exec, event.stageInfo.stageId, now)
--- End diff --
I guess github diff collapse tricked us here. This changes belongs to the
method onStageCompleted (and definitely not for onExecutorUnblacklisted). This
is where I remove completed stages from the live executors.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
