Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/20408#discussion_r164177919
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -594,12 +606,24 @@ private[spark] class AppStatusListener(
stage.executorSummaries.values.foreach(update(_, now))
update(stage, now, last = true)
+
+ val executorIdsForStage = stage.executorSummaries.keySet
+ executorIdsForStage.foreach { executorId =>
+ liveExecutors.get(executorId).foreach { exec =>
+ removeBlackListedStageFrom(exec, event.stageInfo.stageId, now)
--- End diff --
I'm just doing a really quick scan here, but I don't understand why changes
here are necessary. You don't get these events for blacklisting within a stage.
or is this a bug in the current code, and something we should fix in the
pending 2.3 release?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
