Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20203#discussion_r161045636
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -223,6 +228,15 @@ private[spark] class AppStatusListener(
updateNodeBlackList(event.hostId, false)
}
+ def updateBlackListStatusForStage(executorId: String, stageId: Int,
stageAttemptId: Int): Unit = {
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ val now = System.nanoTime()
+ val esummary = stage.executorSummary(executorId)
+ esummary.isBlacklisted = true
+ maybeUpdate(esummary, now)
+ }
+ }
+
--- End diff --
`liveUpdate` / `maybeUpdate` are optimizations to avoid unnecessary writes
to disk. They can be called for intermediate updates (which would be reflected
in a live application), and only force the write at the very last update (so
the data is written in the SHS).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
