cloud-fan commented on a change in pull request #29422:
URL: https://github.com/apache/spark/pull/29422#discussion_r471316953
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -136,7 +139,21 @@ private[spark] class TaskSchedulerImpl(
// IDs of the tasks running on each executor
private val executorIdToRunningTaskIds = new HashMap[String, HashSet[Long]]
- private val executorsPendingDecommission = new HashMap[String,
ExecutorDecommissionInfo]
+ // We add executors here when we first get decommission notification for
them. Executors can
+ // continue to run even after being asked to decommission, but they will
eventually exit.
+ val executorsPendingDecommission = new HashMap[String,
ExecutorDecommissionInfo]
+
+ // When they exit and we know of that via heartbeat failure, we will add
them to this cache.
+ // This cache is consulted to know if a fetch failure is because a source
executor was
+ // decommissioned.
+ lazy val decommissionedExecutorsRemoved = CacheBuilder.newBuilder()
+ .expireAfterWrite(
+ conf.getLong("spark.decommissioningRememberAfterRemoval.seconds", 60L),
TimeUnit.SECONDS)
Review comment:
I'm not sure how frequent decommission happens. Each entry here should
take less than 1kb, so even decommission happens once per second, it's less
than 300kb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
