Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8887#discussion_r42438909
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -482,21 +482,28 @@ private[spark] class TaskSchedulerImpl(
}
}
- /** Remove an executor from all our data structures and mark it as lost
*/
+ /**
+ * Remove an executor from all our data structures and mark it as lost.
If the executor's loss
+ * reason is not yet known, do not yet remove its association with its
host.
+ */
private def removeExecutor(executorId: String, reason:
ExecutorLossReason) {
activeExecutorIds -= executorId
val host = executorIdToHost(executorId)
val execs = executorsByHost.getOrElse(host, new HashSet)
- execs -= executorId
- if (execs.isEmpty) {
- executorsByHost -= host
- for (rack <- getRackForHost(host); hosts <- hostsByRack.get(rack)) {
- hosts -= host
- if (hosts.isEmpty) {
- hostsByRack -= rack
+
+ if (reason != LossReasonPending) {
--- End diff --
I can't remove the information about where the executor was running;
otherwise, the second time `executorLost` is called, `host` (L491) will be null.
I could change the code so that only `executorsByHost -= host` is within
the check, but it didn't seem necessary. What really controls whether things
are scheduled on the executor is `CoarseGrainedSchedulerBackend` calling
`resourceOffers`, and it won't offer resources for executors that are pending
removal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
