Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22288#discussion_r227111412
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -415,9 +420,55 @@ private[spark] class TaskSchedulerImpl(
launchedAnyTask |= launchedTaskAtCurrentMaxLocality
} while (launchedTaskAtCurrentMaxLocality)
}
+
if (!launchedAnyTask) {
- taskSet.abortIfCompletelyBlacklisted(hostToExecutors)
+ taskSet.getCompletelyBlacklistedTaskIfAny(hostToExecutors) match
{
+ case Some(taskIndex) => // Returns the taskIndex which was
unschedulable
+
+ // If the taskSet is unschedulable we try to find an
existing idle blacklisted
+ // executor. If we cannot find one, we abort immediately.
Else we kill the idle
--- End diff --
I don't think this is true -- if there is no idle executor here, you abort
the taskset immediately, you're not starting any timer, from this case lower
down: `case _ => // Abort Immediately`.
I think to do what you described, you would instead need to do something
different in that case, like start the same abortTimer, and also set a flag
`needToKillIdleExecutor` and then on every call to `resourceOffer`, check that
flag and potentially find an executor to kill. (However I haven't totally
thought through that, not sure if it would really work. again, I'm not saying
this has to be addressed now, just thinking this through)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
