Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7888#discussion_r44202096
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -509,6 +511,13 @@ private[spark] class ExecutorAllocationManager(
private def onExecutorBusy(executorId: String): Unit = synchronized {
logDebug(s"Clearing idle timer for $executorId because it is now
running a task")
removeTimes.remove(executorId)
+
+ // Executor is added to remove by misjudgment due to async listener
making it as idle).
+ // see SPARK-9552
+ if (executorsPendingToRemove.contains(executorId)) {
--- End diff --
Instead of fixing the list of pending executors here, I wonder if it
wouldn't be better to change `removeExecutor` to return `false` when the
executors asked to be killed were busy? The they wouldn't even be added to this
list to start with.
That changes slightly the semantics of the return value, but it also sounds
more correct. With your changes, `killExecutors(..., force = false)` will
return `true` even if it didn't kill any executors, which sounds wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
