Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14926#discussion_r77719621
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -392,10 +397,36 @@ private[spark] class ExecutorAllocationManager(
}
/**
- * Request the cluster manager to remove the given executor.
+ * Request the cluster manager to remove the given executors.
* Return whether the request is received.
*/
- private def removeExecutor(executorId: String): Boolean = synchronized {
+ private def removeExecutors(executorIds: Seq[String]): Boolean =
synchronized {
+
+ val executorIdsToBeRemoved = executorIds.filter(canBeKilled)
+
+ // Send a request to the backend to kill this executor
+ val removeRequestAcknowledged = testing ||
client.killExecutors(executorIdsToBeRemoved)
--- End diff --
so I took a quick look at this and I agree with you that ideally its
returning ones actually removed, but right now I don't see any of the backends
actually returning anything other then true unless its something catastrophic
where it wouldn't matter if the executor list is 1 or many.
Yarn mode right now always returns true (ApplicationMaster KillExecutors
endpoint). Standalone mode only return false if the app doesn't exist so if we
pass 1 or many executors doesn't matter, mesos mode only returns false if the
mesosdriver is null.
So changing the interface to do the right things seems like it can be a
separate issue then this one. or perhaps I missed something?
But it looks like there are a few cases that mesos and standalone mode do
return false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
