agrawaldevesh commented on a change in pull request #28818:
URL: https://github.com/apache/spark/pull/28818#discussion_r452561283
##########
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala
##########
@@ -81,6 +81,35 @@ private[spark] trait ExecutorAllocationClient {
countFailures: Boolean,
force: Boolean = false): Seq[String]
+ /**
+ * Request that the cluster manager decommission the specified executors.
+ * Default implementation delegates to kill, scheduler must override
+ * if it supports graceful decommissioning.
+ *
+ * @param executorIds identifiers of executors to decommission
+ * @param adjustTargetNumExecutors whether the target number of executors
will be adjusted down
+ * after these executors have been
decommissioned.
+ * @return the ids of the executors acknowledged by the cluster manager to
be removed.
+ */
+ def decommissionExecutors(
+ executorIds: Seq[String],
+ adjustTargetNumExecutors: Boolean): Seq[String]
+
+
+ /**
+ * Request that the cluster manager decommission the specified executors.
+ * Default implementation delegates to kill, scheduler must override
+ * if it supports graceful decommissioning.
+ *
+ * @param executorIds identifiers of executors to decommission
+ * @return whether the request is acknowledged by the cluster manager.
+ */
+ def decommissionExecutor(executorId: String): Boolean = {
+ val decommissionedExecutors = decommissionExecutors(Seq(executorId),
+ adjustTargetNumExecutors = true)
Review comment:
Why is `adjustTargetNumExecutors` defaulting to true here ? This would
mean that all schedulers would try to replinish the executor when asked to
`DecommissionExecutor(...)` -- for example by the Master or when an executor
gets a SIGPWR.
I think it shouldn't be the default -- it should atleast be configurable. It
only makes sense to have `adjustTargetNumExecutors=true` when called from
`org.apache.spark.streaming.scheduler.ExecutorAllocationManager#killExecutor`
(ie when it is truly called from dynamic allocation codepath and we have
decided that we want to replinish the executor).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
