dongjoon-hyun commented on code in PR #58054:
URL: https://github.com/apache/spark/pull/58054#discussion_r3815690048
##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -739,6 +778,95 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
executorDataMap.keySet.toSeq
}
+ /** See `SparkContext.holdExecutors()`. */
+ private[spark] def setExecutorsHeld(held: Boolean): Unit = {
+ executorsHeld = held
+ }
+
+ /**
+ * Whether this backend can hold the executors gracefully. Overridden by
backends whose
+ * cluster manager would terminate running executors on a zero requirement.
+ */
+ private[spark] def supportsExecutorHold: Boolean = true
Review Comment:
Fixed in a927b1e with the opt-in polarity: the base returns false (with the
scaladoc naming both ways a backend can be unable to hold, including the
unacknowledged-publish one you traced), `StandaloneSchedulerBackend` and
`YarnSchedulerBackend` override it to true, and
`KubernetesClusterSchedulerBackend` keeps the `direct`-allocator check. The
unresumable-hold argument settled it.
##########
docs/web-ui.md:
##########
@@ -70,6 +70,16 @@ The information displayed at the top of the page includes:
The current user, application start time, and total uptime are shown in the
footer at the
bottom of every page.
+When the application can be held, the summary shows an **Application** line
with a **(hold)**
+link; clicking it stops requesting new executors and gracefully decommissions
the running ones,
+so each finishes its tasks and then exits. The line then reads `Held` with a
**(resume)** link
Review Comment:
Documented in a927b1e in all three places (this line, the `holdExecutors()`
scaladoc, and the PR description): each executor finishes its running tasks and
then exits, unless `spark.executor.decommission.forceKillTimeout` is set, in
which case a still-busy executor is killed after that timeout. I kept the
behavior rather than skipping the force-kill for hold-driven drains: the config
exists precisely so an operator can bound a drain, and a hold silently ignoring
it would surprise the same audience in the other direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
