dongjoon-hyun commented on code in PR #58054:
URL: https://github.com/apache/spark/pull/58054#discussion_r3815692618
##########
core/src/main/scala/org/apache/spark/internal/config/UI.scala:
##########
@@ -92,6 +92,18 @@ private[spark] object UI {
.booleanConf
.createWithDefault(true)
+ val UI_HOLD_ENABLED = ConfigBuilder("spark.ui.holdEnabled")
+ .doc("Allows the whole application to be held and resumed from the web UI.
Holding " +
+ "gracefully decommissions all executors and stops requesting new ones.
Cached blocks " +
+ "are not preserved and are recomputed after resuming. This takes effect
only when " +
Review Comment:
Fixed in a927b1e. All four sites (this doc, `configuration.md`, `web-ui.md`,
the PR description) now name the third condition, enumerated positively: a
cluster manager that can hold executors -- Standalone, YARN, and Kubernetes
with the `direct` pods allocator. The scaladoc gained an `@throws
IllegalArgumentException` clause distinguishing the throwing
config-precondition path from the false-returning unsupported-backend path, and
records that fallback storage deliberately does not qualify (blocks not yet
migrated when an executor exits are dropped).
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -2093,8 +2093,15 @@ private[spark] class DAGScheduler(
} catch {
case e: BarrierJobSlotsNumberCheckFailed =>
// If jobId doesn't exist in the map, Scala coverts its value null to
0: Int automatically.
- val numCheckFailures =
barrierJobIdToNumTasksCheckFailures.compute(jobId,
- (_: Int, value: Int) => value + 1)
+ // Do not consume the retry budget while the executors are held: the
slot check sees
+ // zero slots for the whole hold, and the job should wait for the
resume like any
+ // other job instead of failing when the retries run out.
+ val numCheckFailures = if (sc.executorsHeld) {
Review Comment:
Added in a927b1e. `DAGScheduler` gained the `executorsHeld` seam you
suggested, used by both the barrier retry-budget freeze and the new pipelined
deferral, with `DAGSchedulerSuite` tests for each (the barrier one asserts
`barrierJobIdToNumTasksCheckFailures` stays empty while held).
`KubernetesClusterSchedulerBackendSuite` covers `supportsExecutorHold` per
allocator value (`direct` yes; `statefulset`, `deployment`, and a custom class
no), and `UISeleniumSuite` gained the `spark.ui.holdEnabled` twin of the
killEnabled test on `local-cluster`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
