dongjoon-hyun opened a new pull request, #58346:
URL: https://github.com/apache/spark/pull/58346
### What changes were proposed in this pull request?
This PR lets a driver report its hold status to the standalone Master, so
that the Master UI and
`/json/` endpoint can show it. Display only -- the `(hold)` / `(resume)`
controls stay on the
driver web UI.
- `SparkContext` calls a new no-op
`CoarseGrainedSchedulerBackend.reportExecutorHoldStatus` hook
after initialization and on every hold/resume transition.
`StandaloneSchedulerBackend` overrides
it and forwards to `StandaloneAppClient`, which sends the new
`ApplicationHoldUpdated` message to
the Master. The status is cached and re-sent on registration and failover.
- The Master mirrors it onto `ApplicationInfo.holdSupported` / `held`
(`@transient`; a new Master
learns it again from the driver's re-report). The draining count is not
pushed: the Master
derives it from the executors it already tracks.
- The Master UI annotates the state column (e.g. `RUNNING (held, draining 2
executors)`), and
`/json/` gains `holdsupported`, `held`, and `draining` fields per
application.
`spark.ui.holdEnabled` is honored (an opted-out application is not reported
as holdable), and
finished applications are never annotated. No new configuration and no new
public API.
### Why are the changes needed?
SPARK-58828 and SPARK-59010 made the hold status visible only per
application on the driver. An
operator on the Master page cannot tell which applications are held or still
draining. This also
lays the groundwork for offering the controls on the Master UI in a
follow-up PR.
### Does this PR introduce _any_ user-facing change?
Yes, additive: the Master UI annotates held applications in the state
column, and each application
in `/json/` gains `holdsupported`, `held`, and `draining` fields.
### How was this patch tested?
Three new tests: `AppClientSuite` (status mirrored onto the Master across
transitions),
`MasterSuite` (`numDrainingExecutors`), and `JsonProtocolSuite` (new JSON
fields). Also ran
`SparkContextSuite`, `RecoverySuite`, `StandaloneDynamicAllocationSuite`,
and scalastyle locally;
all passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
