This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 65c3913fd3b7 [SPARK-59010][CORE] Expose the application hold status
through the REST API
65c3913fd3b7 is described below
commit 65c3913fd3b762659f426afaae22411255c1d8a1
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Aug 25 23:20:34 2026 -0700
[SPARK-59010][CORE] Expose the application hold status through the REST API
### What changes were proposed in this pull request?
This PR adds a read-only REST API endpoint that reports whether an
application is held by
`SparkContext.holdExecutors()` (SPARK-58828):
- https://github.com/apache/spark/pull/58054
**Case 1: A Spark Job supporting holding features**
```
$ curl -s
"http://localhost:4040/api/v1/applications/app-20260825144903-0000/holdstatus";
{
"supported" : true,
"held" : false,
"draining" : 0
}
$ curl -X POST "http://localhost:4040/jobs/hold/";
$ curl -s
"http://localhost:4040/api/v1/applications/app-20260825144903-0000/holdstatus";
{
"supported" : true,
"held" : true,
"draining" : 0
}
$ curl -X POST "http://localhost:4040/jobs/resume/";
$ curl -s
"http://localhost:4040/api/v1/applications/app-20260825144903-0000/holdstatus";
{
"supported" : true,
"held" : false,
"draining" : 0
}
```
**Case 2: A Spark Job which doesn't support hold feature**
```
$ curl -s
"http://localhost:4040/api/v1/applications/app-20260825151945-0001/holdstatus";
{
"supported" : false,
"held" : false,
"draining" : 0
}
```
`supported` is whether the deployment allows holding, `held` is whether the
executors are
currently held, and `draining` is the number of executors that have not
exited yet (`0` when
the application is not held).
Holding and resuming stay on the existing `/jobs/hold/` and `/jobs/resume/`
POST endpoints,
which require modify permissions, while reading this status only requires
view permissions.
Like the thread dump endpoints, it is live-only and returns `503` on the
history server.
### Why are the changes needed?
`holdExecutors()` is asynchronous: the executors keep running their tasks
and exit afterwards.
That state is rendered only as HTML on the Jobs page today, and the
`/jobs/hold/` POST answers
with a `302` redirect carrying no outcome, so automation has to scrape the
UI. With this
endpoint a client can hold an application and then poll until `held &&
draining == 0`, and can
check `supported` beforehand instead of requesting a hold that silently
does nothing.
### Does this PR introduce _any_ user-facing change?
Yes, this adds a new REST API endpoint,
`/api/v1/applications/[app-id]/holdstatus`, documented
in `docs/monitoring.md`. No existing endpoint or behavior changes.
### How was this patch tested?
Pass the CIs with the new unit tests in `UISeleniumSuite` (the reported
status across a hold
and the following drain, and `supported = false` on a local backend) and
`HistoryServerSuite`
(the `503` through the history server).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #58294 from dongjoon-hyun/SPARK-59010.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 14e217167a3c73bac09ef0822067a3cf36855566)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../status/api/v1/OneApplicationResource.scala | 12 +++++++++
.../scala/org/apache/spark/status/api/v1/api.scala | 5 ++++
.../spark/deploy/history/HistoryServerSuite.scala | 6 +++++
.../org/apache/spark/ui/UISeleniumSuite.scala | 30 ++++++++++++++++++++++
docs/monitoring.md | 11 ++++++++
5 files changed, 64 insertions(+)
diff --git
a/core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
b/core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
index 448eedf9cc22..c0f73559caec 100644
---
a/core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
+++
b/core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
@@ -92,6 +92,18 @@ private[v1] class AbstractApplicationResource extends
BaseAppResource {
@Path("allmiscellaneousprocess")
def allProcessList(): Seq[ProcessSummary] =
withUI(_.store.miscellaneousProcessList(false))
+ @GET
+ @Path("holdstatus")
+ def holdStatus(): ApplicationHoldStatus = {
+ val safeSparkContext = checkAndGetSparkContext("Hold status")
+ val held = safeSparkContext.executorsHeld
+ new ApplicationHoldStatus(
+ supported = safeSparkContext.executorHoldSupported,
+ held = held,
+ // The executors that have not exited yet are still draining their
running tasks.
+ draining = if (held) safeSparkContext.getExecutorIds().size else 0)
+ }
+
@Path("stages")
def stages(): Class[StagesResource] = classOf[StagesResource]
diff --git a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
index 3e7f5e6d2ec3..1eb25f14e334 100644
--- a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
+++ b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
@@ -66,6 +66,11 @@ case class ApplicationAttemptInfo private[spark](
}
+class ApplicationHoldStatus private[spark](
+ val supported: Boolean,
+ val held: Boolean,
+ val draining: Int)
+
class ResourceProfileInfo private[spark](
val id: Int,
val executorResources: Map[String, ExecutorResourceRequest],
diff --git
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index 79e0001fa22f..f252ac22c4a2 100644
---
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -348,6 +348,12 @@ abstract class HistoryServerSuite extends SparkFunSuite
with BeforeAndAfter with
getContentAndCode("foobar")._1 should be (HttpServletResponse.SC_NOT_FOUND)
}
+ test("SPARK-59010: hold status is not available through the history server")
{
+ val holdStatus =
getContentAndCode("applications/local-1422981780767/holdstatus")
+ holdStatus._1 should be (HttpServletResponse.SC_SERVICE_UNAVAILABLE)
+ holdStatus._3 should be (Some("Hold status not available through the
history server."))
+ }
+
test("automatically retrieve uiRoot from request through Knox") {
assert(sys.props.get("spark.ui.proxyBase").isEmpty,
"spark.ui.proxyBase is defined but it should not for this UT")
diff --git a/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
b/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
index 7fe24572cae1..fde52ed57a3d 100644
--- a/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
+++ b/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
@@ -325,6 +325,36 @@ class UISeleniumSuite extends SparkFunSuite with
WebBrowser with Matchers {
}
}
+ test("SPARK-59010: hold status api") {
+ def holdStatus(sc: SparkContext): (Boolean, Boolean, Int) = {
+ val json = getJson(sc.ui.get, "holdstatus")
+ ((json \ "supported").extract[Boolean], (json \ "held").extract[Boolean],
+ (json \ "draining").extract[Int])
+ }
+
+ // A local backend cannot hold its executors.
+ withSpark(newSparkContext()) { sc =>
+ assert(holdStatus(sc) === (false, false, 0))
+ }
+
+ withSpark(newSparkContext(master = "local-cluster[1,1,1024]",
+ additionalConfs = Map(
+ SHUFFLE_SERVICE_ENABLED.key -> "true",
+ DECOMMISSION_ENABLED.key -> "true"))) { sc =>
+ assert(holdStatus(sc) === (true, false, 0))
+ // Wait for the executor to register, so that the hold has something to
drain.
+ eventually(timeout(30.seconds), interval(200.milliseconds)) {
+ assert(sc.getExecutorIds().nonEmpty)
+ }
+ assert(sc.holdExecutors())
+ assert(holdStatus(sc)._2)
+ // The held executors exit once they are done, and the drain is then
complete.
+ eventually(timeout(30.seconds), interval(200.milliseconds)) {
+ assert(holdStatus(sc) === (true, true, 0))
+ }
+ }
+ }
+
test("jobs page should not display job group name unless some job was
submitted in a job group") {
withSpark(newSparkContext()) { sc =>
// If no job has been run in a job group, then "(Job Group)" should not
appear in the header
diff --git a/docs/monitoring.md b/docs/monitoring.md
index c39a140d7990..7b8d61c525f3 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -656,6 +656,17 @@ can be identified by their `[attempt-id]`. In the API
listed below, when running
<td><code>/applications/[app-id]/allexecutors</code></td>
<td>A list of all(active and dead) executors for the given
application.</td>
</tr>
+ <tr>
+ <td><code>/applications/[app-id]/holdstatus</code></td>
+ <td>
+ Whether the given application is held, as <code>supported</code>
(whether the deployment
+ allows holding), <code>held</code>, and <code>draining</code> (the
number of executors
+ that have not exited yet). An application is held and resumed through the
+ <code>/jobs/hold/</code> and <code>/jobs/resume/</code> POST endpoints
of its web UI,
+ which require modify permissions, while reading this status only
requires view
+ permissions. Not available via the history server.
+ </td>
+ </tr>
<tr>
<td><code>/applications/[app-id]/storage/rdd</code></td>
<td>A list of stored RDDs for the given application.</td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
