This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 3a6795b2 [SPARK-59937] Recommend `spark.driver.timeout` instead of
Kueue `max-exec-time-seconds` label
3a6795b2 is described below
commit 3a6795b2a8215eb9ec543bcc768d092d1ef488f2
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 2 00:11:38 2026 -0700
[SPARK-59937] Recommend `spark.driver.timeout` instead of Kueue
`max-exec-time-seconds` label
### What changes were proposed in this pull request?
This PR aims to recommend the Spark native `spark.driver.timeout` instead
of the Kueue `kueue.x-k8s.io/max-exec-time-seconds` label in
`docs/spark_custom_resources.md`, with a link to the existing
`examples/pi-with-driver-timeout.yaml`.
https://github.com/apache/spark-kubernetes-operator/blob/1d07c7f1f255d0223121e7e26e56113fd0106bf1/examples/pi-with-driver-timeout.yaml#L24-L25
### Why are the changes needed?
The operator does not copy the label to the `Workload`. Even if it did, the
deactivation which Kueue applies after that time is not honored for a running
`SparkApplication`, so its driver and executors would keep running while Kueue
no longer counts their quota. On the other hand, `spark.driver.timeout` with
`DriverTimeoutPlugin` is available in all supported Spark versions (4.0+) and
stops the driver itself, so the attempt releases its `Workload` like any other
stopped attempt.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change for the Kueue integration, which is
not released yet in
[1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0)
(2026-07-23).
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #923 from dongjoon-hyun/SPARK-59937.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/spark_custom_resources.md | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index c549bac8..3de5917d 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -786,6 +786,10 @@ spec:
running attempt, delete the application to release its quota earlier. With
`blockAdmission`, an
application whose pods are not all ready holds back every other workload
until they are ready
or its quota is released as described above.
+* To limit the execution time of a `SparkApplication`, the Spark native
`spark.driver.timeout` is
+ recommended instead of the `kueue.x-k8s.io/max-exec-time-seconds` label,
which is not copied to
+ the `Workload`. It requires
`spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin`, as in
+ [this example](../examples/pi-with-driver-timeout.yaml).
## Spark Cluster
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
