This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 954b8a831e32 [SPARK-58334][K8S][DOC] Introduce Apache Spark K8s
Operator in `running-on-kubernetes.md`
954b8a831e32 is described below
commit 954b8a831e32c9cb6215a6a8c4bb12a0cb71a6cb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 24 17:30:46 2026 -0700
[SPARK-58334][K8S][DOC] Introduce Apache Spark K8s Operator in
`running-on-kubernetes.md`
### What changes were proposed in this pull request?
This PR aims to introduce `Apache Spark Kubernetes Operator` in
`running-on-kubernetes.md` by adding a short section after `How it works`
section.
- What the operator is: a separate Apache Spark project that deploys and
manages Spark
workloads declaratively via `SparkApp` and `SparkCluster` custom
resources.
- Its relationship with the `spark-submit` based submission covered by this
document:
the two approaches are complementary and the operator runs on top of the
same native
Kubernetes scheduler backend.
- Links to the operator repository and examples, consistent with the
existing links in
`docs/index.md`.
### Why are the changes needed?
`Apache Spark Kubernetes Operator` v1.0.0 is going to be released Tomorrow.
Currently, it is only mentioned in `docs/index.md`. Users who land directly on
`running-on-kubernetes.md`, the main documentation page for running Spark on
Kubernetes, have no way to discover the operator. This closes that
discoverability gap while keeping the detailed operator usage out of scope
(deferred to the operator repository).
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Manually check.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57512 from dongjoon-hyun/SPARK-58334.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 0700340fa6fc54f1f625b49503c205eab44394eb)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/running-on-kubernetes.md | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index b88a313861ca..e2b88fd9f2a4 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -80,6 +80,20 @@ driver and executor pods on a subset of available nodes
through a [node selector
using the configuration property for it. It will be possible to use more
advanced
scheduling hints like [node/pod
affinities](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity)
in a future release.
+# Spark Kubernetes Operator
+
+In addition to the `spark-submit` based submission described in this document,
users can deploy and
+manage Spark workloads declaratively via [operator
patterns](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
+by using [Apache Spark Kubernetes
Operator](https://github.com/apache/spark-kubernetes-operator),
+a separate Apache Spark project. The operator provides two custom resources:
+
+*
[SparkApp](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/pi-python.yaml):
deploy Spark apps on top of Kubernetes
+*
[SparkCluster](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/cluster.yaml):
deploy Spark clusters on top of Kubernetes
+
+The two approaches are complementary because the operator runs Spark
applications on top of the same
+native Kubernetes scheduler backend described in this document. Please refer
to the operator
+repository for its detailed documentation and usage.
+
# Submitting Applications to Kubernetes
## Docker Images
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
