This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 40404fe546a6 [SPARK-55199][K8S][DOC] Improve K8s integration tests
`README`
40404fe546a6 is described below
commit 40404fe546a62508ec7f461622a760184ebe6db5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Jul 29 11:40:41 2026 -0700
[SPARK-55199][K8S][DOC] Improve K8s integration tests `README`
### What changes were proposed in this pull request?
This PR improves the K8s integration tests `README.md` to match the current
scripts and build:
- Document the default mode which builds Docker images directly from the
current source tree, and
remove the stale `TODO` which was already implemented.
- Add a new `Running the YuniKorn Integration Tests` section, symmetric to
the Volcano section.
- Document missing script flags (`--image-repo`, `--include-tags`,
`--exclude-tags`,
`--default-exclude-tags`, `--java-version`, `--hadoop-profile`,
`--skip-building-dependencies`)
and properties (`test.include.tags`, `test.exclude.tags`,
`test.default.exclude.tags`).
- Fix the default value of `spark.kubernetes.test.dockerFile` and remove an
outdated note.
### Why are the changes needed?
The README has drifted from the actual behavior. Accurate documentation
helps developers run the
K8s integration tests correctly.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57632 from dongjoon-hyun/SPARK-55199.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 8bee0297e94b1750c86aec76e69afd7145d6b279)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../kubernetes/integration-tests/README.md | 93 +++++++++++++++++++---
1 file changed, 83 insertions(+), 10 deletions(-)
diff --git a/resource-managers/kubernetes/integration-tests/README.md
b/resource-managers/kubernetes/integration-tests/README.md
index c8809dd6a516..20f4651d8618 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -5,9 +5,6 @@ title: Spark on Kubernetes Integration Tests
# Running the Kubernetes Integration Tests
-Note that the integration test framework is currently being heavily revised and
-is subject to change.
-
The simplest way to run the integration tests is to install and run Minikube,
then run the following from this
directory:
@@ -84,7 +81,11 @@ image tag that you have built by other means already, pass
the tag to the test s
where if you still want to use images that were built before by the test
framework:
./dev/dev-run-integration-tests.sh --image-tag $(cat target/imageTag.txt)
-
+
+The Docker image repository is customised via the `--image-repo <repo>` flag,
this is the repository
+that contains the images when a specific image tag is set, or to which freshly
built images are
+pushed. The default is `docker.io/kubespark`.
+
### Customising the Image Names
If your image names do not follow the standard Spark naming convention -
`spark`, `spark-py` and `spark-r` - then you can customise the names using
several options.
@@ -99,8 +100,10 @@ Alternatively if you use completely custom names then you
can set each individua
## Spark Distribution Under Test
-The Spark code to test is handed to the integration test system via a tarball.
Here is the option that is used to
-specify the tarball:
+By default, when neither `--spark-tgz` nor `--image-tag` is specified, the
Docker images are built
+directly from the current source tree, so no extra option is needed.
+
+Alternatively, the Spark code to test can be handed to the integration test
system via a tarball:
* `--spark-tgz <path-to-tgz>` - set `<path-to-tgz>` to point to a tarball
containing the Spark distribution to test.
@@ -110,9 +113,6 @@ runnable distribution please see the
[Building
Spark](https://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution)
documentation.
-**TODO:** Don't require the packaging of the built Spark artifacts into this
tarball, just read them out of the current
-tree.
-
## Customizing the Namespace and Service Account
If no namespace is specified then a temporary namespace will be created and
deleted during the test run. Similarly if
@@ -129,6 +129,24 @@ to get, list, watch, and create pods. For clusters with
RBAC turned on, it's imp
granted to the service account in the namespace through an appropriate role
and role binding. A reference RBAC
configuration is provided in `dev/spark-rbac.yaml`.
+## Selecting the Tests to Run
+
+The tests are tagged, and the set of tests to run can be controlled via the
following flags:
+
+* `--include-tags <tags>` - comma separated list of tags to include in
addition to the always
+ included `k8s` tag.
+* `--exclude-tags <tags>` - comma separated list of tags to exclude.
+* `--default-exclude-tags <tags>` - overrides the default list of excluded
tags, which is
+ `org.apache.spark.deploy.k8s.integrationtest.YuniKornTag`. Pass an empty
string to clear the
+ default exclusions, e.g. to run the YuniKorn tests.
+
+## Customising the Build
+
+* `--java-version <version>` - Java version to build Spark and the tests with.
The default is `17`.
+* `--hadoop-profile <profile>` - Hadoop Maven profile to build with. The
default is `hadoop-3`.
+* `--skip-building-dependencies` - skip rebuilding the modules the integration
tests depend on
+ (drops Maven's `-am` flag). Useful for iterating when the current tree is
already built.
+
# Running the Test Directly
If you prefer to run just the integration tests directly, then you can
customise the behaviour via passing system
@@ -251,7 +269,7 @@ to the wrapper scripts and using the wrapper scripts will
simply set these appro
<td>
The path to the custom Dockerfile
</td>
- <td><code>N/A</code></td>
+ <td><code>Dockerfile</code></td>
</tr>
<tr>
<td><code>spark.kubernetes.test.namespace</code></td>
@@ -294,6 +312,29 @@ to the wrapper scripts and using the wrapper scripts will
simply set these appro
</td>
<td></td>
</tr>
+ <tr>
+ <td><code>test.include.tags</code></td>
+ <td>
+ Comma separated list of test tags to include, e.g. <code>k8s</code>,
<code>minikube</code>,
+ <code>local</code>, <code>r</code>, <code>volcano</code>.
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td><code>test.exclude.tags</code></td>
+ <td>
+ Comma separated list of test tags to exclude.
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td><code>test.default.exclude.tags</code></td>
+ <td>
+ The default list of excluded test tags. Set to an empty string to clear
the default
+ exclusions, e.g. to run the YuniKorn tests.
+ </td>
+
<td><code>org.apache.spark.deploy.k8s.integrationtest.YuniKornTag</code></td>
+ </tr>
</table>
# Running the Kubernetes Integration Tests with SBT
@@ -363,3 +404,35 @@ You can also specify `volcano` tag to only run Volcano
test:
kubectl delete -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml
+# Running the YuniKorn Integration Tests
+
+`YuniKornSuite` runs the Kubernetes integration tests with the
+[Apache YuniKorn](https://yunikorn.apache.org/) scheduler instead of the
default scheduler.
+The suite is tagged with `YuniKornTag` which is excluded by default via
+`test.default.exclude.tags`.
+
+## Requirements
+
+- Apache YuniKorn 1.8.0.
+
+## Installation
+
+ helm repo add yunikorn https://apache.github.io/yunikorn-release
+ helm repo update
+ helm install yunikorn yunikorn/yunikorn --namespace yunikorn --version
1.8.0 \
+ --create-namespace --set embedAdmissionController=false
+
+## Run tests
+
+Clear the default excluded tags to enable the YuniKorn tests:
+
+ build/sbt -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dtest.default.exclude.tags= \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/testOnly *YuniKornSuite'
+
+## Cleanup YuniKorn
+
+ helm uninstall yunikorn --namespace yunikorn
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]