This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 8bee0297e94b [SPARK-55199][K8S][DOC] Improve K8s integration tests 
`README`
8bee0297e94b is described below

commit 8bee0297e94b1750c86aec76e69afd7145d6b279
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Jul 29 11:40:41 2026 -0700

    [SPARK-55199][K8S][DOC] Improve K8s integration tests `README`
    
    ### What changes were proposed in this pull request?
    
    This PR improves the K8s integration tests `README.md` to match the current 
scripts and build:
    
    - Document the default mode which builds Docker images directly from the 
current source tree, and
      remove the stale `TODO` which was already implemented.
    - Add a new `Running the YuniKorn Integration Tests` section, symmetric to 
the Volcano section.
    - Document missing script flags (`--image-repo`, `--include-tags`, 
`--exclude-tags`,
      `--default-exclude-tags`, `--java-version`, `--hadoop-profile`, 
`--skip-building-dependencies`)
      and properties (`test.include.tags`, `test.exclude.tags`, 
`test.default.exclude.tags`).
    - Fix the default value of `spark.kubernetes.test.dockerFile` and remove an 
outdated note.
    
    ### Why are the changes needed?
    
    The README has drifted from the actual behavior. Accurate documentation 
helps developers run the
    K8s integration tests correctly.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a documentation-only change.
    
    ### How was this patch tested?
    
    Manual review.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #57632 from dongjoon-hyun/SPARK-55199.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../kubernetes/integration-tests/README.md         | 93 +++++++++++++++++++---
 1 file changed, 83 insertions(+), 10 deletions(-)

diff --git a/resource-managers/kubernetes/integration-tests/README.md 
b/resource-managers/kubernetes/integration-tests/README.md
index dae6cbb58ab7..35c0356acddf 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -5,9 +5,6 @@ title: Spark on Kubernetes Integration Tests
 
 # Running the Kubernetes Integration Tests
 
-Note that the integration test framework is currently being heavily revised and
-is subject to change.
-
 The simplest way to run the integration tests is to install and run Minikube, 
then run the following from this
 directory:
 
@@ -84,7 +81,11 @@ image tag that you have built by other means already, pass 
the tag to the test s
 where if you still want to use images that were built before by the test 
framework:
 
     ./dev/dev-run-integration-tests.sh --image-tag $(cat target/imageTag.txt)
-    
+
+The Docker image repository is customised via the `--image-repo <repo>` flag, 
this is the repository
+that contains the images when a specific image tag is set, or to which freshly 
built images are
+pushed. The default is `docker.io/kubespark`.
+
 ### Customising the Image Names
 
 If your image names do not follow the standard Spark naming convention - 
`spark`, `spark-py` and `spark-r` - then you can customise the names using 
several options.
@@ -99,8 +100,10 @@ Alternatively if you use completely custom names then you 
can set each individua
 
 ## Spark Distribution Under Test
 
-The Spark code to test is handed to the integration test system via a tarball. 
Here is the option that is used to 
-specify the tarball:
+By default, when neither `--spark-tgz` nor `--image-tag` is specified, the 
Docker images are built
+directly from the current source tree, so no extra option is needed.
+
+Alternatively, the Spark code to test can be handed to the integration test 
system via a tarball:
 
 * `--spark-tgz <path-to-tgz>` - set `<path-to-tgz>` to point to a tarball 
containing the Spark distribution to test.
 
@@ -110,9 +113,6 @@ runnable distribution please see the
 [Building 
Spark](https://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution)
 
 documentation.
 
-**TODO:** Don't require the packaging of the built Spark artifacts into this 
tarball, just read them out of the current 
-tree.
-
 ## Customizing the Namespace and Service Account
 
 If no namespace is specified then a temporary namespace will be created and 
deleted during the test run.  Similarly if 
@@ -129,6 +129,24 @@ to get, list, watch, and create pods. For clusters with 
RBAC turned on, it's imp
 granted to the service account in the namespace through an appropriate role 
and role binding. A reference RBAC 
 configuration is provided in `dev/spark-rbac.yaml`.
 
+## Selecting the Tests to Run
+
+The tests are tagged, and the set of tests to run can be controlled via the 
following flags:
+
+* `--include-tags <tags>` - comma separated list of tags to include in 
addition to the always
+  included `k8s` tag.
+* `--exclude-tags <tags>` - comma separated list of tags to exclude.
+* `--default-exclude-tags <tags>` - overrides the default list of excluded 
tags, which is
+  `org.apache.spark.deploy.k8s.integrationtest.YuniKornTag`. Pass an empty 
string to clear the
+  default exclusions, e.g. to run the YuniKorn tests.
+
+## Customising the Build
+
+* `--java-version <version>` - Java version to build Spark and the tests with. 
The default is `17`.
+* `--hadoop-profile <profile>` - Hadoop Maven profile to build with. The 
default is `hadoop-3`.
+* `--skip-building-dependencies` - skip rebuilding the modules the integration 
tests depend on
+  (drops Maven's `-am` flag). Useful for iterating when the current tree is 
already built.
+
 # Running the Test Directly
 
 If you prefer to run just the integration tests directly, then you can 
customise the behaviour via passing system 
@@ -251,7 +269,7 @@ to the wrapper scripts and using the wrapper scripts will 
simply set these appro
     <td>
       The path to the custom Dockerfile
     </td>
-    <td><code>N/A</code></td>
+    <td><code>Dockerfile</code></td>
   </tr>
   <tr>
     <td><code>spark.kubernetes.test.namespace</code></td>
@@ -294,6 +312,29 @@ to the wrapper scripts and using the wrapper scripts will 
simply set these appro
     </td>
     <td></td>
   </tr>
+  <tr>
+    <td><code>test.include.tags</code></td>
+    <td>
+      Comma separated list of test tags to include, e.g. <code>k8s</code>, 
<code>minikube</code>,
+      <code>local</code>, <code>r</code>, <code>volcano</code>.
+    </td>
+    <td></td>
+  </tr>
+  <tr>
+    <td><code>test.exclude.tags</code></td>
+    <td>
+      Comma separated list of test tags to exclude.
+    </td>
+    <td></td>
+  </tr>
+  <tr>
+    <td><code>test.default.exclude.tags</code></td>
+    <td>
+      The default list of excluded test tags.  Set to an empty string to clear 
the default
+      exclusions, e.g. to run the YuniKorn tests.
+    </td>
+    
<td><code>org.apache.spark.deploy.k8s.integrationtest.YuniKornTag</code></td>
+  </tr>
 </table>
 
 # Running the Kubernetes Integration Tests with SBT
@@ -363,3 +404,35 @@ You can also specify `volcano` tag to only run Volcano 
test:
 
     kubectl delete -f 
https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml
 
+# Running the YuniKorn Integration Tests
+
+`YuniKornSuite` runs the Kubernetes integration tests with the
+[Apache YuniKorn](https://yunikorn.apache.org/) scheduler instead of the 
default scheduler.
+The suite is tagged with `YuniKornTag` which is excluded by default via
+`test.default.exclude.tags`.
+
+## Requirements
+
+- Apache YuniKorn 1.8.0.
+
+## Installation
+
+    helm repo add yunikorn https://apache.github.io/yunikorn-release
+    helm repo update
+    helm install yunikorn yunikorn/yunikorn --namespace yunikorn --version 
1.8.0 \
+        --create-namespace --set embedAdmissionController=false
+
+## Run tests
+
+Clear the default excluded tags to enable the YuniKorn tests:
+
+    build/sbt -Pkubernetes -Pkubernetes-integration-tests \
+        -Dtest.exclude.tags=minikube \
+        -Dtest.default.exclude.tags= \
+        -Dspark.kubernetes.test.deployMode=docker-desktop \
+        'kubernetes-integration-tests/testOnly *YuniKornSuite'
+
+## Cleanup YuniKorn
+
+    helm uninstall yunikorn --namespace yunikorn
+


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to