This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 2c6c507 [SPARK-49658] Refactor e2e tests pipelines
2c6c507 is described below
commit 2c6c50757f94d639ad97445477fb1a022812dc0b
Author: Qi Tan <[email protected]>
AuthorDate: Mon Sep 16 21:24:45 2024 -0700
[SPARK-49658] Refactor e2e tests pipelines
### What changes were proposed in this pull request?
e2e pipelines refactor
### Why are the changes needed?
* Current helm installation does not require creation of additional
clusterrolebinding, serviceacount etc. Remove it.
* Current e2e pipeline run 3 times(because of the dimension of test-group)
run of dynamic operator installation and watched-namespaces tests for one k8s
version. Reduce it to 1 run only per k8s version.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
test locally
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #125 from TQJADE/workflow-fix.
Authored-by: Qi Tan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 25 ++++++++++++++++++++-----
1 file changed, 20 insertions(+), 5 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 851a09d..bb30476 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -69,12 +69,23 @@ jobs:
kubernetes-version:
- "1.28.0"
- "1.31.0"
+ mode:
+ - dynamic
+ - static
test-group:
- spark-versions
- python
- state-transition
- dynamic-config-test-group:
- watched-namespaces
+ exclude:
+ - mode: dynamic
+ test-group: spark-versions
+ - mode: dynamic
+ test-group: python
+ - mode: dynamic
+ test-group: state-transition
+ - mode: static
+ test-group: watched-namespaces
steps:
- name: Checkout repository
uses: actions/checkout@v4
@@ -101,8 +112,8 @@ jobs:
kubectl get pods -A
kubectl describe node
- name: Run Spark K8s Operator on K8S with Dynamic Configuration Disabled
+ if: matrix.mode == 'static'
run: |
- kubectl create clusterrolebinding serviceaccounts-cluster-admin
--clusterrole=cluster-admin --group=system:serviceaccounts || true
eval $(minikube docker-env)
./gradlew buildDockerImage
./gradlew spark-operator-api:relocateGeneratedCRD
@@ -111,20 +122,24 @@ jobs:
# Use remote host' s docker image
minikube docker-env --unset
- name: Run E2E Test with Dynamic Configuration Disabled
- run: |
+ if: matrix.mode == 'static'
+ run: |
chainsaw test --test-dir ./tests/e2e/${{ matrix.test-group }}
--parallel 2
- name: Run Spark K8s Operator on K8S with Dynamic Configuration Enabled
+ if: matrix.mode == 'dynamic'
run: |
- helm uninstall spark-kubernetes-operator
eval $(minikube docker-env)
+ ./gradlew buildDockerImage
+ ./gradlew spark-operator-api:relocateGeneratedCRD
helm install spark-kubernetes-operator --create-namespace -f \
build-tools/helm/spark-kubernetes-operator/values.yaml -f \
tests/e2e/helm/dynamic-config-values.yaml \
build-tools/helm/spark-kubernetes-operator/
minikube docker-env --unset
- name: Run E2E Test with Dynamic Configuration Enabled
+ if: matrix.mode == 'dynamic'
run: |
- chainsaw test --test-dir ./tests/e2e/${{
matrix.dynamic-config-test-group }} --parallel 2
+ chainsaw test --test-dir ./tests/e2e/${{ matrix.test-group }}
--parallel 2
lint:
name: "Linter and documentation"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
