This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.8
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.8 by this push:
new 3b2bbb6d7 [KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT
3b2bbb6d7 is described below
commit 3b2bbb6d7aec5c255c77af4a8e5eeb5f8dda1ea8
Author: zwangsheng <[email protected]>
AuthorDate: Tue Oct 24 16:14:20 2023 +0800
[KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT
### _Why are the changes needed?_
Improve Kyuubi On Kubernetes IT
Done:
1. Copy spark submit engine log in kyuubi pod to local and upload when test
failed.
2. pre install spark image into minikube to avoid image pull error
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes #5437 from zwangsheng/KYUUBI#5435.
Closes #5435
0cbbafce7 [zwangsheng] add comment
1f1336c59 [zwangsheng] ready
e1c10a6ea [zwangsheng] debug
32759015c [zwangsheng] debug
8e2f1eaf1 [zwangsheng] debug
80eaae30a [zwangsheng] [KYUUBI #5435][NOT_MERGE][TEST] Improve Kyuubi On
Kubernetes IT
Authored-by: zwangsheng <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 885b8189a1ce36afe88decdf01c9c5ccfd0d32b4)
Signed-off-by: Cheng Pan <[email protected]>
---
.github/workflows/master.yml | 7 +++++++
.../kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala | 2 ++
2 files changed, 9 insertions(+)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 7c442dd0f..68d4fd827 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -446,6 +446,9 @@ jobs:
minikube start --cpus 2 --memory 4096
--kubernetes-version=${KUBERNETES_VERSION} --force
#
https://minikube.sigs.k8s.io/docs/handbook/pushing/#7-loading-directly-to-in-cluster-container-runtime
minikube image load apache/kyuubi:latest
+ # pre-install spark into minikube
+ docker pull apache/spark:3.4.1
+ minikube image load apache/spark:3.4.1
- name: kubectl pre-check
run: |
kubectl get nodes
@@ -468,6 +471,9 @@ jobs:
- name: Cat kyuubi server log
if: failure()
run: kubectl logs kyuubi-test
+ - name: Copy spark engine log from kyuubi pod
+ if: failure()
+ run: kubectl cp kyuubi-test:/opt/kyuubi/work ./target/work
- name: Cat spark driver log
if: failure()
run: |
@@ -480,6 +486,7 @@ jobs:
name: unit-tests-log-kyuubi-on-k8s-it
path: |
**/target/unit-tests.log
+ **/target/work/**
spark-on-k8s-it:
name: Spark Engine On Kubernetes Integration Test
diff --git
a/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala
b/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala
index 73cb5620a..95e15e6eb 100644
---
a/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala
+++
b/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala
@@ -54,7 +54,9 @@ class KyuubiOnKubernetesWithSparkTestsBase extends
WithKyuubiServerOnKubernetes
super.connectionConf ++
Map(
"spark.master" -> s"k8s://$miniKubeApiMaster",
+ // We should update spark docker image in
./github/workflows/master.yml at the same time
"spark.kubernetes.container.image" -> "apache/spark:3.4.1",
+ "spark.kubernetes.container.image.pullPolicy" -> "IfNotPresent",
"spark.executor.memory" -> "512M",
"spark.driver.memory" -> "1024M",
"spark.kubernetes.driver.request.cores" -> "250m",
