This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 08a88e0 [SPARK-31313][K8S][TEST] Add `m01` node name to support
Minikube 1.8.x
08a88e0 is described below
commit 08a88e04f4417c5a0168592d1f01c300b4ae9de5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Apr 1 03:42:26 2020 +0000
[SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x
### What changes were proposed in this pull request?
This PR aims to add `m01` as a node name additionally to `PVTestsSuite`.
### Why are the changes needed?
minikube 1.8.0 ~ 1.8.2 generate a cluster with a nodename `m01` while all
the other versions have `minikube`. This causes `PVTestSuite` failure.
```
$ minikube --vm-driver=hyperkit start --memory 6000 --cpus 8
* minikube v1.8.2 on Darwin 10.15.3
- MINIKUBE_ACTIVE_DOCKERD=minikube
* Using the hyperkit driver based on user configuration
* Creating hyperkit VM (CPUs=8, Memory=6000MB, Disk=20000MB) ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.6 ...
* Launching Kubernetes ...
* Enabling addons: default-storageclass, storage-provisioner
* Waiting for cluster to come online ...
* Done! kubectl is now configured to use "minikube"
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
m01 Ready master 22s v1.17.3
```
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
This only adds a new node name. So, K8S Jenkins job should passed.
In addition, `K8s` integration test suite should be tested on `minikube
1.8.2` manually.
```
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
- Run SparkR on simple dataframe.R example
Run completed in 10 minutes, 23 seconds.
Total number of tests run: 20
Suites: completed 2, aborted 0
Tests: succeeded 20, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
For the above test, Minikube 1.8.2 and K8s v1.18.0 is used.
```
$ minikube version
minikube version: v1.8.2
commit: eb13446e786c9ef70cb0a9f85a633194e62396a1
$ kubectl version --short
Client Version: v1.18.0
Server Version: v1.18.0
```
Closes #28080 from dongjoon-hyun/SPARK-31313.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
(cherry picked from commit dba525c997b0033ac1b6fd24236cd72938f94bbf)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
index f021821..a7cb84e 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
@@ -56,7 +56,8 @@ private[spark] trait PVTestsSuite { k8sSuite: KubernetesSuite
=>
.withMatchExpressions(new NodeSelectorRequirementBuilder()
.withKey("kubernetes.io/hostname")
.withOperator("In")
- .withValues("minikube", "docker-for-desktop",
"docker-desktop").build()).build())
+ .withValues("minikube", "m01", "docker-for-desktop",
"docker-desktop")
+ .build()).build())
.endRequired()
.endNodeAffinity()
.endSpec()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
