jkpl opened a new pull request #28753:
URL: https://github.com/apache/spark/pull/28753
### What changes were proposed in this pull request?
Update the Fabric8 Kubernetes client to the latest release to support
Kubernetes version 1.18 (and 1.17) in Spark 2.4.x.
### Why are the changes needed?
Kubernetes 1.16 will fall out of support after the next Kubernetes release
(which is probably going to be released this summer), so Kubernetes cluster
maintainers will want to upgrade their clusters past it. However, the current
Spark 2.4.x version can't function in Kubernetes clusters with version higher
than 1.16 without upgrading the Kubernetes client.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Using standard `dev/run-tests` script and by executing a demo job in
Minikube. Steps for executing in Minikube:
1. Create a Minikube cluster.
2. Build the project using Maven with Kubernetes support.
3. Build the Docker images using `bin/docker-image-tool.sh` with `-m` flag.
4. Create a namespace and a service account for Spark.
5. Submit a Spark job in the cluster:
```
./bin/spark-submit \
--master "k8s://https://$(minikube ip):8443" \
--name demo \
--deploy-mode cluster \
--conf spark.kubernetes.driver.pod.name=demo-driver \
--conf spark.kubernetes.namespace=spark \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=spark-py:latest \
--conf spark.kubernetes.driver.limit.cores=2 \
--conf spark.kubernetes.container.image.pullPolicy=Never \
local:///opt/spark/examples/src/main/python/pi.py 10
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
