Hi guys,
I try to use Spark 3 on top of Kubernetes and to specify a pod template for
the driver.
Here is my pod manifest or the driver and when I do a spark-submit with the
option:
--conf
spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml
I got the error message that I need to specify an image, but it's the
manifest.
Does my manifest file is wrong, How should it look like?
Thanks for your help,
Michel
--------
The pod manifest:
apiVersion: v1
kind: Pod
metadata:
name: mySpark3App
labels:
app: mySpark3App
customlabel/app-id: "1"
spec:
securityContext:
runAsUser: 1000
volumes:
- name: "test-volume"
emptyDir: {}
containers:
- name: spark3driver
image: mydockerregistry.example.com/images/dev/spark3:latest
instances: 1
resources:
requests:
cpu: "1000m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "512Mi"
volumeMounts:
- name: "test-volume"
mountPath: "/tmp"