Just a correction that in Spark 3.2 documentation it states <https://spark.apache.org/docs/latest/running-on-kubernetes.html#configuration> that
Property NameDefaultMeaning spark.kubernetes.container.image (none) Container image to use for the Spark application. This is usually of the form example.com/repo/spark:v1.0.0. This configuration is required and must be provided by the user, unless explicit images are provided for each different container type. 2.3.0 spark.kubernetes.driver.container.image (value of spark.kubernetes.container.image) Custom container image to use for the driver. 2.3.0 spark.kubernetes.executor.container.image (value of spark.kubernetes.container.image) Custom container image to use for executors. So both driver and executor images are mapped to the container image. In my opinion, they are redundant and will potentially add confusion so they should be removed? view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 8 Dec 2021 at 10:15, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi, > > We have three conf parameters to distribute the docker image with > spark-sumit in Kubernetes cluster. > > These are > > spark-submit --verbose \ > --conf spark.kubernetes.driver.docker.image=${IMAGEGCP} \ > --conf spark.kubernetes.executor.docker.image=${IMAGEGCP} \ > --conf spark.kubernetes.container.image=${IMAGEGCP} \ > > when the above is run, it shows > > (spark.kubernetes.driver.docker.image, > eu.gcr.io/axial-glow-224522/spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages > ) > (spark.kubernetes.executor.docker.image, > eu.gcr.io/axial-glow-224522/spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages > ) > (spark.kubernetes.container.image, > eu.gcr.io/axial-glow-224522/spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages > ) > > You notice that I am using the same docker image for driver, executor and > container. In Spark 3.2 (actually in recent spark versions), I cannot see > reference to driver or executor. Are these depreciated? It appears that > Spark still accepts them? > > Thanks > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > h > > > > > >
