Re: Spark 3.4.0 and 3.4.1 and Java version in Dockerfile

2023-07-22 Thread Mich Talebzadeh
Yes thanks, I know the answer. That was not what I was looking for. The provided script should be working one way or another which is not. good that someone has raised the issue already. that Jira was raised in Oct 2022 Mich Talebzadeh, Solutions Architect/Engineering Lead Palantir Technologies

Re: Spark 3.4.0 and 3.4.1 and Java version in Dockerfile

2023-07-22 Thread Bjørn Jørgensen
https://hub.docker.com/_/openjdk DEPRECATION NOTICE This image is officially deprecated and all users are recommended to find and use suitable replacements ASAP. Some examples of other Official Image alternatives (listed in alphabetical order with no intentional or implied preference): -

Spark 3.4.0 and 3.4.1 and Java version in Dockerfile

2023-07-22 Thread Mich Talebzadeh
Hi, I was checking the contents of Dockerfile for JAVA in Spark directory, .i.e ${SPARK_HOME}/kubernetes/dockerfiles/spark/Dockerfile in version 3.4.1 I recall that in 3.4.0, I made adjustment to Dockerfile content replacing #ARG java_image_tag=17-jre #FROM eclipse-temurin:${java_image_tag}

Re: Spark Docker Official Image is now available

2023-07-22 Thread Mich Talebzadeh
Hi, It helps if Spark binaries were added to PATH in the docker images. used to be there in previous versions like 3.1.3 etc docker run -it apache/spark:3.4.1-scala2.12-java11-r-ubuntu /bin/bash spark@e48cc28ff89e:/opt/spark/work-dir$ which spark-submit spark@e48cc28ff89e:/opt/spark/work-dir$