Re: Spark 3.4.0 and 3.4.1 and Java version in Dockerfile

[Mich Talebzadeh]

Yes thanks, I know the answer. That was not what I was looking for. The
provided script should be working one way or another which is not. good
that someone has raised the issue already. that Jira was raised in Oct 2022

Mich Talebzadeh,
Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sat, 22 Jul 2023 at 18:29, Bjørn Jørgensen 
wrote:

> https://hub.docker.com/_/openjdk
> DEPRECATION NOTICE
>
> This image is officially deprecated and all users are recommended to find
> and use suitable replacements ASAP. Some examples of other Official Image
> alternatives (listed in alphabetical order with no intentional or implied
> preference):
>
>- amazoncorretto 
>- eclipse-temurin 
>- ibm-semeru-runtimes 
>- ibmjava 
>- sapmachine 
>
> See docker-library/openjdk#505
>  for more
> information.
>
>
> [SPARK-40941][K8S] Use Java 17 in K8s Dockerfile by default and remove
> Dockerfile.java17 
>
>
> lør. 22. juli 2023 kl. 17:33 skrev Mich Talebzadeh <
> mich.talebza...@gmail.com>:
>
>> Hi,
>>
>> I was checking  the contents of Dockerfile for JAVA in Spark directory,
>> .i.e
>>
>> ${SPARK_HOME}/kubernetes/dockerfiles/spark/Dockerfile
>>
>> in version 3.4.1
>>
>> I recall that in 3.4.0, I made adjustment to Dockerfile content replacing
>>
>> #ARG java_image_tag=17-jre
>> #FROM eclipse-temurin:${java_image_tag}
>>
>> with
>>
>>
>> *ARG java_image_tag=11-jre-slim*
>>
>> *FROM openjdk:${java_image_tag}*
>>
>> This worked and dockerfile was created
>>
>> With version 3.4.1, the same issue seems to exist
>>
>> Sending build context to Docker daemon  466.1MB
>> Step 1/18 : ARG java_image_tag=17-jre
>> Step 2/18 : FROM eclipse-temurin:${java_image_tag}
>> *manifest for eclipse-temurin:11-jre-slim not found: manifest unknown:
>> manifest unknown*
>> *Failed to build Spark JVM Docker image, please refer to Docker build
>> output for details.*
>>
>> Thanks
>>
>> Mich Talebzadeh,
>> Solutions Architect/Engineering Lead
>> Palantir Technologies Limited
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>

Re: Spark 3.4.0 and 3.4.1 and Java version in Dockerfile

[Bjørn Jørgensen]

https://hub.docker.com/_/openjdk
DEPRECATION NOTICE

This image is officially deprecated and all users are recommended to find
and use suitable replacements ASAP. Some examples of other Official Image
alternatives (listed in alphabetical order with no intentional or implied
preference):

   - amazoncorretto 
   - eclipse-temurin 
   - ibm-semeru-runtimes 
   - ibmjava 
   - sapmachine 

See docker-library/openjdk#505
 for more information.


[SPARK-40941][K8S] Use Java 17 in K8s Dockerfile by default and remove
Dockerfile.java17 


lør. 22. juli 2023 kl. 17:33 skrev Mich Talebzadeh <
mich.talebza...@gmail.com>:

> Hi,
>
> I was checking  the contents of Dockerfile for JAVA in Spark directory,
> .i.e
>
> ${SPARK_HOME}/kubernetes/dockerfiles/spark/Dockerfile
>
> in version 3.4.1
>
> I recall that in 3.4.0, I made adjustment to Dockerfile content replacing
>
> #ARG java_image_tag=17-jre
> #FROM eclipse-temurin:${java_image_tag}
>
> with
>
>
> *ARG java_image_tag=11-jre-slim*
>
> *FROM openjdk:${java_image_tag}*
>
> This worked and dockerfile was created
>
> With version 3.4.1, the same issue seems to exist
>
> Sending build context to Docker daemon  466.1MB
> Step 1/18 : ARG java_image_tag=17-jre
> Step 2/18 : FROM eclipse-temurin:${java_image_tag}
> *manifest for eclipse-temurin:11-jre-slim not found: manifest unknown:
> manifest unknown*
> *Failed to build Spark JVM Docker image, please refer to Docker build
> output for details.*
>
> Thanks
>
> Mich Talebzadeh,
> Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>


-- 
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge

+47 480 94 297

Spark 3.4.0 and 3.4.1 and Java version in Dockerfile

[Mich Talebzadeh]

Hi,

I was checking  the contents of Dockerfile for JAVA in Spark directory, .i.e

${SPARK_HOME}/kubernetes/dockerfiles/spark/Dockerfile

in version 3.4.1

I recall that in 3.4.0, I made adjustment to Dockerfile content replacing

#ARG java_image_tag=17-jre
#FROM eclipse-temurin:${java_image_tag}

with


*ARG java_image_tag=11-jre-slim*

*FROM openjdk:${java_image_tag}*

This worked and dockerfile was created

With version 3.4.1, the same issue seems to exist

Sending build context to Docker daemon  466.1MB
Step 1/18 : ARG java_image_tag=17-jre
Step 2/18 : FROM eclipse-temurin:${java_image_tag}
*manifest for eclipse-temurin:11-jre-slim not found: manifest unknown:
manifest unknown*
*Failed to build Spark JVM Docker image, please refer to Docker build
output for details.*

Thanks

Mich Talebzadeh,
Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: Spark Docker Official Image is now available

[Mich Talebzadeh]

Hi,

It helps if Spark binaries were added to PATH in the docker images. used to
be there in previous versions like 3.1.3 etc

docker run -it apache/spark:3.4.1-scala2.12-java11-r-ubuntu /bin/bash
spark@e48cc28ff89e:/opt/spark/work-dir$ which spark-submit
spark@e48cc28ff89e:/opt/spark/work-dir$
echo $PATH
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
spark@9440b0e46cee:/opt/spark/work-dir$ PATH=$PATH:/opt/spark/bin
spark@9440b0e46cee:/opt/spark/work-dir$ which spark-submit
/opt/spark/bin/spark-submit

HTH

Mich Talebzadeh,
Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 20 Jul 2023 at 02:10, Yikun Jiang  wrote:

> The spark Docker Official Image is now available:
> https://hub.docker.com/_/spark
>
> $ docker run -it --rm *spark* /opt/spark/bin/spark-shell
> $ docker run -it --rm *spark*:python3 /opt/spark/bin/pyspark
> $ docker run -it --rm *spark*:r /opt/spark/bin/sparkR
>
> We had a longer review journey than we expected, if you are also
> interested in this journey, you can see more in:
>
> https://github.com/docker-library/official-images/pull/13089
>
> Thanks to everyone who helps in the Docker and Apache Spark community!
>
> Some background you might want to know:
> *- apache/spark*: https://hub.docker.com/r/apache/spark, the Apache Spark
> docker image, will be published by *Apache Spark community* when the
> Apache Spark is released, no update.
> *- spark*: https://hub.docker.com/_/spark, the Docker Official Image, it
> will be published by the *Docker community*, keep active rebuilding for
> updates and security fixes by the Docker community.
> - The source repo of *apache/spark *and *spark: *
> https://github.com/apache/spark-docker
>
> See more in:
> [1] [DISCUSS] SPIP: Support Docker Official Image for Spark:
> https://lists.apache.org/thread/l1793y5224n8bqkp3s6ltgkykso4htb3
> [2] [VOTE] SPIP: Support Docker Official Image for Spark:
> https://lists.apache.org/thread/ro6olodm1jzdffwjx4oc7ol7oh6kshbl
> [3] https://github.com/docker-library/official-images/pull/13089
> [4]
> https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o/
> [5] https://issues.apache.org/jira/browse/SPARK-40513
>
> Regards,
> Yikun
>