Re: dockerhub does not contain apache/spark-py 3.4.1

[Mich Talebzadeh]

Hi Mark,

I created a spark3.4.1 docker file. Details from
spark-py-3.4.1-scala_2.12-11-jre-slim-buster


Pull instructions are given

docker pull
michtalebzadeh/spark_dockerfiles:spark-py-3.4.1-scala_2.12-11-jre-slim-buster

It is 3.4.1 spark-py with no extra python packages

you can tag it as you wish

login to it as below

docker run -it
michtalebzadeh/spark_dockerfiles:spark-py-3.4.1-scala_2.12-11-jre-slim-buster
bash

185@b031a15c6730:/opt/spark/work-dir$ pip list

Package   Version
- ---
asn1crypto0.24.0
cryptography  2.6.1
entrypoints   0.3
keyring   17.1.1
keyrings.alt  3.1.1
pip   23.2.1
pycrypto  2.6.1
PyGObject 3.30.4
pyxdg 0.25
SecretStorage 2.3.1
setuptools68.0.0
six   1.12.0
wheel 0.32.3

$SPARK_HOME/bin/spark-submit --version
Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.4.1
  /_/

Using Scala version 2.12.17, OpenJDK 64-Bit Server VM, 11.0.11
Branch HEAD
Compiled by user centos on 2023-06-19T23:01:01Z
Revision 6b1ff22dde1ead51cbf370be6e48a802daae58b6
Url https://github.com/apache/spark

Built on java 11

185@b031a15c6730:/opt/spark/work-dir$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment 18.9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9, mixed mode, sharing)

HTH


Mich Talebzadeh,
Solutions Architect/Engineering Lead
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 9 Aug 2023 at 17:41, Mich Talebzadeh 
wrote:

> Hi Mark,
>
> you can build it yourself, no big deal :)
>
> REPOSITORY TAG
> IMAGE ID   CREATED
>  SIZE
> sparkpy/spark-py
>  3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile a876102b2206   1
> second ago1.09GB
> sparkpy/spark
> 3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile 6f74f7475e01   3
> minutes ago   695MB
>
> Based on
>
> ARG java_image_tag=11-jre-slim  ## java 11
> FROM openjdk:${java_image_tag}
>
> BASE_OS="buster"
> SPARK_VERSION="3.4.1"
> SCALA_VERSION="scala_2.12"
> DOCKERFILE="Dockerfile"
> DOCKERIMAGETAG="11-jre-slim"
>
> You need to modify the file
>
> $SPARK_HOME/kubernetes/dockerfiles/spark/Dockerfile
>
> and replace
>
> #ARG java_image_tag=17-jre
> #FROM eclipse-temurin:${java_image_tag}
>
> With
>
> ARG java_image_tag=11-jre-slim
> FROM openjdk:${java_image_tag}
>
> Which is Java 11
>
> HTH
>
>
> Mich Talebzadeh,
> Solutions Architect/Engineering Lead
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 9 Aug 2023 at 16:43, Mark Elliot 
> wrote:
>
>> Hello,
>>
>> I noticed that the apache/spark-py image for Spark's 3.4.1 release is not
>> available (apache/spark@3.4.1 is available). Would it be possible to get
>> the 3.4.1 release build for the apache/spark-py image published?
>>
>> Thanks,
>>
>> Mark
>>
>> --
>>
>> This communication, together with any attachments, is intended only for
>> the addressee(s) and may contain confidential, privileged or proprietary
>> information of Theorem Partners LLC ("Theorem"). By accepting this
>> communication you agree to keep confidential all information contained in
>> this communication, as well as any information derived by you from the
>> confidential information contained in this communication. Theorem does not
>> waive any confidentiality by misdelivery.
>>
>> If you receive this communication in error, any use, dissemination,
>> printing or copying of all or any part of it is strictly prohibited; please
>> destroy all electronic and paper copies and notify the sender immediately.
>> Nothing in this email is intended to constitute (1) investment, legal or
>> tax advice, (2) any recommendation to purchase or sell any security, (3)
>> any advertisement or offer of advisory services or (4) any offer to sell or
>> solicitation of an offer to buy any 

Re: dockerhub does not contain apache/spark-py 3.4.1

[Mich Talebzadeh]

Hi Mark,

you can build it yourself, no big deal :)

REPOSITORY TAG
  IMAGE ID   CREATED
 SIZE
sparkpy/spark-py
 3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile a876102b2206   1
second ago1.09GB
sparkpy/spark
3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile 6f74f7475e01   3
minutes ago   695MB

Based on

ARG java_image_tag=11-jre-slim  ## java 11
FROM openjdk:${java_image_tag}

BASE_OS="buster"
SPARK_VERSION="3.4.1"
SCALA_VERSION="scala_2.12"
DOCKERFILE="Dockerfile"
DOCKERIMAGETAG="11-jre-slim"

You need to modify the file

$SPARK_HOME/kubernetes/dockerfiles/spark/Dockerfile

and replace

#ARG java_image_tag=17-jre
#FROM eclipse-temurin:${java_image_tag}

With

ARG java_image_tag=11-jre-slim
FROM openjdk:${java_image_tag}

Which is Java 11

HTH


Mich Talebzadeh,
Solutions Architect/Engineering Lead
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 9 Aug 2023 at 16:43, Mark Elliot  wrote:

> Hello,
>
> I noticed that the apache/spark-py image for Spark's 3.4.1 release is not
> available (apache/spark@3.4.1 is available). Would it be possible to get
> the 3.4.1 release build for the apache/spark-py image published?
>
> Thanks,
>
> Mark
>
> --
>
> This communication, together with any attachments, is intended only for
> the addressee(s) and may contain confidential, privileged or proprietary
> information of Theorem Partners LLC ("Theorem"). By accepting this
> communication you agree to keep confidential all information contained in
> this communication, as well as any information derived by you from the
> confidential information contained in this communication. Theorem does not
> waive any confidentiality by misdelivery.
>
> If you receive this communication in error, any use, dissemination,
> printing or copying of all or any part of it is strictly prohibited; please
> destroy all electronic and paper copies and notify the sender immediately.
> Nothing in this email is intended to constitute (1) investment, legal or
> tax advice, (2) any recommendation to purchase or sell any security, (3)
> any advertisement or offer of advisory services or (4) any offer to sell or
> solicitation of an offer to buy any securities or other financial
> instrument in any jurisdiction.
>
> Theorem, including its agents or affiliates, reserves the right to
> intercept, archive, monitor and review all communications to and from its
> network, including this email and any email response to it.
>
> Theorem makes no representation as to the accuracy or completeness of the
> information in this communication and does not accept liability for any
> errors or omissions in this communication, including any liability
> resulting from its transmission by email, and undertakes no obligation to
> update any information in this email or its attachments.
>