Re: One click to run Spark on Kubernetes

ngress, and Spark Operator. Then you will >>> be able to use curl or a CLI tool to submit Spark application. After the >>> deployment, you could also install Uber Remote Shuffle Service to enable >>> Dynamic Allocation on Kuberentes. >>> >>> Anyone interested in using or working together on such a tool? >>> >>> Thanks, >>> Bo >>> >>> -- Regards, Prasad Paravatha

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

Apologies, please ignore my previous message On Mon, Feb 21, 2022 at 5:56 PM Prasad Paravatha wrote: > FYI, I am getting 404 for https://hub.docker.com/apache/spark > > On Mon, Feb 21, 2022 at 5:51 PM Holden Karau wrote: > >> Yeah I think we should still adopt that naming c

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

rs to the Apache >>>>> Dockerhub, >>>>> these contain non-ASF artifacts that are subject to different license >>>>> terms than the >>>>> Spark release. The docker containers are built for Linux x86 and ARM64 >>>>> since that's >>>>> what I have access to (thanks to NV for the ARM64 machines). >>>>> >>>>> You can get them from https://hub.docker.com/apache/spark (and >>>>> spark-r and spark-py) :) >>>>> (And version 3.2.1 is also now published on Dockerhub). >>>>> >>>>> Holden >>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> >>>> >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > -- Regards, Prasad Paravatha

Re: docker image distribution in Kubernetes cluster

iver.docker.image=${IMAGEGCP} \ >> >>--conf spark.kubernetes.executor.docker.image=${IMAGEGCP} \ >> >>--conf spark.kubernetes.container.image=${IMAGEGCP} \ >> >> >> >> when the above is run, it shows >> >> >> >> (spark.kubernetes.driver.docker.image, >> eu.gcr.io/axial-glow-224522/spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages >> ) >> >> (spark.kubernetes.executor.docker.image, >> eu.gcr.io/axial-glow-224522/spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages >> ) >> >> (spark.kubernetes.container.image, >> eu.gcr.io/axial-glow-224522/spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages >> ) >> >> >> >> You notice that I am using the same docker image for driver, executor and >> container. In Spark 3.2 (actually in recent spark versions), I cannot see >> reference to driver or executor. Are these depreciated? It appears that >> Spark still accepts them? >> >> >> >> Thanks >> >> >> >> >> >> >> [image: Image removed by sender.] view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> h >> >> >> >> >> >> >> >> >> >> -- Regards, Prasad Paravatha

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

RA: https://issues.apache.org/jira/browse/SPARK-36057 >>>> Part of PRs: >>>> Ability to create resources https://github.com/apache/spark/pull/34599 >>>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456 >>>> >>>> Regards, >>>> Yikun >>>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> > > -- > John Zhuge > -- Regards, Prasad Paravatha

Re: [ANNOUNCE] Apache Spark 3.2.0

fixed now. > Could you try again on https://spark.apache.org/downloads.html? > >> On Tue, Oct 19, 2021 at 10:53 PM Prasad Paravatha >> wrote: >> https://www.apache.org/dyn/closer.lua/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.3.tgz >> >> FYI, unable to downloa

Re: [ANNOUNCE] Apache Spark 3.2.0

https://www.apache.org/dyn/closer.lua/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.3.tgz FYI, unable to download from this location. Also, I don’t see Hadoop 3.3 version in the dist > On Oct 19, 2021, at 9:39 AM, Bode, Meikel, NMA-CFD > wrote: > >  > Many thanks!  > > From: Gengliang Wang