Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

Thank you, Yikun. I am working on SPARK-37809 and SPARK-38310 . They are the major stuff for the yunikorn part. Keep in mind we also need to add the documents. Thanks for building up the common

Re: One click to run Spark on Kubernetes

It uses Helm to deploy Spark Operator and Nginx. For other parts like creating EKS, IAM role, node group, etc, it uses AWS SDK to provision those AWS resources. On Wed, Feb 23, 2022 at 11:28 AM Bjørn Jørgensen wrote: > So if I get this right you will make a Helm chart to >

Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

First, much thanks for all your help (Spark/Volcano/Yunikorn community) to make this SPIP happen! Especially,@dongjoon-hyun @holdenk @william-wang @attilapiros @HyukjinKwon @martin-g @yangwwei @tgravescs The SPIP is near the end of the stage. It can be said that it is beta available at the basic

RE: [Fork] ]RE: One click to run Spark on Kubernetes

Mich, Not sure I follow you since I do not fully understand what GKE conventional is (which at first glance, appears to help customers to setup Kubernetes environment). EMR on EKS offers a fully managed control plane (among other benefits such as Spark UI for completed jobs) that allows

Re: One click to run Spark on Kubernetes

Right, normally people start with simple script, then add more stuff, like permission and more components. After some time, people want to run the script consistently in different environments. Things will become complex. That is why we want to see whether people have interest for such a "one

Spark 3.1.3 docker pre-built with Python Data science packages

Some people asked me whether it was possible to create a docker file (spark 3.1.3) with Python packages geared towards DS etc., having the following pre-built packages pyyaml TensorFlow Theano Pandas Keras NumPy SciPy Scrapy SciKit-Learn XGBoost Matplotlib Seaborn Bokeh Plotly pydot Statsmodels

Re: One click to run Spark on Kubernetes

Hi bo I am interested to contribute. But I don’t have free access to any cloud provider. Not sure how I can get free access. I know Google, aws, azure only provides temp free access, it may not be sufficient. Guidance is appreciated. Sarath Sent from my iPhone > On Feb 23, 2022, at 2:01

Re: [VOTE] SPIP: Catalog API for view metadata

Holden has graciously agreed to shepherd the SPIP. Thanks! On Thu, Feb 10, 2022 at 9:19 AM John Zhuge wrote: > The vote is now closed and the vote passes. Thank you to everyone who took > the time to review and vote on this SPIP. I’m looking forward to adding > this feature to the next Spark

Re: One click to run Spark on Kubernetes

Hi Sarath, let's follow up offline on this. On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy wrote: > Hi bo > > How do we start? > > Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc > > > Thanks > Sarath > > > Sent from my iPhone > > On Feb 23, 2022, at 10:27 AM, bo yang

Re: One click to run Spark on Kubernetes

So if I get this right you will make a Helm chart to deploy Spark and some other stuff on K8S? ons. 23. feb. 2022 kl. 17:49 skrev bo yang : > Hi Sarath, let's follow up offline on this. > > On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy < > sarath.annare...@gmail.com> wrote:

Re: [Fork] ]RE: One click to run Spark on Kubernetes

Thanks Janak, the same as GKE conventional or GKE autopilot. Putting conventional aside, why do you think customers should choose a fully managed package* for Spark*? thanks view my Linkedin profile

Re: One click to run Spark on Kubernetes

Hi Sarath, thanks for your interest and willing to contribute! The project supports local development using MiniKube. Similarly there is a one click command with one extra argument to deploy all components in MiniKube, and people could use that to develop on their local MacBook. On Wed, Feb 23,

Re: One click to run Spark on Kubernetes

Hi bo How do we start? Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc Thanks Sarath Sent from my iPhone > On Feb 23, 2022, at 10:27 AM, bo yang wrote: > >  > Hi Sarath, thanks for your interest and willing to contribute! The project > supports local development

Re: [Fork] ]RE: One click to run Spark on Kubernetes

Hi Janak, Are you talking about EKS Fargate? Thanks view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction

RE: [Fork] ]RE: One click to run Spark on Kubernetes

Hey Mich, EMR on EKS works on both EKS-Fargate and EKS-managed/self-managed EC2 based node groups. Thanks, Janak From: Mich Talebzadeh Sent: Wednesday, February 23, 2022 10:46 AM To: Agarwal, Janak Cc: Spark dev list Subject: RE: [EXTERNAL] [Fork]