Re: One click to run Spark on Kubernetes

It uses Helm to deploy Spark Operator and Nginx. For other parts like creating EKS, IAM role, node group, etc, it uses AWS SDK to provision those AWS resources. On Wed, Feb 23, 2022 at 11:28 AM Bjørn Jørgensen wrote: > So if I get this right you will make a Helm chart to >

Re: One click to run Spark on Kubernetes

So if I get this right you will make a Helm chart to deploy Spark and some other stuff on K8S? ons. 23. feb. 2022 kl. 17:49 skrev bo yang : > Hi Sarath, let's follow up offline on this. > > On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy < > sarath.annare...@gmail.com> wrote:

Re: One click to run Spark on Kubernetes

Hi Sarath, let's follow up offline on this. On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy wrote: > Hi bo > > How do we start? > > Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc > > > Thanks > Sarath > > > Sent from my iPhone > > On Feb 23, 2022, at 10:27 AM, bo yang

Re: One click to run Spark on Kubernetes

Hi bo How do we start? Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc Thanks Sarath Sent from my iPhone > On Feb 23, 2022, at 10:27 AM, bo yang wrote: > >  > Hi Sarath, thanks for your interest and willing to contribute! The project > supports local development

Re: One click to run Spark on Kubernetes

Hi Sarath, thanks for your interest and willing to contribute! The project supports local development using MiniKube. Similarly there is a one click command with one extra argument to deploy all components in MiniKube, and people could use that to develop on their local MacBook. On Wed, Feb 23,

Re: One click to run Spark on Kubernetes

Hi bo I am interested to contribute. But I don’t have free access to any cloud provider. Not sure how I can get free access. I know Google, aws, azure only provides temp free access, it may not be sufficient. Guidance is appreciated. Sarath Sent from my iPhone > On Feb 23, 2022, at 2:01

Re: One click to run Spark on Kubernetes

from my viewpoints, if there is such a pay as you go service I would like to use. otherwise I have to deploy a regular spark cluster with GCP/AWS etc and the cost is not low. Thanks. On Wed, Feb 23, 2022 at 4:00 PM bo yang wrote: > Right, normally people start with simple script, then add more

Re: One click to run Spark on Kubernetes

Right, normally people start with simple script, then add more stuff, like permission and more components. After some time, people want to run the script consistently in different environments. Things will become complex. That is why we want to see whether people have interest for such a "one

Re: One click to run Spark on Kubernetes

Hi, There are two distinct actions here; namely Deploy and Run. Deployment can be done by command line script with autoscaling. In the newer versions of Kubernnetes you don't even need to specify the node types, you can leave it to the Kubernetes cluster to scale up and down and decide on node

Re: One click to run Spark on Kubernetes

Merging another email from Prasad. It could co-exist with livy. Livy is similar like the REST Service + Spark Operator. Unfortunately Livy is not very active right now. To Amihay, the link is: https://github.com/datapunchorg/punch. On Tue, Feb 22, 2022 at 8:53 PM amihay gonen wrote: > Can you

Re: One click to run Spark on Kubernetes

Can you share link to the source? בתאריך יום ד׳, 23 בפבר׳ 2022, 6:52, מאת bo yang ‏: > We do not have SaaS yet. Now it is an open source project we build in our > part time , and we welcome more people working together on that. > > You could specify cluster size (EC2 instance type and number of

Re: One click to run Spark on Kubernetes

We do not have SaaS yet. Now it is an open source project we build in our part time , and we welcome more people working together on that. You could specify cluster size (EC2 instance type and number of instances) and run it for 1 hour. Then you could run one click command to destroy the cluster.

Re: One click to run Spark on Kubernetes

Hi Bo Yang, Would it be something along the lines of Apache livy? Thanks, Prasad On Tue, Feb 22, 2022 at 10:22 PM bo yang wrote: > It is not a standalone spark cluster. In some details, it deploys a Spark > Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) > and an extra

Re: One click to run Spark on Kubernetes

How can I specify the cluster memory and cores? For instance, I want to run a job with 16 cores and 300 GB memory for about 1 hour. Do you have the SaaS solution for this? I can pay as I did. Thanks On Wed, Feb 23, 2022 at 12:21 PM bo yang wrote: > It is not a standalone spark cluster. In some

Re: One click to run Spark on Kubernetes

It is not a standalone spark cluster. In some details, it deploys a Spark Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and an extra REST Service. When people submit Spark application to that REST Service, the REST Service will create a CRD inside the Kubernetes cluster.

Re: One click to run Spark on Kubernetes

Can it be a cluster installation of spark? or just the standalone node? Thanks On Wed, Feb 23, 2022 at 12:06 PM bo yang wrote: > Hi Spark Community, > > We built an open source tool to deploy and run Spark on Kubernetes with a > one click command. For example, on AWS, it could automatically