Hi Sarath, thanks for your interest and willing to contribute! The project supports local development using MiniKube. Similarly there is a one click command with one extra argument to deploy all components in MiniKube, and people could use that to develop on their local MacBook.
On Wed, Feb 23, 2022 at 7:41 AM Sarath Annareddy <sarath.annare...@gmail.com> wrote: > Hi bo > > I am interested to contribute. > But I don’t have free access to any cloud provider. Not sure how I can get > free access. I know Google, aws, azure only provides temp free access, it > may not be sufficient. > > Guidance is appreciated. > > Sarath > > Sent from my iPhone > > On Feb 23, 2022, at 2:01 AM, bo yang <bobyan...@gmail.com> wrote: > > > > Right, normally people start with simple script, then add more stuff, like > permission and more components. After some time, people want to run the > script consistently in different environments. Things will become complex. > > That is why we want to see whether people have interest for such a "one > click" tool to make things easy. > > > On Tue, Feb 22, 2022 at 11:31 PM Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Hi, >> >> There are two distinct actions here; namely Deploy and Run. >> >> Deployment can be done by command line script with autoscaling. In the >> newer versions of Kubernnetes you don't even need to specify the node >> types, you can leave it to the Kubernetes cluster to scale up and down and >> decide on node type. >> >> The second point is the running spark that you will need to submit. >> However, that depends on setting up access permission, use of service >> accounts, pulling the correct dockerfiles for the driver and the executors. >> Those details add to the complexity. >> >> Thanks >> >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: >> >>> Hi Spark Community, >>> >>> We built an open source tool to deploy and run Spark on Kubernetes with >>> a one click command. For example, on AWS, it could automatically create an >>> EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will >>> be able to use curl or a CLI tool to submit Spark application. After the >>> deployment, you could also install Uber Remote Shuffle Service to enable >>> Dynamic Allocation on Kuberentes. >>> >>> Anyone interested in using or working together on such a tool? >>> >>> Thanks, >>> Bo >>> >>>
