Hi bo How do we start?
Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc Thanks Sarath Sent from my iPhone > On Feb 23, 2022, at 10:27 AM, bo yang <bobyan...@gmail.com> wrote: > > > Hi Sarath, thanks for your interest and willing to contribute! The project > supports local development using MiniKube. Similarly there is a one click > command with one extra argument to deploy all components in MiniKube, and > people could use that to develop on their local MacBook. > > >> On Wed, Feb 23, 2022 at 7:41 AM Sarath Annareddy >> <sarath.annare...@gmail.com> wrote: >> Hi bo >> >> I am interested to contribute. >> But I don’t have free access to any cloud provider. Not sure how I can get >> free access. I know Google, aws, azure only provides temp free access, it >> may not be sufficient. >> >> Guidance is appreciated. >> >> Sarath >> >> Sent from my iPhone >> >>>> On Feb 23, 2022, at 2:01 AM, bo yang <bobyan...@gmail.com> wrote: >>>> >>> >> >>> Right, normally people start with simple script, then add more stuff, like >>> permission and more components. After some time, people want to run the >>> script consistently in different environments. Things will become complex. >>> >>> That is why we want to see whether people have interest for such a "one >>> click" tool to make things easy. >>> >>> >>>> On Tue, Feb 22, 2022 at 11:31 PM Mich Talebzadeh >>>> <mich.talebza...@gmail.com> wrote: >>>> Hi, >>>> >>>> There are two distinct actions here; namely Deploy and Run. >>>> >>>> Deployment can be done by command line script with autoscaling. In the >>>> newer versions of Kubernnetes you don't even need to specify the node >>>> types, you can leave it to the Kubernetes cluster to scale up and down >>>> and decide on node type. >>>> >>>> The second point is the running spark that you will need to submit. >>>> However, that depends on setting up access permission, use of service >>>> accounts, pulling the correct dockerfiles for the driver and the >>>> executors. Those details add to the complexity. >>>> >>>> Thanks >>>> >>>> >>>> view my Linkedin profile >>>> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> Disclaimer: Use it at your own risk. Any and all responsibility for any >>>> loss, damage or destruction of data or any other property which may arise >>>> from relying on this email's technical content is explicitly disclaimed. >>>> The author will in no case be liable for any monetary damages arising from >>>> such loss, damage or destruction. >>>> >>>> >>>> >>>>> On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: >>>>> Hi Spark Community, >>>>> >>>>> We built an open source tool to deploy and run Spark on Kubernetes with a >>>>> one click command. For example, on AWS, it could automatically create an >>>>> EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will >>>>> be able to use curl or a CLI tool to submit Spark application. After the >>>>> deployment, you could also install Uber Remote Shuffle Service to enable >>>>> Dynamic Allocation on Kuberentes. >>>>> >>>>> Anyone interested in using or working together on such a tool? >>>>> >>>>> Thanks, >>>>> Bo >>>>>
