So if I get this right you will make a Helm <https://helm.sh> chart to deploy Spark and some other stuff on K8S?
ons. 23. feb. 2022 kl. 17:49 skrev bo yang <bobyan...@gmail.com>: > Hi Sarath, let's follow up offline on this. > > On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy < > sarath.annare...@gmail.com> wrote: > >> Hi bo >> >> How do we start? >> >> Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc >> >> >> Thanks >> Sarath >> >> >> Sent from my iPhone >> >> On Feb 23, 2022, at 10:27 AM, bo yang <bobyan...@gmail.com> wrote: >> >> >> Hi Sarath, thanks for your interest and willing to contribute! The >> project supports local development using MiniKube. Similarly there is a one >> click command with one extra argument to deploy all components in MiniKube, >> and people could use that to develop on their local MacBook. >> >> >> On Wed, Feb 23, 2022 at 7:41 AM Sarath Annareddy < >> sarath.annare...@gmail.com> wrote: >> >>> Hi bo >>> >>> I am interested to contribute. >>> But I don’t have free access to any cloud provider. Not sure how I can >>> get free access. I know Google, aws, azure only provides temp free access, >>> it may not be sufficient. >>> >>> Guidance is appreciated. >>> >>> Sarath >>> >>> Sent from my iPhone >>> >>> On Feb 23, 2022, at 2:01 AM, bo yang <bobyan...@gmail.com> wrote: >>> >>> >>> >>> Right, normally people start with simple script, then add more stuff, >>> like permission and more components. After some time, people want to run >>> the script consistently in different environments. Things will become >>> complex. >>> >>> That is why we want to see whether people have interest for such a "one >>> click" tool to make things easy. >>> >>> >>> On Tue, Feb 22, 2022 at 11:31 PM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> There are two distinct actions here; namely Deploy and Run. >>>> >>>> Deployment can be done by command line script with autoscaling. In the >>>> newer versions of Kubernnetes you don't even need to specify the node >>>> types, you can leave it to the Kubernetes cluster to scale up and down and >>>> decide on node type. >>>> >>>> The second point is the running spark that you will need to submit. >>>> However, that depends on setting up access permission, use of service >>>> accounts, pulling the correct dockerfiles for the driver and the executors. >>>> Those details add to the complexity. >>>> >>>> Thanks >>>> >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: >>>> >>>>> Hi Spark Community, >>>>> >>>>> We built an open source tool to deploy and run Spark on Kubernetes >>>>> with a one click command. For example, on AWS, it could automatically >>>>> create an EKS cluster, node group, NGINX ingress, and Spark Operator. Then >>>>> you will be able to use curl or a CLI tool to submit Spark application. >>>>> After the deployment, you could also install Uber Remote Shuffle Service >>>>> to >>>>> enable Dynamic Allocation on Kuberentes. >>>>> >>>>> Anyone interested in using or working together on such a tool? >>>>> >>>>> Thanks, >>>>> Bo >>>>> >>>>> -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
