from my viewpoints, if there is such a pay as you go service I would like to use. otherwise I have to deploy a regular spark cluster with GCP/AWS etc and the cost is not low.
Thanks. On Wed, Feb 23, 2022 at 4:00 PM bo yang <bobyan...@gmail.com> wrote: > Right, normally people start with simple script, then add more stuff, like > permission and more components. After some time, people want to run the > script consistently in different environments. Things will become complex. > > That is why we want to see whether people have interest for such a "one > click" tool to make things easy. > > > On Tue, Feb 22, 2022 at 11:31 PM Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Hi, >> >> There are two distinct actions here; namely Deploy and Run. >> >> Deployment can be done by command line script with autoscaling. In the >> newer versions of Kubernnetes you don't even need to specify the node >> types, you can leave it to the Kubernetes cluster to scale up and down and >> decide on node type. >> >> The second point is the running spark that you will need to submit. >> However, that depends on setting up access permission, use of service >> accounts, pulling the correct dockerfiles for the driver and the executors. >> Those details add to the complexity. >> >> Thanks >> >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: >> >>> Hi Spark Community, >>> >>> We built an open source tool to deploy and run Spark on Kubernetes with >>> a one click command. For example, on AWS, it could automatically create an >>> EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will >>> be able to use curl or a CLI tool to submit Spark application. After the >>> deployment, you could also install Uber Remote Shuffle Service to enable >>> Dynamic Allocation on Kuberentes. >>> >>> Anyone interested in using or working together on such a tool? >>> >>> Thanks, >>> Bo >>> >>>
