Thanks. I also have a three node cluster in my lab running Red Hat 7.6 with 64GB of RAM etc. However, I doubt whether minikube will be useful.
If we can get a Google Kubernetes Engine <https://cloud.google.com/kubernetes-engine>(GKE) cluster (which is a fully managed service) from Google on a loan, then it will be great. That will take out the hassle of setting up the K8 cluster manually and dealing with compatibility issues further down the line. To take this further, I would like to suggest having a discussion here with Klaus Mao and the other colleagues who represent the Volcano project on the best way of progressing on this. I am a Google Advantage partner so I can put such an agreed proposal to the account manager and ask whether Google will agree to support this R &D work (which BTW I think would be beneficial to both parties) as Google started Kubernetes themselves. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Thu, 1 Jul 2021 at 21:04, Holden Karau <hol...@pigscanfly.ca> wrote: > I do my own dev work on a personal cluster I have down in Fremont which > I’ve got setup using k3sup. I know some devs use minikube (and our > integration tests can). But yeah if there was a vendor willing to hand out > Kube resources that could simplify our dev cycles. > > On Thu, Jul 1, 2021 at 12:52 PM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Hi, >> >> A rather simple question. >> >> As Kubernetes is a special work requiring some effort in setting it up >> properly, do we have a dev/test bed to conduct development work? >> >> What I am trying to get at is if there is official support for Volcano >> stuff that a vendor can provide free cluster usage in exchange for R & D. >> For example Google themselves? >> >> Thanks, >> >> Mich >> >> >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Thu, 1 Jul 2021 at 05:00, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >>> Hi Klaus, >>> >>> Thanks >>> >>> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1289 >>> >>> >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Thu, 1 Jul 2021 at 03:16, Klaus Ma <klaus1982...@gmail.com> wrote: >>> >>>> Hi Mich, >>>> >>>> Would you help to open an issue at spark-on-k8s-operator repo? We're >>>> going to submit a PR to update the install steps :) >>>> >>>> -- Klaus >>>> >>>> On Wed, Jun 30, 2021 at 12:24 AM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi Yikun >>>>> >>>>> In reference >>>>> >>>>> >>>>> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/volcano-integration.md >>>>> >>>>> Trying to install Volcano I am getting this error >>>>> >>>>> helm repo add incubator >>>>> http://storage.googleapis.com/kubernetes-charts-incubator >>>>> Error: looks like " >>>>> http://storage.googleapis.com/kubernetes-charts-incubator"; is not a >>>>> valid chart repository or cannot be reached: failed to fetch >>>>> http://storage.googleapis.com/kubernetes-charts-incubator/index.yaml >>>>> : 404 Not Found >>>>> >>>>> Any ideas will be appreciated. >>>>> >>>>> Thanks, >>>>> >>>>> Mich >>>>> >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, 29 Jun 2021 at 09:14, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Cool, thanks! >>>>>> >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 29 Jun 2021 at 07:33, Yikun Jiang <yikunk...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> > Is this the correct link for integrating Volcano with Spark? >>>>>>> >>>>>>> Yes, it is Kubernetes operator style of integrating Volcano. And if >>>>>>> you want to just use spark submit style to submit a native support job, >>>>>>> you >>>>>>> can see [2] as ref. >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/huawei-cloudnative/spark/commit/6c1f37525f026353eaead34216d47dad653f13a4 >>>>>>> >>>>>>> Regards, >>>>>>> Yikun >>>>>>> >>>>>>> >>>>>>> Mich Talebzadeh <mich.talebza...@gmail.com> 于2021年6月28日周一 下午6:03写道： >>>>>>> >>>>>>>> Hi Yikun, >>>>>>>> >>>>>>>> Is this the correct link for integrating Volcano with Spark? >>>>>>>> >>>>>>>> spark-on-k8s-operator/volcano-integration.md at master · >>>>>>>> GoogleCloudPlatform/spark-on-k8s-operator · GitHub >>>>>>>> <https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/volcano-integration.md> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> Mich >>>>>>>> >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>>> for any loss, damage or destruction of data or any other property >>>>>>>> which may >>>>>>>> arise from relying on this email's technical content is explicitly >>>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>>> damages >>>>>>>> arising from such loss, damage or destruction. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, 25 Jun 2021 at 09:45, Yikun Jiang <yikunk...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Oops, sorry for the error link, it should be: >>>>>>>>> >>>>>>>>> We will also prepare to propose an initial design and POC[3] on a >>>>>>>>> shared branch (based on spark master branch) where we can collaborate >>>>>>>>> on >>>>>>>>> it, so I created the spark-volcano[1] org in github to make it happen. >>>>>>>>> >>>>>>>>> [3] >>>>>>>>> https://github.com/huawei-cloudnative/spark/commit/6c1f37525f026353eaead34216d47dad653f13a4 >>>>>>>>> >>>>>>>>> >>>>>>>>> And >>>>>>>>> Regards, >>>>>>>>> Yikun >>>>>>>>> >>>>>>>>> >>>>>>>>> Yikun Jiang <yikunk...@gmail.com> 于2021年6月25日周五 上午11:53写道： >>>>>>>>> >>>>>>>>>> Hi, folks. >>>>>>>>>> >>>>>>>>>> As @Klaus mentioned, We have some work on Spark on k8s with >>>>>>>>>> volcano native support. Also, there were also some production >>>>>>>>>> deployment >>>>>>>>>> validation from our partners in China, like JingDong, XiaoHongShu, >>>>>>>>>> VIPshop. >>>>>>>>>> >>>>>>>>>> We will also prepare to propose an initial design and POC[3] on a >>>>>>>>>> shared branch (based on spark master branch) where we can >>>>>>>>>> collaborate on >>>>>>>>>> it, so I created the spark-volcano[1] org in github to make it >>>>>>>>>> happen. >>>>>>>>>> >>>>>>>>>> Pls feel free to comment on it [2] if you guys have any questions >>>>>>>>>> or concerns. >>>>>>>>>> >>>>>>>>>> [1] https://github.com/spark-volcano >>>>>>>>>> [2] https://github.com/spark-volcano/spark/issues/1 >>>>>>>>>> [3] >>>>>>>>>> https://github.com/huawei-cloudnative/spark/commit/6c1f37525f026353eaead34216d47dad653f13a4 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Yikun >>>>>>>>>> >>>>>>>>>> Holden Karau <hol...@pigscanfly.ca> 于2021年6月25日周五 上午12:00写道： >>>>>>>>>> >>>>>>>>>>> Hi Mich, >>>>>>>>>>> >>>>>>>>>>> I certainly think making Spark on Kubernetes run well is going >>>>>>>>>>> to be a challenge. However I think, and I could be wrong about this >>>>>>>>>>> as >>>>>>>>>>> well, that in terms of cluster managers Kubernetes is likely to be >>>>>>>>>>> our >>>>>>>>>>> future. Talking with people I don't hear about new standalone, YARN >>>>>>>>>>> or >>>>>>>>>>> mesos deployments of Spark, but I do hear about people trying to >>>>>>>>>>> migrate to >>>>>>>>>>> Kubernetes. >>>>>>>>>>> >>>>>>>>>>> To be clear I certainly agree that we need more work on >>>>>>>>>>> structured streaming, but its important to remember that the Spark >>>>>>>>>>> developers are not all fully interchangeable, we work on the things >>>>>>>>>>> that >>>>>>>>>>> we're interested in pursuing so even if structured streaming needs >>>>>>>>>>> more >>>>>>>>>>> love if I'm not super interested in structured streaming I'm less >>>>>>>>>>> likely to >>>>>>>>>>> work on it. That being said I am certainly spinning up a bit more >>>>>>>>>>> in the >>>>>>>>>>> Spark SQL area especially around our data source/connectors because >>>>>>>>>>> I can >>>>>>>>>>> see the need there too. >>>>>>>>>>> >>>>>>>>>>> On Wed, Jun 23, 2021 at 8:26 AM Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Please allow me to be diverse and express a different point of >>>>>>>>>>>> view on this roadmap. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I believe from a technical point of view spending time and >>>>>>>>>>>> effort plus talent on batch scheduling on Kubernetes could be >>>>>>>>>>>> rewarding. >>>>>>>>>>>> However, if I may say I doubt whether such an approach and the >>>>>>>>>>>> so-called >>>>>>>>>>>> democratization of Spark on whatever platform is really should be >>>>>>>>>>>> of great >>>>>>>>>>>> focus. >>>>>>>>>>>> >>>>>>>>>>>> Having worked on Google Dataproc >>>>>>>>>>>> <https://cloud.google.com/dataproc> (A fully managed and >>>>>>>>>>>> highly scalable service for running Apache Spark, Hadoop and more >>>>>>>>>>>> recently >>>>>>>>>>>> other artefacts) for that past two years, and Spark on >>>>>>>>>>>> Kubernetes on-premise, I have come to the conclusion that Spark is >>>>>>>>>>>> not a >>>>>>>>>>>> beast that that one can fully commoditize it much like one can do >>>>>>>>>>>> with >>>>>>>>>>>> Zookeeper, Kafka etc. There is always a struggle to make some >>>>>>>>>>>> niche areas >>>>>>>>>>>> of Spark like Spark Structured Streaming (SSS) work seamlessly and >>>>>>>>>>>> effortlessly on these commercial platforms with whatever as a >>>>>>>>>>>> Service. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Moreover, Spark (and I stand corrected) from the ground up has >>>>>>>>>>>> already a lot of resiliency and redundancy built in. It is truly an >>>>>>>>>>>> enterprise class product (requires enterprise class support) that >>>>>>>>>>>> will be >>>>>>>>>>>> difficult to commoditize with Kubernetes and expect the same >>>>>>>>>>>> performance. >>>>>>>>>>>> After all, Kubernetes is aimed at efficient resource sharing and >>>>>>>>>>>> potential >>>>>>>>>>>> cost saving for the mass market. In short I can see commercial >>>>>>>>>>>> enterprises >>>>>>>>>>>> will work on these platforms ,but may be the great talents on dev >>>>>>>>>>>> team >>>>>>>>>>>> should focus on stuff like the perceived limitation of SSS in >>>>>>>>>>>> dealing with >>>>>>>>>>>> chain of aggregation( if I am correct it is not yet supported on >>>>>>>>>>>> streaming >>>>>>>>>>>> datasets) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> These are my opinions and they are not facts, just opinions so >>>>>>>>>>>> to speak :) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>> responsibility for any loss, damage or destruction of data or any >>>>>>>>>>>> other >>>>>>>>>>>> property which may arise from relying on this email's technical >>>>>>>>>>>> content is >>>>>>>>>>>> explicitly disclaimed. The author will in no case be liable for any >>>>>>>>>>>> monetary damages arising from such loss, damage or destruction. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, 18 Jun 2021 at 23:18, Holden Karau < >>>>>>>>>>>> hol...@pigscanfly.ca> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think these approaches are good, but there are limitations >>>>>>>>>>>>> (eg dynamic scaling) without us making changes inside of the >>>>>>>>>>>>> Spark Kube >>>>>>>>>>>>> scheduler. >>>>>>>>>>>>> >>>>>>>>>>>>> Certainly whichever scheduler extensions we add support for we >>>>>>>>>>>>> should collaborate with the people developing those extensions >>>>>>>>>>>>> insofar as >>>>>>>>>>>>> they are interested. My first place that I checked was >>>>>>>>>>>>> #sig-scheduling >>>>>>>>>>>>> which is fairly quite on the Kubernetes slack but if there are >>>>>>>>>>>>> more places >>>>>>>>>>>>> to look for folks interested in batch scheduling on Kubernetes we >>>>>>>>>>>>> should >>>>>>>>>>>>> definitely give it a shot :) >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jun 18, 2021 at 1:41 AM Mich Talebzadeh < >>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regarding your point and I quote >>>>>>>>>>>>>> >>>>>>>>>>>>>> ".. I know that one of the Spark on Kube operators >>>>>>>>>>>>>> supports volcano/kube-batch so I was thinking that might be a >>>>>>>>>>>>>> place I would >>>>>>>>>>>>>> start exploring..." >>>>>>>>>>>>>> >>>>>>>>>>>>>> There seems to be ongoing work on say Volcano as part of Cloud >>>>>>>>>>>>>> Native Computing Foundation <https://cncf.io/> (CNCF). For >>>>>>>>>>>>>> example through https://github.com/volcano-sh/volcano >>>>>>>>>>>>>> >>>>>>>>>>>>> <https://github.com/volcano-sh/volcano> >>>>>>>>>>>>>> >>>>>>>>>>>>>> There may be value-add in collaborating with such groups >>>>>>>>>>>>>> through CNCF in order to have a collective approach to such >>>>>>>>>>>>>> work. There >>>>>>>>>>>>>> also seems to be some work on Integration of Spark with >>>>>>>>>>>>>> Volcano for Batch Scheduling. >>>>>>>>>>>>>> <https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/volcano-integration.md> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> What is not very clear is the degree of progress of these >>>>>>>>>>>>>> projects. You may be kind enough to elaborate on KPI for each of >>>>>>>>>>>>>> these >>>>>>>>>>>>>> projects and where you think your contributions is going to be. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTH, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mich >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>>>> responsibility for any loss, damage or destruction of data or >>>>>>>>>>>>>> any other >>>>>>>>>>>>>> property which may arise from relying on this email's technical >>>>>>>>>>>>>> content is >>>>>>>>>>>>>> explicitly disclaimed. The author will in no case be liable for >>>>>>>>>>>>>> any >>>>>>>>>>>>>> monetary damages arising from such loss, damage or destruction. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, 18 Jun 2021 at 00:44, Holden Karau < >>>>>>>>>>>>>> hol...@pigscanfly.ca> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Folks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm continuing my adventures to make Spark on containers >>>>>>>>>>>>>>> party and I >>>>>>>>>>>>>>> was wondering if folks have experience with the different >>>>>>>>>>>>>>> batch >>>>>>>>>>>>>>> scheduler options that they prefer? I was thinking so that >>>>>>>>>>>>>>> we can >>>>>>>>>>>>>>> better support dynamic allocation it might make sense for us >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> support using different schedulers and I wanted to see if >>>>>>>>>>>>>>> there are >>>>>>>>>>>>>>> any that the community is more interested in? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I know that one of the Spark on Kube operators supports >>>>>>>>>>>>>>> volcano/kube-batch so I was thinking that might be a place I >>>>>>>>>>>>>>> start >>>>>>>>>>>>>>> exploring but also want to be open to other schedulers that >>>>>>>>>>>>>>> folks >>>>>>>>>>>>>>> might be interested in. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Holden :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>> >>>>>>>>>> -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
