Re: Spark on Kubernetes scheduler variety

Splendid. Please invite me to the next meeting mich.talebza...@gmail.com Timezone London, UK *GMT+1* Thanks, view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or

Re: Spark on Kubernetes scheduler variety

Hi Y'all, We had an initial meeting which went well, got some more context around Volcano and its near-term roadmap. Talked about the impact around scheduler deadlocking and some ways that we could potentially improve integration from the Spark side and Volcano sides respectively. I'm going to

Re: Spark on Kubernetes scheduler variety

Hi Holden, Thank you for your points. I guess coming from a corporate world I had an oversight on how an open source project like Spark does leverage resources and interest :). As @KlausMa kindly volunteered it would be good to hear scheduling ideas on Spark on Kubernetes and of course as I am

Re: Spark on Kubernetes scheduler variety

That's awesome, I'm just starting to get context around Volcano but maybe we can schedule an initial meeting for all of us interested in pursuing this to get on the same page. On Wed, Jun 23, 2021 at 6:54 PM Klaus Ma wrote: > Hi team, > > I'm kube-batch/Volcano founder, and I'm excited to hear

Re: Spark on Kubernetes scheduler variety

spark" Cc: dev Subject: RE: [EXTERNAL] Spark on Kubernetes scheduler variety CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Please allow me to be diverse and express a

Re: Spark on Kubernetes scheduler variety

Thanks Klaus! I am interested in more details. On Wed, Jun 23, 2021 at 6:54 PM Klaus Ma wrote: > Hi team, > > I'm kube-batch/Volcano founder, and I'm excited to hear that the spark > community also has such requirements :) > > Volcano provides several features for batch workload, e.g.

Re: Spark on Kubernetes scheduler variety

Thanks Klaus. That will be great. It will also be intuitive if you elaborate the need for this feature in line with the limitation of the current batch workload. Regards, Mich view my Linkedin profile *Disclaimer:* Use it at

Re: Spark on Kubernetes scheduler variety

Hi team, I'm kube-batch/Volcano founder, and I'm excited to hear that the spark community also has such requirements :) Volcano provides several features for batch workload, e.g. fair-share, queue, reservation, preemption/reclaim and so on. It has been used in several product environments with

Re: Spark on Kubernetes scheduler variety

Please allow me to be diverse and express a different point of view on this roadmap. I believe from a technical point of view spending time and effort plus talent on batch scheduling on Kubernetes could be rewarding. However, if I may say I doubt whether such an approach and the so-called