Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Yikun Jiang]

@Weiwei @Chenya

> Thanks for bringing this up. This is quite interesting, we definitely
should participate more in the discussions.

Thanks for your reply and welcome to join the discussion, I think the input
from Yunikorn is very critical.

> The main thing here is, the Spark community should make Spark pluggable
in order to support other schedulers, not just for Volcano. It looks like
this proposal is pushing really hard for adopting PodGroup, which isn't
part of K8s yet, that to me is problematic.

Definitely yes, we are on the same page.

I think we have the same goal: propose a general and reasonable mechanism
to make spark on k8s with a custom scheduler more usable.

But for the PodGroup, just allow me to do a brief introduction:
- The PodGroup definition has been approved by Kubernetes officially in
KEP-583. [1]
- It can be regarded as a general concept/standard in Kubernetes rather
than a specific concept in Volcano, there are also others to implement it,
such as [2][3].
- Kubernetes recommends using CRD to do more extension to implement what
they want. [4]
- Volcano as extension provides an interface to maintain the life cycle
PodGroup CRD and use volcano-scheduler to complete the scheduling.

[1]
https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
[2]
https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling#podgroup
[3] https://github.com/kubernetes-sigs/kube-batch
[4]
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/

Regards,
Yikun


Weiwei Yang  于2021年12月1日周三 上午5:57写道：

> Hi Chenya
>
> Thanks for bringing this up. This is quite interesting, we definitely
> should participate more in the discussions.
> The main thing here is, the Spark community should make Spark pluggable in
> order to support other schedulers, not just for Volcano. It looks like this
> proposal is pushing really hard for adopting PodGroup, which isn't part of
> K8s yet, that to me is problematic.
>
> On Tue, Nov 30, 2021 at 9:21 AM Prasad Paravatha <
> prasad.parava...@gmail.com> wrote:
>
>> This is a great feature/idea.
>> I'd love to get involved in some form (testing and/or documentation).
>> This could be my 1st contribution to Spark!
>>
>> On Tue, Nov 30, 2021 at 10:46 PM John Zhuge  wrote:
>>
>>> +1 Kudos to Yikun and the community for starting the discussion!
>>>
>>> On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang <
>>> chenyazhangche...@gmail.com> wrote:
>>>
 Thanks folks for bringing up the topic of natively integrating Volcano
 and other alternative schedulers into Spark!

 +Weiwei, Wilfred, Chaoran. We would love to contribute to the
 discussion as well.

 From our side, we have been using and improving on one alternative
 resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/),
 for Spark on Kubernetes in production at Apple with solid results in the
 past year. It is capable of supporting Gang scheduling (similar to
 PodGroups), multi-tenant resource queues (similar to YARN), FIFO, and other
 handy features like bin packing to enable efficient autoscaling, etc.

 Natively integrating with Spark would provide more flexibility for
 users and reduce the extra cost and potential inconsistency of maintaining
 different layers of resource strategies. One interesting topic we hope to
 discuss more about is dynamic allocation, which would benefit from native
 coordination between Spark and resource schedulers in K8s &
 cloud environment for an optimal resource efficiency.


 On Tue, Nov 30, 2021 at 8:10 AM Holden Karau 
 wrote:

> Thanks for putting this together, I’m really excited for us to add
> better batch scheduling integrations.
>
> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang 
> wrote:
>
>> Hey everyone,
>>
>> I'd like to start a discussion on "Support Volcano/Alternative
>> Schedulers Proposal".
>>
>> This SPIP is proposed to make spark k8s schedulers provide more YARN
>> like features (such as queues and minimum resources before scheduling 
>> jobs)
>> that many folks want on Kubernetes.
>>
>> The goal of this SPIP is to improve current spark k8s scheduler
>> implementations, add the ability of batch scheduling and support volcano 
>> as
>> one of implementations.
>>
>> Design doc:
>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
>> Part of PRs:
>> Ability to create resources
>> https://github.com/apache/spark/pull/34599
>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>>
>> Regards,
>> Yikun
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Weiwei Yang]

IIUC, PodGroup is only supported in Volcano, this is not a common API
adopted by K8s, at least not today.
Spark needs to be agnostic about the schedulers, as an example, when we run
Spark on YARN, does Spark need to know if that is FairScheduler or
CapacityScheduler?
IMO, we should build things general enough in Spark in order to support
different schedulers, instead of having extra effort to support them one by
one.

On Tue, Nov 30, 2021 at 2:18 PM Mich Talebzadeh 
wrote:

> Hi,
>
> Well, in mitigation, one cannot address all the available scheduler
> options at once. Certainly PodGroup is an option, unless there are reasons
> to believe that this is not a right choice. Therefore, I stand corrected, I
> fail to see where problematic comes into it, unless you may care to
> elaborate your concerns.
>
>
> HTH
>
>
>view my Linkedin profile
> 
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Tue, 30 Nov 2021 at 21:57, Weiwei Yang  wrote:
>
>> Hi Chenya
>>
>> Thanks for bringing this up. This is quite interesting, we definitely
>> should participate more in the discussions.
>> The main thing here is, the Spark community should make Spark pluggable
>> in order to support other schedulers, not just for Volcano. It looks like
>> this proposal is pushing really hard for adopting PodGroup, which isn't
>> part of K8s yet, that to me is problematic.
>>
>> On Tue, Nov 30, 2021 at 9:21 AM Prasad Paravatha <
>> prasad.parava...@gmail.com> wrote:
>>
>>> This is a great feature/idea.
>>> I'd love to get involved in some form (testing and/or documentation).
>>> This could be my 1st contribution to Spark!
>>>
>>> On Tue, Nov 30, 2021 at 10:46 PM John Zhuge  wrote:
>>>
 +1 Kudos to Yikun and the community for starting the discussion!

 On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang <
 chenyazhangche...@gmail.com> wrote:

> Thanks folks for bringing up the topic of natively integrating Volcano
> and other alternative schedulers into Spark!
>
> +Weiwei, Wilfred, Chaoran. We would love to contribute to the
> discussion as well.
>
> From our side, we have been using and improving on one alternative
> resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/),
> for Spark on Kubernetes in production at Apple with solid results in the
> past year. It is capable of supporting Gang scheduling (similar to
> PodGroups), multi-tenant resource queues (similar to YARN), FIFO, and 
> other
> handy features like bin packing to enable efficient autoscaling, etc.
>
> Natively integrating with Spark would provide more flexibility for
> users and reduce the extra cost and potential inconsistency of maintaining
> different layers of resource strategies. One interesting topic we hope to
> discuss more about is dynamic allocation, which would benefit from native
> coordination between Spark and resource schedulers in K8s &
> cloud environment for an optimal resource efficiency.
>
>
> On Tue, Nov 30, 2021 at 8:10 AM Holden Karau 
> wrote:
>
>> Thanks for putting this together, I’m really excited for us to add
>> better batch scheduling integrations.
>>
>> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang 
>> wrote:
>>
>>> Hey everyone,
>>>
>>> I'd like to start a discussion on "Support Volcano/Alternative
>>> Schedulers Proposal".
>>>
>>> This SPIP is proposed to make spark k8s schedulers provide more YARN
>>> like features (such as queues and minimum resources before scheduling 
>>> jobs)
>>> that many folks want on Kubernetes.
>>>
>>> The goal of this SPIP is to improve current spark k8s scheduler
>>> implementations, add the ability of batch scheduling and support 
>>> volcano as
>>> one of implementations.
>>>
>>> Design doc:
>>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
>>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
>>> Part of PRs:
>>> Ability to create resources
>>> https://github.com/apache/spark/pull/34599
>>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>>>
>>> Regards,
>>> Yikun
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

 --
 John Zhuge

>>>
>>>
>>> --
>>> Regards,
>>> Prasad Paravatha
>>>
>>

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Mich Talebzadeh]

Hi,

Well, in mitigation, one cannot address all the available scheduler options
at once. Certainly PodGroup is an option, unless there are reasons to
believe that this is not a right choice. Therefore, I stand corrected, I
fail to see where problematic comes into it, unless you may care to
elaborate your concerns.


HTH


   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 30 Nov 2021 at 21:57, Weiwei Yang  wrote:

> Hi Chenya
>
> Thanks for bringing this up. This is quite interesting, we definitely
> should participate more in the discussions.
> The main thing here is, the Spark community should make Spark pluggable in
> order to support other schedulers, not just for Volcano. It looks like this
> proposal is pushing really hard for adopting PodGroup, which isn't part of
> K8s yet, that to me is problematic.
>
> On Tue, Nov 30, 2021 at 9:21 AM Prasad Paravatha <
> prasad.parava...@gmail.com> wrote:
>
>> This is a great feature/idea.
>> I'd love to get involved in some form (testing and/or documentation).
>> This could be my 1st contribution to Spark!
>>
>> On Tue, Nov 30, 2021 at 10:46 PM John Zhuge  wrote:
>>
>>> +1 Kudos to Yikun and the community for starting the discussion!
>>>
>>> On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang <
>>> chenyazhangche...@gmail.com> wrote:
>>>
 Thanks folks for bringing up the topic of natively integrating Volcano
 and other alternative schedulers into Spark!

 +Weiwei, Wilfred, Chaoran. We would love to contribute to the
 discussion as well.

 From our side, we have been using and improving on one alternative
 resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/),
 for Spark on Kubernetes in production at Apple with solid results in the
 past year. It is capable of supporting Gang scheduling (similar to
 PodGroups), multi-tenant resource queues (similar to YARN), FIFO, and other
 handy features like bin packing to enable efficient autoscaling, etc.

 Natively integrating with Spark would provide more flexibility for
 users and reduce the extra cost and potential inconsistency of maintaining
 different layers of resource strategies. One interesting topic we hope to
 discuss more about is dynamic allocation, which would benefit from native
 coordination between Spark and resource schedulers in K8s &
 cloud environment for an optimal resource efficiency.


 On Tue, Nov 30, 2021 at 8:10 AM Holden Karau 
 wrote:

> Thanks for putting this together, I’m really excited for us to add
> better batch scheduling integrations.
>
> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang 
> wrote:
>
>> Hey everyone,
>>
>> I'd like to start a discussion on "Support Volcano/Alternative
>> Schedulers Proposal".
>>
>> This SPIP is proposed to make spark k8s schedulers provide more YARN
>> like features (such as queues and minimum resources before scheduling 
>> jobs)
>> that many folks want on Kubernetes.
>>
>> The goal of this SPIP is to improve current spark k8s scheduler
>> implementations, add the ability of batch scheduling and support volcano 
>> as
>> one of implementations.
>>
>> Design doc:
>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
>> Part of PRs:
>> Ability to create resources
>> https://github.com/apache/spark/pull/34599
>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>>
>> Regards,
>> Yikun
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

>>>
>>> --
>>> John Zhuge
>>>
>>
>>
>> --
>> Regards,
>> Prasad Paravatha
>>
>

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Weiwei Yang]

Hi Chenya

Thanks for bringing this up. This is quite interesting, we definitely
should participate more in the discussions.
The main thing here is, the Spark community should make Spark pluggable in
order to support other schedulers, not just for Volcano. It looks like this
proposal is pushing really hard for adopting PodGroup, which isn't part of
K8s yet, that to me is problematic.

On Tue, Nov 30, 2021 at 9:21 AM Prasad Paravatha 
wrote:

> This is a great feature/idea.
> I'd love to get involved in some form (testing and/or documentation). This
> could be my 1st contribution to Spark!
>
> On Tue, Nov 30, 2021 at 10:46 PM John Zhuge  wrote:
>
>> +1 Kudos to Yikun and the community for starting the discussion!
>>
>> On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang 
>> wrote:
>>
>>> Thanks folks for bringing up the topic of natively integrating Volcano
>>> and other alternative schedulers into Spark!
>>>
>>> +Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion
>>> as well.
>>>
>>> From our side, we have been using and improving on one alternative
>>> resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/), for
>>> Spark on Kubernetes in production at Apple with solid results in the past
>>> year. It is capable of supporting Gang scheduling (similar to PodGroups),
>>> multi-tenant resource queues (similar to YARN), FIFO, and other handy
>>> features like bin packing to enable efficient autoscaling, etc.
>>>
>>> Natively integrating with Spark would provide more flexibility for users
>>> and reduce the extra cost and potential inconsistency of maintaining
>>> different layers of resource strategies. One interesting topic we hope to
>>> discuss more about is dynamic allocation, which would benefit from native
>>> coordination between Spark and resource schedulers in K8s &
>>> cloud environment for an optimal resource efficiency.
>>>
>>>
>>> On Tue, Nov 30, 2021 at 8:10 AM Holden Karau 
>>> wrote:
>>>
 Thanks for putting this together, I’m really excited for us to add
 better batch scheduling integrations.

 On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang 
 wrote:

> Hey everyone,
>
> I'd like to start a discussion on "Support Volcano/Alternative
> Schedulers Proposal".
>
> This SPIP is proposed to make spark k8s schedulers provide more YARN
> like features (such as queues and minimum resources before scheduling 
> jobs)
> that many folks want on Kubernetes.
>
> The goal of this SPIP is to improve current spark k8s scheduler
> implementations, add the ability of batch scheduling and support volcano 
> as
> one of implementations.
>
> Design doc:
> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
> Part of PRs:
> Ability to create resources https://github.com/apache/spark/pull/34599
> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>
> Regards,
> Yikun
>
 --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>
>>
>> --
>> John Zhuge
>>
>
>
> --
> Regards,
> Prasad Paravatha
>

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Tom Graves]

 Great to have other integrations and improved K8s support.  Left some 
comments/questions in the design doc. 
TomOn Tuesday, November 30, 2021, 02:46:42 AM CST, Yikun Jiang 
 wrote:  
 
 Hey everyone,

I'd like to start a discussion on "Support Volcano/Alternative Schedulers 
Proposal".

This SPIP is proposed to make spark k8s schedulers provide more YARN like 
features (such as queues and minimum resources before scheduling jobs) that 
many folks want on Kubernetes.

The goal of this SPIP is to improve current spark k8s scheduler 
implementations, add the ability of batch scheduling and support volcano as one 
of implementations.

Design doc: 
https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6FgJIRA:
 https://issues.apache.org/jira/browse/SPARK-36057Part of PRs:
Ability to create resources https://github.com/apache/spark/pull/34599
Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456

Regards,Yikun  

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Prasad Paravatha]

This is a great feature/idea.
I'd love to get involved in some form (testing and/or documentation). This
could be my 1st contribution to Spark!

On Tue, Nov 30, 2021 at 10:46 PM John Zhuge  wrote:

> +1 Kudos to Yikun and the community for starting the discussion!
>
> On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang 
> wrote:
>
>> Thanks folks for bringing up the topic of natively integrating Volcano
>> and other alternative schedulers into Spark!
>>
>> +Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion
>> as well.
>>
>> From our side, we have been using and improving on one alternative
>> resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/), for
>> Spark on Kubernetes in production at Apple with solid results in the past
>> year. It is capable of supporting Gang scheduling (similar to PodGroups),
>> multi-tenant resource queues (similar to YARN), FIFO, and other handy
>> features like bin packing to enable efficient autoscaling, etc.
>>
>> Natively integrating with Spark would provide more flexibility for users
>> and reduce the extra cost and potential inconsistency of maintaining
>> different layers of resource strategies. One interesting topic we hope to
>> discuss more about is dynamic allocation, which would benefit from native
>> coordination between Spark and resource schedulers in K8s &
>> cloud environment for an optimal resource efficiency.
>>
>>
>> On Tue, Nov 30, 2021 at 8:10 AM Holden Karau 
>> wrote:
>>
>>> Thanks for putting this together, I’m really excited for us to add
>>> better batch scheduling integrations.
>>>
>>> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang 
>>> wrote:
>>>
 Hey everyone,

 I'd like to start a discussion on "Support Volcano/Alternative
 Schedulers Proposal".

 This SPIP is proposed to make spark k8s schedulers provide more YARN
 like features (such as queues and minimum resources before scheduling jobs)
 that many folks want on Kubernetes.

 The goal of this SPIP is to improve current spark k8s scheduler
 implementations, add the ability of batch scheduling and support volcano as
 one of implementations.

 Design doc:
 https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
 JIRA: https://issues.apache.org/jira/browse/SPARK-36057
 Part of PRs:
 Ability to create resources https://github.com/apache/spark/pull/34599
 Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456

 Regards,
 Yikun

>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>
> --
> John Zhuge
>


-- 
Regards,
Prasad Paravatha

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[John Zhuge]

+1 Kudos to Yikun and the community for starting the discussion!

On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang 
wrote:

> Thanks folks for bringing up the topic of natively integrating Volcano and
> other alternative schedulers into Spark!
>
> +Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion
> as well.
>
> From our side, we have been using and improving on one alternative
> resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/), for
> Spark on Kubernetes in production at Apple with solid results in the past
> year. It is capable of supporting Gang scheduling (similar to PodGroups),
> multi-tenant resource queues (similar to YARN), FIFO, and other handy
> features like bin packing to enable efficient autoscaling, etc.
>
> Natively integrating with Spark would provide more flexibility for users
> and reduce the extra cost and potential inconsistency of maintaining
> different layers of resource strategies. One interesting topic we hope to
> discuss more about is dynamic allocation, which would benefit from native
> coordination between Spark and resource schedulers in K8s &
> cloud environment for an optimal resource efficiency.
>
>
> On Tue, Nov 30, 2021 at 8:10 AM Holden Karau  wrote:
>
>> Thanks for putting this together, I’m really excited for us to add better
>> batch scheduling integrations.
>>
>> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang  wrote:
>>
>>> Hey everyone,
>>>
>>> I'd like to start a discussion on "Support Volcano/Alternative
>>> Schedulers Proposal".
>>>
>>> This SPIP is proposed to make spark k8s schedulers provide more YARN
>>> like features (such as queues and minimum resources before scheduling jobs)
>>> that many folks want on Kubernetes.
>>>
>>> The goal of this SPIP is to improve current spark k8s scheduler
>>> implementations, add the ability of batch scheduling and support volcano as
>>> one of implementations.
>>>
>>> Design doc:
>>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
>>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
>>> Part of PRs:
>>> Ability to create resources https://github.com/apache/spark/pull/34599
>>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>>>
>>> Regards,
>>> Yikun
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

-- 
John Zhuge

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Chenya Zhang]

Thanks folks for bringing up the topic of natively integrating Volcano and
other alternative schedulers into Spark!

+Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion as
well.

>From our side, we have been using and improving on one alternative resource
scheduler, Apache YuniKorn (https://yunikorn.apache.org/), for Spark on
Kubernetes in production at Apple with solid results in the past year. It
is capable of supporting Gang scheduling (similar to PodGroups),
multi-tenant resource queues (similar to YARN), FIFO, and other handy
features like bin packing to enable efficient autoscaling, etc.

Natively integrating with Spark would provide more flexibility for users
and reduce the extra cost and potential inconsistency of maintaining
different layers of resource strategies. One interesting topic we hope to
discuss more about is dynamic allocation, which would benefit from native
coordination between Spark and resource schedulers in K8s &
cloud environment for an optimal resource efficiency.


On Tue, Nov 30, 2021 at 8:10 AM Holden Karau  wrote:

> Thanks for putting this together, I’m really excited for us to add better
> batch scheduling integrations.
>
> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang  wrote:
>
>> Hey everyone,
>>
>> I'd like to start a discussion on "Support Volcano/Alternative Schedulers
>> Proposal".
>>
>> This SPIP is proposed to make spark k8s schedulers provide more YARN like
>> features (such as queues and minimum resources before scheduling jobs) that
>> many folks want on Kubernetes.
>>
>> The goal of this SPIP is to improve current spark k8s scheduler
>> implementations, add the ability of batch scheduling and support volcano as
>> one of implementations.
>>
>> Design doc:
>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
>> Part of PRs:
>> Ability to create resources https://github.com/apache/spark/pull/34599
>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>>
>> Regards,
>> Yikun
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Holden Karau]

Thanks for putting this together, I’m really excited for us to add better
batch scheduling integrations.

On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang  wrote:

> Hey everyone,
>
> I'd like to start a discussion on "Support Volcano/Alternative Schedulers
> Proposal".
>
> This SPIP is proposed to make spark k8s schedulers provide more YARN like
> features (such as queues and minimum resources before scheduling jobs) that
> many folks want on Kubernetes.
>
> The goal of this SPIP is to improve current spark k8s scheduler
> implementations, add the ability of batch scheduling and support volcano as
> one of implementations.
>
> Design doc:
> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
> Part of PRs:
> Ability to create resources https://github.com/apache/spark/pull/34599
> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>
> Regards,
> Yikun
>
-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Mich Talebzadeh]

Hi,

Good points. I have started making some suggestions to the Design doc as
referenced above.

My main concern with the current scheduling is the fact that there seems to
be one pod per node association with k8s application. The model works on
the basis of the "one-container-per-Pod" model
 meaning that for
each node of the cluster you will have one node running the driver and each
remaining node running one executor each. I tested this using Google
Kubernetes cluster (GKE)  with dynamic adjustment of the number of nodes.
Now there is also the lead time for the driver to create the executors.
when the number of executors set to a higher number of the nodes
through --conf spark.executor.instances=$NEXEC,


For example I am testing this in Google Kubernetes Cluster with three nodes
and no of executors set at 9


NAME READY   STATUS
RESTARTS   AGE

randomdatabigquery-fbfaa97d7019ffa7-exec-1   0/1 ContainerCreating   0
22s

randomdatabigquery-fbfaa97d7019ffa7-exec-2   0/1 ContainerCreating   0
22s

randomdatabigquery-fbfaa97d7019ffa7-exec-3   1/1 Running 0
22s

randomdatabigquery-fbfaa97d7019ffa7-exec-4   0/1 ContainerCreating   0
22s

randomdatabigquery-fbfaa97d7019ffa7-exec-5   0/1 ContainerCreating   0
21s

randomdatabigquery-fbfaa97d7019ffa7-exec-6   1/1 Running 0
20s

randomdatabigquery-fbfaa97d7019ffa7-exec-7   0/1 ContainerCreating   0
20s

randomdatabigquery-fbfaa97d7019ffa7-exec-8   0/1 ContainerCreating   0
20s

randomdatabigquery-fbfaa97d7019ffa7-exec-9   0/1 Pending 0
20s

sparkbq-ad80717d701955ad-driver  1/1 Running 0
66s

Some of these exectors are totally redundant but the driver still takes
time to create them which I believe is a waste of resources. This needs to
be avoided. Secondally, the current scheduler is based on Spark standalone
mode, which I think is less strict than the Yarn mode. I am not sure how
much volcano contribution is to make this better. However, we need to
improve the current scheduling regardless.


HTH



   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 30 Nov 2021 at 08:47, Yikun Jiang  wrote:

> Hey everyone,
>
> I'd like to start a discussion on "Support Volcano/Alternative Schedulers
> Proposal".
>
> This SPIP is proposed to make spark k8s schedulers provide more YARN like
> features (such as queues and minimum resources before scheduling jobs) that
> many folks want on Kubernetes.
>
> The goal of this SPIP is to improve current spark k8s scheduler
> implementations, add the ability of batch scheduling and support volcano as
> one of implementations.
>
> Design doc:
> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
> Part of PRs:
> Ability to create resources https://github.com/apache/spark/pull/34599
> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>
> Regards,
> Yikun
>

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Hyukjin Kwon]

Adding @Holden Karau  @Dongjoon Hyun
 @wuyi  FYI

On Tue, 30 Nov 2021 at 17:46, Yikun Jiang  wrote:

> Hey everyone,
>
> I'd like to start a discussion on "Support Volcano/Alternative Schedulers
> Proposal".
>
> This SPIP is proposed to make spark k8s schedulers provide more YARN like
> features (such as queues and minimum resources before scheduling jobs) that
> many folks want on Kubernetes.
>
> The goal of this SPIP is to improve current spark k8s scheduler
> implementations, add the ability of batch scheduling and support volcano as
> one of implementations.
>
> Design doc:
> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
> JIRA: https://issues.apache.org/jira/browse/SPARK-36057
> Part of PRs:
> Ability to create resources https://github.com/apache/spark/pull/34599
> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456
>
> Regards,
> Yikun
>

[DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Yikun Jiang]

Hey everyone,

I'd like to start a discussion on "Support Volcano/Alternative Schedulers
Proposal".

This SPIP is proposed to make spark k8s schedulers provide more YARN like
features (such as queues and minimum resources before scheduling jobs) that
many folks want on Kubernetes.

The goal of this SPIP is to improve current spark k8s scheduler
implementations, add the ability of batch scheduling and support volcano as
one of implementations.

Design doc:
https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg
JIRA: https://issues.apache.org/jira/browse/SPARK-36057
Part of PRs:
Ability to create resources https://github.com/apache/spark/pull/34599
Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456

Regards,
Yikun