Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Yikun Jiang]

> Any guidance on how to best contribute?

@Agarwal Thanks for the feedback.

- It would be very good if you could share your idea and suggestion on
native scheduler support in SPARK-36057
, it would be considered
as part of this feature or next advanced improvement in followup.
- You could also feel free to help review the existing PR.

Anyway, you could regard the scope of this feature as enabling the basic
ability to integrate the customized scheduler and help job level scheduling
at some level, it's just a start, if you have any other concern, feel free
to leave any comments.

Regards,
Yikun


Agarwal, Janak  于2022年1月5日周三 02:05写道：

> Hello Folks, Happy new year to one and all.
>
>
>
> I’m from the EMR on EKS  team.
> We help customers to run Spark workloads on Kubernetes.
>
> My team had similar ideas, and we have also sourced requirements from
> customers who use EMR on EKS / Spark on EKS. Would love to participate in
> the design to help solve the problem for the vast majority of Spark on
> Kubernetes users.
>
>
>
> Any guidance on how to best contribute?
>
>
>
> Best,
>
> Janak
>
>
>
> *From:* Mich Talebzadeh 
> *Sent:* Tuesday, January 4, 2022 2:12 AM
> *To:* Yikun Jiang 
> *Cc:* dev ; Weiwei Yang ; Holden
> Karau ; wang.platf...@gmail.com; Prasad Paravatha <
> prasad.parava...@gmail.com>; John Zhuge ; Chenya Zhang
> ; Chaoran Yu ;
> Wilfred Spiegelenburg ; Klaus Ma <
> klaus1982...@gmail.com>
> *Subject:* RE: [EXTERNAL] [DISCUSSION] SPIP: Support Volcano/Alternative
> Schedulers Proposal
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Interesting,thanks
>
>
>
> Do you have any indication of the ballpark figure (a rough numerical
> estimate) of adding Volcano as an alternative scheduler is going to
> improve Spark on k8s performance?
>
>
>
> Thanks
>
>
>
>
>view my Linkedin profile
> 
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
>
>
> On Tue, 4 Jan 2022 at 09:43, Yikun Jiang  wrote:
>
> Hi, folks! Wishing you all the best in 2022.
>
>
>
> I'd like to share the current status on "Support Customized K8S Scheduler
> in Spark".
>
>
> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg/edit#heading=h.1quyr1r2kr5n
>
>
>
> Framework/Common support
>
> - Volcano and Yunikorn team join the discussion and complete the initial
> doc on framework/common part.
>
> - SPARK-37145  (under
> reviewing): We proposed to extend the customized scheduler by just using a
> custom feature step, it will meet the requirement of customized scheduler
> after it gets merged. After this, the user can enable featurestep and
> scheduler like:
>
> spark-submit \
>
> --conf spark.kubernete.scheduler.name volcano \
>
> --conf spark.kubernetes.driver.pod.featureSteps
> org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep
>
> --conf spark.kubernete.job.queue xxx
>
> (such as above, the VolcanoFeatureStep will help to set the the spark
> scheduler queue according user specified conf)
>
> - SPARK-37331 : Added
> the ability to create kubernetes resources before driver pod creation.
>
> - SPARK-36059 : Add
> the ability to specify a scheduler in driver/executor
>
> After above all, the framework/common support would be ready for most of
> customized schedulers
>
>
>
> Volcano part:
>
> - SPARK-37258 :
> Upgrade kubernetes-client to 5.11.1 to add volcano scheduler API support.
>
> - SPARK-36061 : Add a
> VolcanoFeatureStep to help users to create a PodGroup with user specified
> minimum resources required, there is also a WIP commit to show the
> preview of this
> 
> .
>
>
>
> Yunikorn part:
>
> - @WeiweiYang is completing the doc of the Yunikorn part and implementing
> the Yunikorn part.
>
>
>
> Regards,
>
> Yikun
>
>
>
>
>
> Weiwei Yang  于2021年12月2日周四 02:00写道：
>
> Thank you Yikun for the info, and thanks for inviting me to a meeting to
> discuss this.
>
> I appreciate your effort to put these together, and I agree that the
> purpose is to make Spark easy/flexible enough to support other K8s
> schedulers (not just for Volcano).
>
> As discussed, could you 

About contribution

[Dennis Jung]

Hello, I hope this is not a silly question.
(I couldn't find any chat room on spark project, so asking on mail)

It has been about a year since using spark in work, and try to make a
contribution to this project.

I'm currently looking at documents in more detail, and checking the issue
in JIRA now. Is there some suggestion of reviewing the code?

- Which code part will be good to start?
- What will be more helpful for the project?

Thanks.

RE: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Agarwal, Janak]

Hello Folks, Happy new year to one and all.

I’m from the EMR on EKS team. We help 
customers to run Spark workloads on Kubernetes.
My team had similar ideas, and we have also sourced requirements from customers 
who use EMR on EKS / Spark on EKS. Would love to participate in the design to 
help solve the problem for the vast majority of Spark on Kubernetes users.

Any guidance on how to best contribute?

Best,
Janak

From: Mich Talebzadeh 
Sent: Tuesday, January 4, 2022 2:12 AM
To: Yikun Jiang 
Cc: dev ; Weiwei Yang ; Holden Karau 
; wang.platf...@gmail.com; Prasad Paravatha 
; John Zhuge ; Chenya Zhang 
; Chaoran Yu ; Wilfred 
Spiegelenburg ; Klaus Ma 
Subject: RE: [EXTERNAL] [DISCUSSION] SPIP: Support Volcano/Alternative 
Schedulers Proposal


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Interesting,thanks

Do you have any indication of the ballpark figure (a rough numerical estimate) 
of adding Volcano as an alternative scheduler is going to improve Spark on k8s 
performance?

Thanks



 
[https://docs.google.com/uc?export=download=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ]
   view my Linkedin 
profile



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destruction of data or any other property which may arise from 
relying on this email's technical content is explicitly disclaimed. The author 
will in no case be liable for any monetary damages arising from such loss, 
damage or destruction.




On Tue, 4 Jan 2022 at 09:43, Yikun Jiang 
mailto:yikunk...@gmail.com>> wrote:

Hi, folks! Wishing you all the best in 2022.


I'd like to share the current status on "Support Customized K8S Scheduler in 
Spark".

https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg/edit#heading=h.1quyr1r2kr5n


Framework/Common support

- Volcano and Yunikorn team join the discussion and complete the initial doc on 
framework/common part.

- SPARK-37145 (under 
reviewing): We proposed to extend the customized scheduler by just using a 
custom feature step, it will meet the requirement of customized scheduler after 
it gets merged. After this, the user can enable featurestep and scheduler like:

spark-submit \

--conf 
spark.kubernete.scheduler.name volcano \

--conf spark.kubernetes.driver.pod.featureSteps 
org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep

--conf spark.kubernete.job.queue xxx

(such as above, the VolcanoFeatureStep will help to set the the spark scheduler 
queue according user specified conf)

- SPARK-37331: Added the 
ability to create kubernetes resources before driver pod creation.

- SPARK-36059: Add the 
ability to specify a scheduler in driver/executor

After above all, the framework/common support would be ready for most of 
customized schedulers



Volcano part:

- SPARK-37258: Upgrade 
kubernetes-client to 5.11.1 to add volcano scheduler API support.

- SPARK-36061: Add a 
VolcanoFeatureStep to help users to create a PodGroup with user specified 
minimum resources required, there is also a WIP commit to show the preview of 
this.


Yunikorn part:

- @WeiweiYang is completing the doc of the Yunikorn part and implementing the 
Yunikorn part.


Regards,
Yikun


Weiwei Yang mailto:w...@apache.org>> 于2021年12月2日周四 02:00写道：
Thank you Yikun for the info, and thanks for inviting me to a meeting to 
discuss this.
I appreciate your effort to put these together, and I agree that the purpose is 
to make Spark easy/flexible enough to support other K8s schedulers (not just 
for Volcano).
As discussed, could you please help to abstract out the things in common and 
allow Spark to plug different implementations? I'd be happy to work with you 
guys on this issue.


On Tue, Nov 30, 2021 at 6:49 PM Yikun Jiang 
mailto:yikunk...@gmail.com>> wrote:
@Weiwei @Chenya

> Thanks for bringing this up. This is quite interesting, we definitely should 
> participate more in the discussions.

Thanks for your reply and welcome to join the discussion, I think the input 
from Yunikorn is very critical.

> The main thing here is, the Spark community should make Spark pluggable in 
> order to support other schedulers, not just for Volcano. It looks like this 
> proposal is pushing really hard for adopting PodGroup, which isn't part of 
> K8s yet, that to me is problematic.

Definitely yes, we are on the 

Re: Time for Spark 3.2.1?

[Mich Talebzadeh]

+1 non-binding



   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 4 Jan 2022 at 16:13, John Zhuge  wrote:

> +1 thanks
>
> On Tue, Jan 4, 2022 at 7:59 AM huaxin gao  wrote:
>
>> Happy New Year, everyone!
>>
>> I will start preparing for Spark 3.2.1 release. I plan to do the branch
>> cut on Friday 1/7. Please let me know if there are any issues I need to be
>> aware of.
>>
>> Thanks,
>> Huaxin
>>
>>
>> On Tue, Dec 7, 2021 at 11:03 PM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>>
>>> +1 for both releases and the time!
>>>
>>> On Wed, Dec 8, 2021 at 3:46 PM Mridul Muralidharan 
>>> wrote:
>>>

 +1 for maintenance release, and also +1 for doing this in Jan !

 Thanks,
 Mridul

 On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang 
 wrote:

> +1 for new maintenance releases for all 3.x branches as well.
>
> On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon 
> wrote:
>
>> SGTM!
>>
>> On Wed, 8 Dec 2021 at 09:07, huaxin gao 
>> wrote:
>>
>>> I prefer to start rolling the release in January if there is no need
>>> to publish it sooner :)
>>>
>>> On Tue, Dec 7, 2021 at 3:59 PM Hyukjin Kwon 
>>> wrote:
>>>
 Oh BTW, I realised that it's a holiday season soon this month
 including Christmas and new year.
 Shall we maybe start rolling the release around next January? I
 would leave it to @huaxin gao  :-).

 On Wed, 8 Dec 2021 at 06:19, Dongjoon Hyun 
 wrote:

> +1 for new releases.
>
> Dongjoon.
>
> On Mon, Dec 6, 2021 at 8:51 PM Wenchen Fan 
> wrote:
>
>> +1 to make new maintenance releases for all 3.x branches.
>>
>> On Tue, Dec 7, 2021 at 8:57 AM Sean Owen 
>> wrote:
>>
>>> Always fine by me if someone wants to roll a release.
>>>
>>> It's been ~6 months since the last 3.0.x and 3.1.x releases,
>>> too; a new release of those wouldn't hurt either, if any of our 
>>> release
>>> managers have the time or inclination. 3.0.x is reaching unofficial
>>> end-of-life around now anyway.
>>>
>>>
>>> On Mon, Dec 6, 2021 at 6:55 PM Hyukjin Kwon 
>>> wrote:
>>>
 Hi all,

 It's been two months since Spark 3.2.0 release, and we have
 resolved many bug fixes and regressions. What do you guys think 
 about
 rolling Spark 3.2.1 release?

 cc @huaxin gao  FYI who I happened to
 overhear that is interested in rolling the maintenance release :-).

>>> --
> John Zhuge
>

Re: Time for Spark 3.2.1?

[John Zhuge]

+1 thanks

On Tue, Jan 4, 2022 at 7:59 AM huaxin gao  wrote:

> Happy New Year, everyone!
>
> I will start preparing for Spark 3.2.1 release. I plan to do the branch
> cut on Friday 1/7. Please let me know if there are any issues I need to be
> aware of.
>
> Thanks,
> Huaxin
>
>
> On Tue, Dec 7, 2021 at 11:03 PM Jungtaek Lim 
> wrote:
>
>> +1 for both releases and the time!
>>
>> On Wed, Dec 8, 2021 at 3:46 PM Mridul Muralidharan 
>> wrote:
>>
>>>
>>> +1 for maintenance release, and also +1 for doing this in Jan !
>>>
>>> Thanks,
>>> Mridul
>>>
>>> On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang  wrote:
>>>
 +1 for new maintenance releases for all 3.x branches as well.

 On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon 
 wrote:

> SGTM!
>
> On Wed, 8 Dec 2021 at 09:07, huaxin gao 
> wrote:
>
>> I prefer to start rolling the release in January if there is no need
>> to publish it sooner :)
>>
>> On Tue, Dec 7, 2021 at 3:59 PM Hyukjin Kwon 
>> wrote:
>>
>>> Oh BTW, I realised that it's a holiday season soon this month
>>> including Christmas and new year.
>>> Shall we maybe start rolling the release around next January? I
>>> would leave it to @huaxin gao  :-).
>>>
>>> On Wed, 8 Dec 2021 at 06:19, Dongjoon Hyun 
>>> wrote:
>>>
 +1 for new releases.

 Dongjoon.

 On Mon, Dec 6, 2021 at 8:51 PM Wenchen Fan 
 wrote:

> +1 to make new maintenance releases for all 3.x branches.
>
> On Tue, Dec 7, 2021 at 8:57 AM Sean Owen  wrote:
>
>> Always fine by me if someone wants to roll a release.
>>
>> It's been ~6 months since the last 3.0.x and 3.1.x releases, too;
>> a new release of those wouldn't hurt either, if any of our release 
>> managers
>> have the time or inclination. 3.0.x is reaching unofficial 
>> end-of-life
>> around now anyway.
>>
>>
>> On Mon, Dec 6, 2021 at 6:55 PM Hyukjin Kwon 
>> wrote:
>>
>>> Hi all,
>>>
>>> It's been two months since Spark 3.2.0 release, and we have
>>> resolved many bug fixes and regressions. What do you guys think 
>>> about
>>> rolling Spark 3.2.1 release?
>>>
>>> cc @huaxin gao  FYI who I happened to
>>> overhear that is interested in rolling the maintenance release :-).
>>>
>> --
John Zhuge

Re: Time for Spark 3.2.1?

[huaxin gao]

Happy New Year, everyone!

I will start preparing for Spark 3.2.1 release. I plan to do the branch cut
on Friday 1/7. Please let me know if there are any issues I need to be
aware of.

Thanks,
Huaxin


On Tue, Dec 7, 2021 at 11:03 PM Jungtaek Lim 
wrote:

> +1 for both releases and the time!
>
> On Wed, Dec 8, 2021 at 3:46 PM Mridul Muralidharan 
> wrote:
>
>>
>> +1 for maintenance release, and also +1 for doing this in Jan !
>>
>> Thanks,
>> Mridul
>>
>> On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang  wrote:
>>
>>> +1 for new maintenance releases for all 3.x branches as well.
>>>
>>> On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon  wrote:
>>>
 SGTM!

 On Wed, 8 Dec 2021 at 09:07, huaxin gao  wrote:

> I prefer to start rolling the release in January if there is no need
> to publish it sooner :)
>
> On Tue, Dec 7, 2021 at 3:59 PM Hyukjin Kwon 
> wrote:
>
>> Oh BTW, I realised that it's a holiday season soon this month
>> including Christmas and new year.
>> Shall we maybe start rolling the release around next January? I would
>> leave it to @huaxin gao  :-).
>>
>> On Wed, 8 Dec 2021 at 06:19, Dongjoon Hyun 
>> wrote:
>>
>>> +1 for new releases.
>>>
>>> Dongjoon.
>>>
>>> On Mon, Dec 6, 2021 at 8:51 PM Wenchen Fan 
>>> wrote:
>>>
 +1 to make new maintenance releases for all 3.x branches.

 On Tue, Dec 7, 2021 at 8:57 AM Sean Owen  wrote:

> Always fine by me if someone wants to roll a release.
>
> It's been ~6 months since the last 3.0.x and 3.1.x releases, too;
> a new release of those wouldn't hurt either, if any of our release 
> managers
> have the time or inclination. 3.0.x is reaching unofficial end-of-life
> around now anyway.
>
>
> On Mon, Dec 6, 2021 at 6:55 PM Hyukjin Kwon 
> wrote:
>
>> Hi all,
>>
>> It's been two months since Spark 3.2.0 release, and we have
>> resolved many bug fixes and regressions. What do you guys think about
>> rolling Spark 3.2.1 release?
>>
>> cc @huaxin gao  FYI who I happened to
>> overhear that is interested in rolling the maintenance release :-).
>>
>

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Mich Talebzadeh]

Interesting,thanks

Do you have any indication of the ballpark figure (a rough numerical
estimate) of adding Volcano as an alternative scheduler is going to improve
Spark on k8s performance?

Thanks


   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 4 Jan 2022 at 09:43, Yikun Jiang  wrote:

> Hi, folks! Wishing you all the best in 2022.
>
> I'd like to share the current status on "Support Customized K8S Scheduler
> in Spark".
>
>
> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg/edit#heading=h.1quyr1r2kr5n
>
> Framework/Common support
>
> - Volcano and Yunikorn team join the discussion and complete the initial
> doc on framework/common part.
>
> - SPARK-37145  (under
> reviewing): We proposed to extend the customized scheduler by just using a
> custom feature step, it will meet the requirement of customized scheduler
> after it gets merged. After this, the user can enable featurestep and
> scheduler like:
>
> spark-submit \
>
> --conf spark.kubernete.scheduler.name volcano \
>
> --conf spark.kubernetes.driver.pod.featureSteps
> org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep
>
> --conf spark.kubernete.job.queue xxx
>
> (such as above, the VolcanoFeatureStep will help to set the the spark
> scheduler queue according user specified conf)
>
> - SPARK-37331 : Added
> the ability to create kubernetes resources before driver pod creation.
>
> - SPARK-36059 : Add
> the ability to specify a scheduler in driver/executor
>
> After above all, the framework/common support would be ready for most of
> customized schedulers
>
> Volcano part:
>
> - SPARK-37258 :
> Upgrade kubernetes-client to 5.11.1 to add volcano scheduler API support.
>
> - SPARK-36061 : Add a
> VolcanoFeatureStep to help users to create a PodGroup with user specified
> minimum resources required, there is also a WIP commit to show the
> preview of this
> 
> .
>
> Yunikorn part:
>
> - @WeiweiYang is completing the doc of the Yunikorn part and implementing
> the Yunikorn part.
>
> Regards,
> Yikun
>
>
> Weiwei Yang  于2021年12月2日周四 02:00写道：
>
>> Thank you Yikun for the info, and thanks for inviting me to a meeting to
>> discuss this.
>> I appreciate your effort to put these together, and I agree that the
>> purpose is to make Spark easy/flexible enough to support other K8s
>> schedulers (not just for Volcano).
>> As discussed, could you please help to abstract out the things in common
>> and allow Spark to plug different implementations? I'd be happy to work
>> with you guys on this issue.
>>
>>
>> On Tue, Nov 30, 2021 at 6:49 PM Yikun Jiang  wrote:
>>
>>> @Weiwei @Chenya
>>>
>>> > Thanks for bringing this up. This is quite interesting, we definitely
>>> should participate more in the discussions.
>>>
>>> Thanks for your reply and welcome to join the discussion, I think the
>>> input from Yunikorn is very critical.
>>>
>>> > The main thing here is, the Spark community should make Spark
>>> pluggable in order to support other schedulers, not just for Volcano. It
>>> looks like this proposal is pushing really hard for adopting PodGroup,
>>> which isn't part of K8s yet, that to me is problematic.
>>>
>>> Definitely yes, we are on the same page.
>>>
>>> I think we have the same goal: propose a general and reasonable
>>> mechanism to make spark on k8s with a custom scheduler more usable.
>>>
>>> But for the PodGroup, just allow me to do a brief introduction:
>>> - The PodGroup definition has been approved by Kubernetes officially in
>>> KEP-583. [1]
>>> - It can be regarded as a general concept/standard in Kubernetes rather
>>> than a specific concept in Volcano, there are also others to implement it,
>>> such as [2][3].
>>> - Kubernetes recommends using CRD to do more extension to implement what
>>> they want. [4]
>>> - Volcano as extension provides an interface to maintain the life cycle
>>> PodGroup CRD and use volcano-scheduler to complete the scheduling.
>>>
>>> [1]
>>> https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
>>> [2]
>>> https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling#podgroup
>>> [3] https://github.com/kubernetes-sigs/kube-batch
>>> [4]
>>> 

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

[Yikun Jiang]

Hi, folks! Wishing you all the best in 2022.

I'd like to share the current status on "Support Customized K8S Scheduler
in Spark".

https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg/edit#heading=h.1quyr1r2kr5n

Framework/Common support

- Volcano and Yunikorn team join the discussion and complete the initial
doc on framework/common part.

- SPARK-37145  (under
reviewing): We proposed to extend the customized scheduler by just using a
custom feature step, it will meet the requirement of customized scheduler
after it gets merged. After this, the user can enable featurestep and
scheduler like:

spark-submit \

--conf spark.kubernete.scheduler.name volcano \

--conf spark.kubernetes.driver.pod.featureSteps
org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep

--conf spark.kubernete.job.queue xxx

(such as above, the VolcanoFeatureStep will help to set the the spark
scheduler queue according user specified conf)

- SPARK-37331 : Added
the ability to create kubernetes resources before driver pod creation.

- SPARK-36059 : Add the
ability to specify a scheduler in driver/executor

After above all, the framework/common support would be ready for most of
customized schedulers

Volcano part:

- SPARK-37258 : Upgrade
kubernetes-client to 5.11.1 to add volcano scheduler API support.

- SPARK-36061 : Add a
VolcanoFeatureStep to help users to create a PodGroup with user specified
minimum resources required, there is also a WIP commit to show the preview
of this

.

Yunikorn part:

- @WeiweiYang is completing the doc of the Yunikorn part and implementing
the Yunikorn part.

Regards,
Yikun


Weiwei Yang  于2021年12月2日周四 02:00写道：

> Thank you Yikun for the info, and thanks for inviting me to a meeting to
> discuss this.
> I appreciate your effort to put these together, and I agree that the
> purpose is to make Spark easy/flexible enough to support other K8s
> schedulers (not just for Volcano).
> As discussed, could you please help to abstract out the things in common
> and allow Spark to plug different implementations? I'd be happy to work
> with you guys on this issue.
>
>
> On Tue, Nov 30, 2021 at 6:49 PM Yikun Jiang  wrote:
>
>> @Weiwei @Chenya
>>
>> > Thanks for bringing this up. This is quite interesting, we definitely
>> should participate more in the discussions.
>>
>> Thanks for your reply and welcome to join the discussion, I think the
>> input from Yunikorn is very critical.
>>
>> > The main thing here is, the Spark community should make Spark pluggable
>> in order to support other schedulers, not just for Volcano. It looks like
>> this proposal is pushing really hard for adopting PodGroup, which isn't
>> part of K8s yet, that to me is problematic.
>>
>> Definitely yes, we are on the same page.
>>
>> I think we have the same goal: propose a general and reasonable mechanism
>> to make spark on k8s with a custom scheduler more usable.
>>
>> But for the PodGroup, just allow me to do a brief introduction:
>> - The PodGroup definition has been approved by Kubernetes officially in
>> KEP-583. [1]
>> - It can be regarded as a general concept/standard in Kubernetes rather
>> than a specific concept in Volcano, there are also others to implement it,
>> such as [2][3].
>> - Kubernetes recommends using CRD to do more extension to implement what
>> they want. [4]
>> - Volcano as extension provides an interface to maintain the life cycle
>> PodGroup CRD and use volcano-scheduler to complete the scheduling.
>>
>> [1]
>> https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
>> [2]
>> https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling#podgroup
>> [3] https://github.com/kubernetes-sigs/kube-batch
>> [4]
>> https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
>>
>> Regards,
>> Yikun
>>
>>
>> Weiwei Yang  于2021年12月1日周三 上午5:57写道：
>>
>>> Hi Chenya
>>>
>>> Thanks for bringing this up. This is quite interesting, we definitely
>>> should participate more in the discussions.
>>> The main thing here is, the Spark community should make Spark pluggable
>>> in order to support other schedulers, not just for Volcano. It looks like
>>> this proposal is pushing really hard for adopting PodGroup, which isn't
>>> part of K8s yet, that to me is problematic.
>>>
>>> On Tue, Nov 30, 2021 at 9:21 AM Prasad Paravatha <
>>> prasad.parava...@gmail.com> wrote:
>>>
 This is a great feature/idea.
 I'd love to get involved in some form (testing and/or documentation).
 This could be my 1st contribution to Spark!

 On Tue, Nov 30, 2021 at