Thank you, Yikun. I am working on SPARK-37809 <https://issues.apache.org/jira/browse/SPARK-37809> and SPARK-38310 <https://issues.apache.org/jira/browse/SPARK-38310>. They are the major stuff for the yunikorn part. Keep in mind we also need to add the documents. Thanks for building up the common things, great work.
On Wed, Feb 23, 2022 at 7:35 PM Yikun Jiang <yikunk...@gmail.com> wrote: > First, much thanks for all your help (Spark/Volcano/Yunikorn community) to > make this SPIP happen! > > Especially,@dongjoon-hyun @holdenk @william-wang @attilapiros @HyukjinKwon > @martin-g @yangwwei @tgravescs > > The SPIP is near the end of the stage. It can be said that it is beta > available at the basic level. > > I also draft a simple slide to show how to use and help you understand > what we have done: > > https://docs.google.com/presentation/d/1XDsTWPcsBe4PQ-1MlBwd9pRl8mySdziE_dJE6iATNw8 > > Below are also some recap to help you understand current implementation > and next step on SPIP: > > *# Existing work* > *## Basic part:* > - SPARK-36059 <https://issues.apache.org/jira/browse/SPARK-36059> *New > configuration:* ability to specify "schedulerName" in driver/executor for > Spark on K8S > - SPARK-37331 <https://issues.apache.org/jira/browse/SPARK-37331> *New > workflow：*ability to create pre-populated resources before driver pod > for Spark on K8S > - SPARK-37145 <https://issues.apache.org/jira/browse/SPARK-37145> *New > developer API:* support user feature step with configuration for Spark on > K8S > - *(reviewing)* *New Job Configurations* for Spark on K8S: > - SPARK-38188 <https://issues.apache.org/jira/browse/SPARK-38188>: > spark.kubernetes.job.queue > - SPARK-38187 <https://issues.apache.org/jira/browse/SPARK-38187>: > spark.kubernetes.job.[minCPU|minMemory] > - SPARK-38189 <https://issues.apache.org/jira/browse/SPARK-38189>: > spark.kubernetes.job.priorityClassName > > *## Volcano Part:* > - SPARK-37258 <https://issues.apache.org/jira/browse/SPARK-37258> *New > volcano extension* in kubernetes-client fabric8io/kubernetes-client#3579 > - SPARK-36061 <https://issues.apache.org/jira/browse/SPARK-36061> *New > profile: *-Pvolcano > - SPARK-36061 <https://issues.apache.org/jira/browse/SPARK-36061> *New > Feature Step:* VolcanoFeatureStep > - SPARK-36061 <https://issues.apache.org/jira/browse/SPARK-36061> *New > integration test:* > *- Passed on x86 and Arm64 (Linux on Huawei Kunpeng 920 and MacOS on > Apple Silicon M1).* > - Test basic volcano workflow > - Test all existing tests based on the volcano. > > *## Yunikorn Part:* > @yangwwei will also make the efforts for Yunikorn module feature step > since this week. > I will help to complete the yunikorn integration based on previous > experience. > > *# Next Plan* > There are also 3 main tasks to be completed before v3.3 code freeze: > 1. (reviewing) SPARK-38188 > <https://issues.apache.org/jira/browse/SPARK-38188>: Support queue > scheduling configuration > https://github.com/apache/spark/pull/35553 > 2. (reviewing) SPARK-38187 > <https://issues.apache.org/jira/browse/SPARK-38187>: Support resource > reservation (minCPU/minMemory configuration) > https://github.com/apache/spark/pull/35640 > 3. (reviewing) SPARK-38187 > <https://issues.apache.org/jira/browse/SPARK-38187>: Support priority > scheduling (priorityClass configuration): > https://issues.apache.org/jira/browse/SPARK-38189 > https://github.com/apache/spark/pull/35639 > 4. (WIP) SPARK-37809 <https://issues.apache.org/jira/browse/SPARK-37809>: > Yunikorn integration > > Also several misc work is gonna be completed before 3.3: > 1. Integrated volcano deploy into integration test (x86 and arm) > - Add it to spark kubernetes integration test once cross compile support: > https://github.com/volcano-sh/volcano/pull/1571 > 2. Complete doc and test guideline. > > Please feel free to contact me if you have any other concerns! Thanks! > > [1] https://issues.apache.org/jira/browse/SPARK-36057 >
