[
https://issues.apache.org/jira/browse/YUNIKORN-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298466#comment-17298466
]
Weiwei Yang commented on YUNIKORN-558:
--------------------------------------
Thanks [~phoenixjiangnan] for creating this issue. YK can work with
spark-operator today seamlessly. As long as the admission controller is enabled
(that adds the schedulerName upon admission), YK automatically recognizes the
Spark jobs and schedules them as needed. Deeper integration with Spark operator
means a better app lifecycle mgmt, and the support of gang scheduling. A rough
thought about the work to be done including:
# [dev] Manage SparkApplication/ScheduledSparkApplication end-to-end lifecycle
with app-mgmt plugin
# [dev] Support YK gang scheduling by properly setting up taskGroups info in
the Spark app CRD
# [test] e2e test with Spark operator w/ or w/o gang scheduling
# [doc] User instructions
cc folks who might be interested too: [~leftnoteasy],
[[email protected]], [~wilfreds], [~kmarton], [~ayubpathan], [~rozhang],
[~Zhou Kang], [~Huang Ting Yao].
> Integrate YuniKorn with Spark K8S Operator
> ------------------------------------------
>
> Key: YUNIKORN-558
> URL: https://issues.apache.org/jira/browse/YUNIKORN-558
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - scheduler
> Reporter: Bowen Li
> Priority: Major
>
> Integrate YuniKorn with Spark K8S Operator
> [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator]
>
> Using Spark K8S Operator is a standard way to run Spark applications on K8S.
> The native integration will make using Yunikorn more easier to use and get
> more exposure to OSS community
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
