[
https://issues.apache.org/jira/browse/SPARK-51250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-51250:
-----------------------------------
Labels: pull-request-available (was: )
> Add Support for K8s PriorityClass Configuration for spark-submit via spark
> config
> ---------------------------------------------------------------------------------
>
> Key: SPARK-51250
> URL: https://issues.apache.org/jira/browse/SPARK-51250
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 4.1.0
> Reporter: Zemin Piao
> Priority: Major
> Labels: pull-request-available
>
> h2. Context
> Currently when running spark on k8s, with spark-submit approach, in order to
> specify priority class name, the only possible way is to specify in the pod
> template. In a cluster with numerous priority classes, it means multiple pod
> template files are to be used, each of which has variant priority class name
> only.
> One could dynamically generate the pod templates based on priority classes,
> however this introduces additional overhead on user side (e.g. building a
> process to dynamically generate templates just for priority class name,
> making sure dynamically generated templates are still valid)
> Furthermore, for SPARK on YARN there is a configuration `spark.yarn.priority`
> that is flexible and easy to use for users. It is good to achieve the similar
> level of simplicity for users moving from SPARK on YARN to SPARK on K8s.
> h2. Proposal
> * Introduce a new spark configuration
> {*}spark.kubernetes.priorityclass.name{*}.
> * Create a new FeatureStep class that passes configured priority class name
> to driver and executor pods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
