[
https://issues.apache.org/jira/browse/SPARK-59769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-59769:
----------------------------------
Summary: Apply the flavors of an admitted Kueue `Workload` to a resource
whose queue label was removed (was: Keep the admitted Kueue flavors when the
queue label is removed after the admission)
> Apply the flavors of an admitted Kueue `Workload` to a resource whose queue
> label was removed
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-59769
> URL: https://issues.apache.org/jira/browse/SPARK-59769
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.1.0
> Reporter: Dongjoon Hyun
> Priority: Major
>
> When the Kueue integration is enabled, removing the
> `kueue.x-k8s.io/queue-name` label of a `SparkApplication` or `SparkCluster`
> after its `Workload` is admitted can remove the node selector and tolerations
> that Kueue assigned from its resources.
> `KueueWorkloadUtils.releaseDequeuedWorkload` keeps an admitted `Workload`,
> but it returns without calling `applyAdmittedFlavors`. `AppInitStep` and
> `ClusterInitStep` then apply their resources again without the flavors:
> - `SparkApplication`: the executor pod template ConfigMap in the driver
> pre-resources is applied again. Executors created later lose the node
> selector and tolerations.
> - `SparkCluster`: the master and worker `StatefulSet`s are applied again with
> a changed pod template, which restarts the master and worker pods.
> The labeled path already handles this case. When the driver or master already
> exists, it calls `applyAdmittedFlavors`.
> This can happen in the following case:
> 1. The `Workload` is admitted, and the driver or master is created.
> 2. The status update to `DriverRequested` or `RunningHealthy` fails, so the
> next reconciliation runs the init step again.
> 3. The queue label is removed before that reconciliation.
> This issue exists before SPARK-59766, and it was found in the review of
> https://github.com/apache/spark-kubernetes-operator/pull/881#discussion_r4096444681.
> Proposed fix: apply the admitted flavors again when the `Workload` is
> admitted.
> ```java
> if (workload.isEmpty()) {
> return Optional.empty();
> }
> if (isAdmitted(workload.get())) {
> return applyAdmittedFlavors(context);
> }
> ```
> Trade-off: `applyAdmittedFlavors` throws `IllegalArgumentException` on a node
> selector conflict, and `ClusterInitStep` turns it into the terminal
> `SchedulingFailure`. The labeled path behaves the same way, but this path
> would then fail instead of changing the pod templates without notice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
