Peter Toth created SPARK-59758:
----------------------------------
Summary: Release the Kueue `Workload` of a terminated
`SparkApplication` regardless of the queue label
Key: SPARK-59758
URL: https://issues.apache.org/jira/browse/SPARK-59758
Project: Spark
Issue Type: Sub-task
Components: Kubernetes
Affects Versions: kubernetes-operator-1.1.0
Reporter: Peter Toth
SPARK-59754 removed the `kueue.x-k8s.io/queue-name` label check from the two
suspension paths, so a `Workload` created before the label was removed is
released. Two release paths in `AppCleanUpStep` still gate on the label and
keep the same leak:
* `AppCleanUpStep` releases the `Workload` of a terminating or restarting
attempt only `if (KueueWorkloadFactory.hasQueueName(application))`.
* `finishKueueWorkloadOfRetainedApp` records the Kueue `Finished` condition
only under the same check.
Measured on an application reaching `Succeeded`, with a mocked
`KueueWorkloadUtils`:
{noformat}
labeled, resourceRetainPolicy=Never -> releaseWorkload called
unlabeled, resourceRetainPolicy=Never -> neither called
labeled, resourceRetainPolicy=Always -> finishWorkload called
unlabeled, resourceRetainPolicy=Always -> neither called
{noformat}
So an application whose queue label was removed after its `Workload` was
admitted keeps that quota until its custom resource is deleted, which with the
default `ttlAfterStopMillis` of -1 may be never.
After SPARK-59754, dropping both checks is safe:
`KueueWorkloadUtils.deleteWorkloadOf` already treats a `403 Forbidden` as no
`Workload` when the owner has no queue label and the `Workload` informer is
disabled, so an operator that was never granted access to `Workload`s is
unaffected. `releaseWorkload` swallows every other failure, and
`finishWorkload` falls back to `releaseWorkload`.
The two admission gates, `ClusterInitStep.holdForKueueAdmission` and the
equivalent in `AppInitStep`, must keep the label check: an unlabeled resource
has no admission to request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
