This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new c4db4c35 [SPARK-60512] Fix `kueue` E2E test flakiness by making
`QuotaReserved` assertions nil-safe
c4db4c35 is described below
commit c4db4c350ee2248511be7dec778a75a076c2d877
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 09:48:17 2026 -0700
[SPARK-60512] Fix `kueue` E2E test flakiness by making `QuotaReserved`
assertions nil-safe
### What changes were proposed in this pull request?
This PR aims to fix the flakiness of the `kueue` E2E test by making the two
`QuotaReserved` assertions of the
`pod-set-updates-of-admission-checks-reach-the-pods` step nil-safe.
```diff
- (conditions[?type == 'QuotaReserved' && status == 'True'] |
length()): 1
+ (length(conditions[?type == 'QuotaReserved' && status ==
'True'] || `[]`)): 1
```
While a `Workload` has a `status` without `conditions`, the new expression
evaluates to `0`. This is a mismatch, which Chainsaw retries, instead of an
evaluation error. It is the same idiom as `contains(message || '', ...)` in the
same file.
### Why are the changes needed?
The step, which SPARK-60042
(https://github.com/apache/spark-kubernetes-operator/pull/951) added, failed
once on `main`.
-
https://github.com/apache/spark-kubernetes-operator/actions/runs/39643093114/job/119375108125
```
| 17:47:06 | spark-operator-kueue |
pod-set-updates-of-admission-checks-reach-the-pods | ASSERT | RUN |
kueue.x-k8s.io/v1beta2/Workload
default/sparkcluster-spark-cluster-kueue-admission-check-test
| 17:47:07 | spark-operator-kueue |
pod-set-updates-of-admission-checks-reach-the-pods | ASSERT | ERROR |
kueue.x-k8s.io/v1beta2/Workload
default/sparkcluster-spark-cluster-kueue-admission-check-test
=== ERROR
status.(conditions[?type == 'QuotaReserved' && status == 'True'] |
length()): Internal error: invalid type for: <nil>, expected:
[]functions.JpType{"string", "array", "object"}
```
The assertion failed 72ms after it started, although its timeout is `60s`.
The `Workload`, which the `catch` block described right after, already had
`QuotaReserved=True`.
- For a `Workload` whose `ClusterQueue` has admission checks, Kueue adds
the `Pending` admission check to `status.admissionChecks` in a separate status
update
([reconcileSyncAdmissionChecks](https://github.com/kubernetes-sigs/kueue/blob/v0.20.0/pkg/controller/core/workload_controller.go#L1024-L1037)
of [Kueue
v0.20.0](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.20.0)
(2026-09-30)). This update can land before the scheduler reserves the quota and
adds the first condition. [...]
- In this window, `conditions[?...]` is `null`, and `length(null)` is a
JMESPath error.
- Chainsaw retries an assertion only on a mismatch. An evaluation error
stops the assertion at once
([assert/operation.go](https://github.com/kyverno/chainsaw/blob/v0.2.15/pkg/engine/operations/assert/operation.go#L109-L112)
of [Chainsaw
v0.2.15](https://github.com/kyverno/chainsaw/releases/tag/v0.2.15)
(2026-05-06),
[assertion.go](https://github.com/kyverno/kyverno-json/blob/b294ee72a2bf180791daebee6baf4f6bf8437f45/pkg/core/assertion/assertion.go#L89-L93)
of `kyverno-json`). In contr [...]
Only this step uses an admission check. The other `conditions[?...] |
length()` assertions of `kueue` and `kueue-pods-ready` check `Workload`s whose
`ClusterQueue` has no admission check, and their first status update already
has the conditions. The `tolerations[?...] | length()` assertions check pods
and `StatefulSet`s, which have tolerations when they are created. So, this PR
keeps them as they are.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
1. Reproduced the failure with Chainsaw v0.2.15 on a custom resource
without a controller. Its `status` has only `admissionChecks`, and a script
adds `QuotaReserved=True` to its `conditions` 3 seconds later.
| Expression | `status` at first | Result |
|---|---|---|
| Before | `admissionChecks` only | Fails in 1 second with the same
`Internal error`. |
| After | `admissionChecks` only | Passes in 3 seconds, when the condition
is added. |
| After | `admissionChecks` only, and the condition is never added | Fails
at the timeout with `Invalid value: 0: Expected value: 1`. |
| Before | None, and then the whole `status` | Passes in 3 seconds, since
`status: Required value` is retried. |
2. Ran the `kueue` E2E test on minikube in the same way as the CI
(Kubernetes v1.37.0, Kueue v0.20.0 with the shortened `waitForPodsReady`, and
the operator image of this branch). It passed. A watch of the `Workload`s
showed that only the two `Workload`s of this step had a `status` without
`conditions`, for about 5ms each. The other 11 `Workload`s had conditions in
their first status.
3. Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #982 from dongjoon-hyun/SPARK-60512.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
tests/e2e/kueue/chainsaw-test.yaml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/e2e/kueue/chainsaw-test.yaml
b/tests/e2e/kueue/chainsaw-test.yaml
index 5c00d636..fb77460b 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -1009,7 +1009,7 @@ spec:
name: sparkapplication-spark-job-kueue-admission-check-test
namespace: default
status:
- (conditions[?type == 'QuotaReserved' && status == 'True'] |
length(@)): 1
+ (length(conditions[?type == 'QuotaReserved' && status == 'True']
|| `[]`)): 1
admissionChecks:
- name: ($ADMISSION_CHECK)
state: Pending
@@ -1022,7 +1022,7 @@ spec:
name: sparkcluster-spark-cluster-kueue-admission-check-test
namespace: default
status:
- (conditions[?type == 'QuotaReserved' && status == 'True'] |
length(@)): 1
+ (length(conditions[?type == 'QuotaReserved' && status == 'True']
|| `[]`)): 1
admissionChecks:
- name: ($ADMISSION_CHECK)
state: Pending
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]