This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git


The following commit(s) were added to refs/heads/main by this push:
     new c4db4c35 [SPARK-60512] Fix `kueue` E2E test flakiness by making 
`QuotaReserved` assertions nil-safe
c4db4c35 is described below

commit c4db4c350ee2248511be7dec778a75a076c2d877
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 09:48:17 2026 -0700

    [SPARK-60512] Fix `kueue` E2E test flakiness by making `QuotaReserved` 
assertions nil-safe
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix the flakiness of the `kueue` E2E test by making the two 
`QuotaReserved` assertions of the 
`pod-set-updates-of-admission-checks-reach-the-pods` step nil-safe.
    
    ```diff
    -            (conditions[?type == 'QuotaReserved' && status == 'True'] | 
length()): 1
    +            (length(conditions[?type == 'QuotaReserved' && status == 
'True'] || `[]`)): 1
    ```
    
    While a `Workload` has a `status` without `conditions`, the new expression 
evaluates to `0`. This is a mismatch, which Chainsaw retries, instead of an 
evaluation error. It is the same idiom as `contains(message || '', ...)` in the 
same file.
    
    ### Why are the changes needed?
    
    The step, which SPARK-60042 
(https://github.com/apache/spark-kubernetes-operator/pull/951) added, failed 
once on `main`.
    - 
https://github.com/apache/spark-kubernetes-operator/actions/runs/39643093114/job/119375108125
    
    ```
    | 17:47:06 | spark-operator-kueue | 
pod-set-updates-of-admission-checks-reach-the-pods | ASSERT | RUN   | 
kueue.x-k8s.io/v1beta2/Workload  
default/sparkcluster-spark-cluster-kueue-admission-check-test
    | 17:47:07 | spark-operator-kueue | 
pod-set-updates-of-admission-checks-reach-the-pods | ASSERT | ERROR | 
kueue.x-k8s.io/v1beta2/Workload  
default/sparkcluster-spark-cluster-kueue-admission-check-test
        === ERROR
        status.(conditions[?type == 'QuotaReserved' && status == 'True'] | 
length()): Internal error: invalid type for: <nil>, expected: 
[]functions.JpType{"string", "array", "object"}
    ```
    
    The assertion failed 72ms after it started, although its timeout is `60s`. 
The `Workload`, which the `catch` block described right after, already had 
`QuotaReserved=True`.
    
    - For a `Workload` whose `ClusterQueue` has admission checks, Kueue adds 
the `Pending` admission check to `status.admissionChecks` in a separate status 
update 
([reconcileSyncAdmissionChecks](https://github.com/kubernetes-sigs/kueue/blob/v0.20.0/pkg/controller/core/workload_controller.go#L1024-L1037)
 of [Kueue 
v0.20.0](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.20.0) 
(2026-09-30)). This update can land before the scheduler reserves the quota and 
adds the first condition.  [...]
    - In this window, `conditions[?...]` is `null`, and `length(null)` is a 
JMESPath error.
    - Chainsaw retries an assertion only on a mismatch. An evaluation error 
stops the assertion at once 
([assert/operation.go](https://github.com/kyverno/chainsaw/blob/v0.2.15/pkg/engine/operations/assert/operation.go#L109-L112)
 of [Chainsaw 
v0.2.15](https://github.com/kyverno/chainsaw/releases/tag/v0.2.15) 
(2026-05-06), 
[assertion.go](https://github.com/kyverno/kyverno-json/blob/b294ee72a2bf180791daebee6baf4f6bf8437f45/pkg/core/assertion/assertion.go#L89-L93)
 of `kyverno-json`). In contr [...]
    
    Only this step uses an admission check. The other `conditions[?...] | 
length()` assertions of `kueue` and `kueue-pods-ready` check `Workload`s whose 
`ClusterQueue` has no admission check, and their first status update already 
has the conditions. The `tolerations[?...] | length()` assertions check pods 
and `StatefulSet`s, which have tolerations when they are created. So, this PR 
keeps them as they are.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a test-only change.
    
    ### How was this patch tested?
    
    1. Reproduced the failure with Chainsaw v0.2.15 on a custom resource 
without a controller. Its `status` has only `admissionChecks`, and a script 
adds `QuotaReserved=True` to its `conditions` 3 seconds later.
    
    | Expression | `status` at first | Result |
    |---|---|---|
    | Before | `admissionChecks` only | Fails in 1 second with the same 
`Internal error`. |
    | After | `admissionChecks` only | Passes in 3 seconds, when the condition 
is added. |
    | After | `admissionChecks` only, and the condition is never added | Fails 
at the timeout with `Invalid value: 0: Expected value: 1`. |
    | Before | None, and then the whole `status` | Passes in 3 seconds, since 
`status: Required value` is retried. |
    
    2. Ran the `kueue` E2E test on minikube in the same way as the CI 
(Kubernetes v1.37.0, Kueue v0.20.0 with the shortened `waitForPodsReady`, and 
the operator image of this branch). It passed. A watch of the `Workload`s 
showed that only the two `Workload`s of this step had a `status` without 
`conditions`, for about 5ms each. The other 11 `Workload`s had conditions in 
their first status.
    
    3. Pass the CIs.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #982 from dongjoon-hyun/SPARK-60512.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 tests/e2e/kueue/chainsaw-test.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/e2e/kueue/chainsaw-test.yaml 
b/tests/e2e/kueue/chainsaw-test.yaml
index 5c00d636..fb77460b 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -1009,7 +1009,7 @@ spec:
             name: sparkapplication-spark-job-kueue-admission-check-test
             namespace: default
           status:
-            (conditions[?type == 'QuotaReserved' && status == 'True'] | 
length(@)): 1
+            (length(conditions[?type == 'QuotaReserved' && status == 'True'] 
|| `[]`)): 1
             admissionChecks:
             - name: ($ADMISSION_CHECK)
               state: Pending
@@ -1022,7 +1022,7 @@ spec:
             name: sparkcluster-spark-cluster-kueue-admission-check-test
             namespace: default
           status:
-            (conditions[?type == 'QuotaReserved' && status == 'True'] | 
length(@)): 1
+            (length(conditions[?type == 'QuotaReserved' && status == 'True'] 
|| `[]`)): 1
             admissionChecks:
             - name: ($ADMISSION_CHECK)
               state: Pending


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to