dongjoon-hyun opened a new pull request, #866: URL: https://github.com/apache/spark-kubernetes-operator/pull/866
### What changes were proposed in this pull request? This PR aims to stop `ClusterUnknownStateStep` from appending a new `Failed` state to a `SparkCluster` that is already `Failed`. For such a cluster, it now completes the reconciliation without persisting any status and without requeueing. The existing `SchedulingFailure -> Failed` transition is unchanged. ### Why are the changes needed? Currently, a failed `SparkCluster` falls into an infinite status-write loop. - `ClusterTerminatedStep` stops the reconciliation only for `ResourceReleased`, which is never set for a cluster. So a `SchedulingFailure` or `Failed` cluster always reaches `ClusterUnknownStateStep`. - `ClusterUnknownStateStep` always appends a new `Failed` state, persists it, and returns `completeAndImmediateRequeue()`. - Each appended state has a new history key and a new timestamp, so `StatusRecorder` never skips the write as unchanged, and every reconciliation issues a `/status` PUT. As a result, any `ClusterInitStep` failure (for example, a transient 429/5xx/timeout on server-side apply, or a Kueue spec conflict) makes the cluster loop at the JOSDK rate limit (about 20 status PUTs per minute), emitting `Failed` events. `stateTransitionHistory` grows without bound until the object approaches the etcd size limit. The only recovery is to delete the CR. ### Does this PR introduce _any_ user-facing change? Yes, this fixes a bug. A failed `SparkCluster` no longer keeps growing `status.stateTransitionHistory` or emitting repeated `Failed` status updates. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
