dongjoon-hyun commented on code in PR #910:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/910#discussion_r4136886719
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java:
##########
@@ -124,7 +124,13 @@ public ReconcileProgress reconcile(
Optional<ReconcileProgress> waiting =
releaseResources(context, statusRecorder, keepWorkload.isEmpty());
if (waiting.isPresent()) {
- return waiting.get();
+ // Pods which are stuck terminating are looked at again only as often
as a suspended
+ // cluster otherwise, while the Workload which they no longer hold is
released once its
+ // backoff elapses, when nothing else reconciles the cluster
+ return keepWorkload
+ .filter(backoff ->
backoff.compareTo(waiting.get().getRequeueAfterDuration()) < 0)
Review Comment:
Thank you, @peter-toth. I confirmed that only these two tests fail with
`.filter(backoff -> true)`, and added them in 6beb2301. I renamed the first one
to `failedReleaseIsRetriedWhileDeactivatedWorkloadIsKept`, since it is the
`Workload` which is deactivated, and let it pass the mock client to
`stubContext` directly.
##########
docs/spark_custom_resources.md:
##########
@@ -739,7 +739,10 @@ spec:
evicted `Workload` until then, so the operator then deletes it, and the
resource is queued again
with a new `Workload`. Unlike Kueue built-in integrations, which keep the
evicted
`Workload`, the new `Workload` is ordered in its queue like a newly created
one, and it starts
- over the retry counts of its admission checks.
+ over the retry counts of its admission checks and the requeue count of its
`PodsReadyTimeout`s.
+ So their backoff does not grow with each eviction, e.g. it stays at
+ `requeuingStrategy.backoffBaseSeconds` (60 seconds by default) for a
`PodsReadyTimeout`, and a
+ `requeuingStrategy.backoffLimitCount` above 0 never deactivates the
`Workload`.
Review Comment:
Thank you for the alternative. I'd like to keep the current design on
purpose, to stay decoupled from Kueue's internals. The operator writes only
`PodsReady`, and leaves the other `Workload` status fields, including
`requeueState`, to Kueue. The resulting limitation is documented in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
