peter-toth commented on code in PR #869:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/869#discussion_r4080663845
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppCleanUpStep.java:
##########
@@ -195,10 +195,13 @@ public ReconcileProgress reconcile(
context, statusRecorder, state,
Duration.ofMillis(requeueAfterMillis));
} else {
+ // The resources have been released above. An application retaining them
has returned
+ // already, so it cannot terminate as
`TerminatedWithoutReleaseResources` here even if the
+ // retain policy applies and no more attempt is made.
updatedStatus =
currentStatus.terminateOrRestart(
tolerations.getRestartConfig(),
- tolerations.getResourceRetainPolicy(),
+ ResourceRetainPolicy.Never,
Review Comment:
**Finding 1.** `docs/spark_custom_resources.md` documents this exact
configuration together with its final state, so the change makes that example
wrong.
I ran the doc's spec (`restartPolicy: OnFailure`, `maxRestartAttempts: 1`,
`resourceRetainPolicy: Always`) with a `Succeeded` final attempt through
`AppCleanUpStep.reconcile` on both arms. With
`tolerations.getResourceRetainPolicy()` the persisted final state is
`TerminatedWithoutReleaseResources`, with `ResourceRetainPolicy.Never` it is
`ResourceReleased`. So state `12` at `docs/spark_custom_resources.md:511-512`
becomes:
```yaml
"12":
currentStateSummary: ResourceReleased
```
Two sentences of prose around it go with it:
* `docs/spark_custom_resources.md:483-485` says that for an application
configured to restart, `resourceRetainPolicy`, `resourceRetainDurationMillis`
and `ttlAfterStopMillis` "would be applied only to the last attempt". After
this PR the first two are not applied at all, because every attempt including
the last force releases its resources. Only `ttlAfterStopMillis` still applies.
* `docs/spark_custom_resources.md:515` says "The retain policy only takes
effect after the final state `12`." For this spec it never takes effect.
`docs/architecture.md:158` also labels the `Failures -->
TerminatedWithoutReleaseResources` edge with "Retain Policy". That edge now
needs the restart condition too, something like `Retain Policy, No Restart`,
which the prose at `docs/architecture.md:188-190` already states.
The rest of the documentation is on your side here.
`docs/architecture.md:188-190` and the `ResourceRetainPolicy` javadoc at
`spark-operator-api/src/main/java/org/apache/spark/k8s/operator/spec/ResourceRetainPolicy.java:31-33`
both say resources are not retained when the application is configured to
restart. The promise at `docs/spark_custom_resources.md:484` is the outlier, so
correcting it is what matches the code. If it was meant as a feature instead,
retaining the final attempt is a different change and a different ticket.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppCleanUpStep.java:
##########
@@ -195,10 +195,13 @@ public ReconcileProgress reconcile(
context, statusRecorder, state,
Duration.ofMillis(requeueAfterMillis));
} else {
+ // The resources have been released above. An application retaining them
has returned
+ // already, so it cannot terminate as
`TerminatedWithoutReleaseResources` here even if the
+ // retain policy applies and no more attempt is made.
Review Comment:
**Finding 2.** This is the only production caller of `terminateOrRestart`,
and it now always passes `Never`, so the retain handling inside that method is
dead code.
`AppCleanUpStep:202` is the single callsite outside `ApplicationStatusTest`.
With `Never` fixed here, neither `ApplicationStatus.java:116-120` nor
`ApplicationStatus.java:175-179` can fire, and
`terminateAppWithoutReleaseResource` at `ApplicationStatus.java:293-299` has no
live caller. Every `TerminatedWithoutReleaseResources` a running operator
writes now comes from `AppCleanUpStep:136-139`.
I checked rather than assumed it. Deleting both `if` blocks and the private
method leaves the whole `:spark-operator` suite green, and exactly two tests
fail: `ApplicationStatusTest.testTerminateOrRestartWithoutRetry` and
`testTerminateOrRestartWithRetry`, the two that call the method directly with
`ResourceRetainPolicy.Always`. So the branch's only remaining coverage asserts
a state that no production path can reach, which is what makes it read as live.
Worth dropping the two blocks, `terminateAppWithoutReleaseResource`, and the
`resourceRetainPolicy` parameter, and adjusting those two assertions. That also
removes a duplicated predicate: `retainReleaseResourceForPolicyAndState` at
`AppCleanUpStep.java:386-393` and the two `if`s are the same test on the same
`currentState`, which is exactly why passing `Never` here is safe.
`ApplicationStatus` is published in `spark-operator-api`, so if dropping the
parameter is more than you want in this PR, a follow-up ticket is fine.
##########
spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppCleanUpStepTest.java:
##########
@@ -352,6 +354,61 @@ void
cleanupWithRetainPolicyWithoutQueueNameDoesNotFinishKueueWorkload() {
}
}
+ @Test
+ void cleanupWithRetainPolicyAndRestartConfigTerminatesAsResourceReleased() {
+ // The resources of an application configured to restart are force
released, so it terminates
+ // as `ResourceReleased` even when the retain policy applies and no more
attempt is made.
+ Map<RestartConfig, ApplicationStateSummary> cases =
+ Map.of(
+ // The restart policy does not restart a succeeded application
+
RestartConfig.builder().restartPolicy(RestartPolicy.OnFailure).build(),
+ ApplicationStateSummary.Succeeded,
+ // The restart limit is exceeded
+ RestartConfig.builder()
+ .restartPolicy(RestartPolicy.Always)
+ .maxRestartAttempts(0L)
+ .build(),
+ ApplicationStateSummary.Failed);
+ for (Map.Entry<RestartConfig, ApplicationStateSummary> entry :
cases.entrySet()) {
+ SparkAppStatusRecorder mockRecorder = mock(SparkAppStatusRecorder.class);
+ AppCleanUpStep routineCheck = new AppCleanUpStep();
+ SparkApplication app = new SparkApplication();
+ app.setMetadata(
+ new ObjectMetaBuilder()
+ .withName("app1")
+ .withNamespace("default")
+ .withLabels(Map.of(Constants.LABEL_QUEUE_NAME, "test-queue"))
+ .build());
+ app.setSpec(
+ ApplicationSpec.builder()
+ .applicationTolerations(
+ ApplicationTolerations.builder()
+ .resourceRetainPolicy(ResourceRetainPolicy.Always)
Review Comment:
**Finding 3.** Both cases hold `resourceRetainPolicy` at `Always`, so the
`OnFailure` half of the description is not covered.
The description states the fix for "`resourceRetainPolicy` `Always` (or
`OnFailure` on failure)". `retainReleaseResourceForPolicyAndState` treats
`OnFailure` plus a failure state identically to `Always`, so I expected the
same flip, and measured it: with `resourceRetainPolicy: OnFailure`,
`restartPolicy: Always`, `maxRestartAttempts: 0` and a `Failed` current state,
the persisted state is `TerminatedWithoutReleaseResources` before the change
and `ResourceReleased` after.
The retain policy is fixed outside the map, so covering it needs the policy
to vary per case. A helper keeps that short:
```java
private void assertTerminatesAsResourceReleased(
ResourceRetainPolicy retainPolicy,
RestartConfig restartConfig,
ApplicationStateSummary stateSummary) {
// the body of the current loop, with `retainPolicy` in place of
`ResourceRetainPolicy.Always`
}
@Test
void cleanupWithRetainPolicyAndRestartConfigTerminatesAsResourceReleased()
{
// The restart policy does not restart a succeeded application
assertTerminatesAsResourceReleased(
ResourceRetainPolicy.Always,
RestartConfig.builder().restartPolicy(RestartPolicy.OnFailure).build(),
ApplicationStateSummary.Succeeded);
// The restart limit is exceeded
assertTerminatesAsResourceReleased(
ResourceRetainPolicy.Always,
RestartConfig.builder().restartPolicy(RestartPolicy.Always).maxRestartAttempts(0L).build(),
ApplicationStateSummary.Failed);
// `OnFailure` retains on a failure, so the resources are force released
the same way
assertTerminatesAsResourceReleased(
ResourceRetainPolicy.OnFailure,
RestartConfig.builder().restartPolicy(RestartPolicy.Always).maxRestartAttempts(0L).build(),
ApplicationStateSummary.Failed);
}
```
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppCleanUpStep.java:
##########
@@ -195,10 +195,13 @@ public ReconcileProgress reconcile(
context, statusRecorder, state,
Duration.ofMillis(requeueAfterMillis));
} else {
+ // The resources have been released above. An application retaining them
has returned
+ // already, so it cannot terminate as
`TerminatedWithoutReleaseResources` here even if the
+ // retain policy applies and no more attempt is made.
updatedStatus =
currentStatus.terminateOrRestart(
tolerations.getRestartConfig(),
- tolerations.getResourceRetainPolicy(),
+ ResourceRetainPolicy.Never,
stateUpdateMessage,
Review Comment:
**Finding 4.** The message this now lands on `ResourceReleased` runs two
sentences together with no space between them.
Measured on the restart-limit case: `The maximum number of restart attempts
(0) has been exceeded.Application is configured to restart, resources created
in current attempt would be force released.` The join is
`ApplicationStatus.java:170`, `stateMessage += stateMessageOverride`.
Pre-existing, and your change actually removes one of the two such joins
that were there before, since `terminateAppWithoutReleaseResource` used to glue
a third sentence onto the front. The one that is left is now the whole message
a user reads on the terminal state, so it is worth the space:
```java
if (stateMessageOverride != null && !stateMessageOverride.isEmpty()) {
stateMessage += " " + stateMessageOverride;
}
```
Asserting the message in the new test would pin it, and would also record
that the misleading "Application is terminated without releasing resources as
configured." prefix is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
