Dennis-Mircea commented on code in PR #1248:
URL: 
https://github.com/apache/flink-kubernetes-operator/pull/1248#discussion_r4585862247


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkSessionJobController.java:
##########
@@ -134,14 +134,19 @@ public DeleteControl cleanup(FlinkSessionJob sessionJob, 
Context josdkContext) {
         if (canaryResourceManager.handleCanaryResourceDeletion(sessionJob)) {
             return DeleteControl.defaultDelete();
         }
+        statusRecorder.updateStatusFromCache(sessionJob);
+        if 
(sessionJob.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
+            LOG.info("Nothing was deployed yet, nothing to clean up");
+            onCleanupFinished(sessionJob);
+            return DeleteControl.defaultDelete();
+        }
         eventRecorder.triggerEvent(

Review Comment:
   Same here.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java:
##########
@@ -94,14 +94,19 @@ public DeleteControl cleanup(FlinkDeployment flinkApp, 
Context<FlinkDeployment>
         if (canaryResourceManager.handleCanaryResourceDeletion(flinkApp)) {
             return DeleteControl.defaultDelete();
         }
+        statusRecorder.updateStatusFromCache(flinkApp);
+        if 
(flinkApp.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
+            LOG.info("Nothing was deployed yet, nothing to clean up");
+            onCleanupFinished(flinkApp);
+            return DeleteControl.defaultDelete();
+        }
         eventRecorder.triggerEvent(

Review Comment:
   Why is the " Cleaning up FlinkDeployment " event placed after the first 
deployment reconcile attempt? In my opinion, this event should always be placed 
before because it targets the FlinkDeployment CRD, not the Flink Cluster, which 
is not deployed in this case. Having this event placed before it will also let 
the users know that a cleanup was initiated for a concrete CRD.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java:
##########
@@ -110,17 +115,28 @@ public DeleteControl cleanup(FlinkDeployment flinkApp, 
Context<FlinkDeployment>
             LOG.error("Error while observing for cleanup", err);
         }
 
-        var deleteControl = 
reconcilerFactory.getOrCreate(flinkApp).cleanup(ctx);
+        DeleteControl deleteControl;
+        try {
+            deleteControl = 
reconcilerFactory.getOrCreate(flinkApp).cleanup(ctx);
+        } catch (RuntimeException e) {
+            // Keep the status changes of the failed attempt, e.g. a completed 
savepoint
+            statusRecorder.patchAndCacheStatus(flinkApp, 
ctx.getKubernetesClient());
+            throw e;
+        }
         if (deleteControl.isRemoveFinalizer()) {
-            
flinkApp.getStatus().setLifecycleState(ResourceLifecycleState.DELETED);
-            statusRecorder.cleanupForDeletion(flinkApp);
-            ctxFactory.cleanup(flinkApp);
+            onCleanupFinished(flinkApp);
         } else {
             statusRecorder.patchAndCacheStatus(flinkApp, 
ctx.getKubernetesClient());
         }
         return deleteControl;
     }
 
+    private void onCleanupFinished(FlinkDeployment flinkApp) {
+        flinkApp.getStatus().setLifecycleState(ResourceLifecycleState.DELETED);

Review Comment:
   I checked out and saw that, actually, the `DELETED` lifecycle state is never 
materialized as part of the status or metrics, and it is only audited and sent 
to the listener plugins. It would be good if we could send an update first, at 
least on the metrics side, to mark this state as reached before removal.
   
   I know that is not fully in the scope of the PR, just wanted to point it 
out. We can tackle it as part of a separate ticket.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -265,24 +266,32 @@ protected JobUpgrade 
getJobUpgrade(FlinkResourceContext<CR> ctx, Configuration d
             return JobUpgrade.savepoint(true);
         }
 
-        if (ReconciliationUtils.isJobCancelling(status)) {
-            LOG.info("Cancellation is in progress. Waiting for cancelled 
state.");
-            return JobUpgrade.pendingCancellation();
-        }
-
         boolean running = ReconciliationUtils.isJobRunning(status);
         boolean versionChanged =
                 flinkVersionChanged(
                         ReconciliationUtils.getDeployedSpec(resource), 
resource.getSpec());
 
+        if (ReconciliationUtils.isJobCancelling(status)) {
+            boolean fallbackAllowed =
+                    !versionChanged
+                            && (upgradeMode == UpgradeMode.LAST_STATE
+                                    || lastStateFallbackEnabled(deployConfig));
+            if (!fallbackAllowed || !cancellationTimedOut(ctx)) {
+                LOG.info("Cancellation is in progress. Waiting for cancelled 
state.");
+                return JobUpgrade.pendingCancellation();
+            }
+            LOG.warn(
+                    "Job cancellation did not complete within {}",
+                    TimeUtils.formatWithHighestUnit(
+                            
ctx.getOperatorConfig().getFlinkShutdownClusterTimeout()));

Review Comment:
   Same here about the timeout in use.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java:
##########
@@ -289,7 +291,58 @@ protected abstract boolean reconcileSpecChange(
     @Override
     public DeleteControl cleanup(FlinkResourceContext<CR> ctx) {
         autoscaler.cleanup(ctx.getJobAutoScalerContext());

Review Comment:
   I'd also place this autoscaler cleanup inside the try-catch block, because 
currently it can bypass the newly added logic.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkSessionJobController.java:
##########
@@ -134,14 +134,19 @@ public DeleteControl cleanup(FlinkSessionJob sessionJob, 
Context josdkContext) {
         if (canaryResourceManager.handleCanaryResourceDeletion(sessionJob)) {
             return DeleteControl.defaultDelete();
         }
+        statusRecorder.updateStatusFromCache(sessionJob);
+        if 
(sessionJob.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
+            LOG.info("Nothing was deployed yet, nothing to clean up");
+            onCleanupFinished(sessionJob);
+            return DeleteControl.defaultDelete();
+        }
         eventRecorder.triggerEvent(
                 sessionJob,
                 EventRecorder.Type.Normal,
                 EventRecorder.Reason.Cleanup,
                 EventRecorder.Component.Operator,
                 "Cleaning up FlinkSessionJob",
                 josdkContext.getClient());
-        statusRecorder.updateStatusFromCache(sessionJob);

Review Comment:
   Why to have this removal in place? Without this we may risk to act on a 
stale status.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/SessionJobReconciler.java:
##########
@@ -192,25 +192,32 @@ public DeleteControl 
cleanupInternal(FlinkResourceContext<FlinkSessionJob> ctx)
         // Only check for pending cancellation once the cluster is ready, so 
that other deletion
         // paths (cluster missing, deleting, unhealthy without HA) are not 
blocked.
         if (ReconciliationUtils.isJobCancelling(status)) {
+            if (cancellationTimedOut(ctx)) {
+                eventRecorder.triggerEvent(

Review Comment:
   Looks like this event is triggered only for session jobs. It will be great 
to have the application jobs in sync with this implementation.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java:
##########
@@ -94,14 +94,19 @@ public DeleteControl cleanup(FlinkDeployment flinkApp, 
Context<FlinkDeployment>
         if (canaryResourceManager.handleCanaryResourceDeletion(flinkApp)) {
             return DeleteControl.defaultDelete();
         }
+        statusRecorder.updateStatusFromCache(flinkApp);
+        if 
(flinkApp.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {

Review Comment:
   Besides having the argument for placing this code block and this check after 
the event triggering, I even think that this check should be placed under the 
observation flow that happens at line 113, mainly because if the observer sees 
the reconciliationStatus.state in UPGRADING with nothing deployed, it clears 
the `lastReconciledSpec`. This flow together with the removal that this PR is 
making at `ApplicationReconciler` line 440 will throw an exception which will 
be wrongly labelled as CleanupFailed, which isn't true.
   
   In my opinion there are 2 paths here to follow:
   1. Rollback the `status.getReconciliationStatus().isBeforeFirstDeployment()` 
within `ApplicationReconciler` and `SessionJobReconciler` + add the same guard 
for `SessionReconciler`. This will align all the reconciler implementations, 
and this code block can be fully dropped. 
   2. Keep the removals and place this check under observe call .
   
   The advantage of option 1 is that the check lives in the layer where it 
belongs, where this assumption actually holds. It also ensures that leftovers 
specific to each deployment mode are cleaned up in the way that mode requires. 
Option 2 relies only on `lastReconciledSpec` being null, which doesn’t 
guarantee that resources such as an autoscaler ConfigMap have been removed, nor 
does it account for other resources added in the future that could end up as 
leftovers.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -331,6 +340,22 @@ protected JobUpgrade 
getJobUpgrade(FlinkResourceContext<CR> ctx, Configuration d
         return JobUpgrade.unavailable();
     }
 
+    private static boolean lastStateFallbackEnabled(Configuration 
deployConfig) {
+        return deployConfig.get(
+                
KubernetesOperatorConfigOptions.OPERATOR_JOB_UPGRADE_LAST_STATE_FALLBACK_ENABLED);
+    }
+
+    protected boolean cancellationTimedOut(FlinkResourceContext<CR> ctx) {
+        var updateTime = 
ctx.getResource().getStatus().getJobStatus().getUpdateTime();
+        return updateTime != null
+                && clock.instant()
+                        .isAfter(
+                                
Instant.ofEpochMilli(Long.parseLong(updateTime))
+                                        .plus(
+                                                ctx.getOperatorConfig()
+                                                        
.getFlinkShutdownClusterTimeout()));

Review Comment:
   Using this timeout for job-level cancellation doesn’t feel right to me, 
since it’s currently used for cluster-level deletion. For the job level, I’d 
rather introduce a separate timeout owned by the operator, the same way 
`resource.cleanup.timeout` is. I’d also rename the existing one to 
cluster.cleanup.timeout and add a sibling `job.cleanup.timeout`. Alternatively, 
we could fall back on Flink’s core `task.cancellation.timeout`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to