Dennis-Mircea commented on code in PR #1248:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/1248#discussion_r4585862247
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkSessionJobController.java:
##########
@@ -134,14 +134,19 @@ public DeleteControl cleanup(FlinkSessionJob sessionJob,
Context josdkContext) {
if (canaryResourceManager.handleCanaryResourceDeletion(sessionJob)) {
return DeleteControl.defaultDelete();
}
+ statusRecorder.updateStatusFromCache(sessionJob);
+ if
(sessionJob.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
+ LOG.info("Nothing was deployed yet, nothing to clean up");
+ onCleanupFinished(sessionJob);
+ return DeleteControl.defaultDelete();
+ }
eventRecorder.triggerEvent(
Review Comment:
Same here.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java:
##########
@@ -94,14 +94,19 @@ public DeleteControl cleanup(FlinkDeployment flinkApp,
Context<FlinkDeployment>
if (canaryResourceManager.handleCanaryResourceDeletion(flinkApp)) {
return DeleteControl.defaultDelete();
}
+ statusRecorder.updateStatusFromCache(flinkApp);
+ if
(flinkApp.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
+ LOG.info("Nothing was deployed yet, nothing to clean up");
+ onCleanupFinished(flinkApp);
+ return DeleteControl.defaultDelete();
+ }
eventRecorder.triggerEvent(
Review Comment:
Why is the " Cleaning up FlinkDeployment " event placed after the first
deployment reconcile attempt? In my opinion, this event should always be placed
before because it targets the FlinkDeployment CRD, not the Flink Cluster, which
is not deployed in this case. Having this event placed before it will also let
the users know that a cleanup was initiated for a concrete CRD.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java:
##########
@@ -110,17 +115,28 @@ public DeleteControl cleanup(FlinkDeployment flinkApp,
Context<FlinkDeployment>
LOG.error("Error while observing for cleanup", err);
}
- var deleteControl =
reconcilerFactory.getOrCreate(flinkApp).cleanup(ctx);
+ DeleteControl deleteControl;
+ try {
+ deleteControl =
reconcilerFactory.getOrCreate(flinkApp).cleanup(ctx);
+ } catch (RuntimeException e) {
+ // Keep the status changes of the failed attempt, e.g. a completed
savepoint
+ statusRecorder.patchAndCacheStatus(flinkApp,
ctx.getKubernetesClient());
+ throw e;
+ }
if (deleteControl.isRemoveFinalizer()) {
-
flinkApp.getStatus().setLifecycleState(ResourceLifecycleState.DELETED);
- statusRecorder.cleanupForDeletion(flinkApp);
- ctxFactory.cleanup(flinkApp);
+ onCleanupFinished(flinkApp);
} else {
statusRecorder.patchAndCacheStatus(flinkApp,
ctx.getKubernetesClient());
}
return deleteControl;
}
+ private void onCleanupFinished(FlinkDeployment flinkApp) {
+ flinkApp.getStatus().setLifecycleState(ResourceLifecycleState.DELETED);
Review Comment:
I checked out and saw that, actually, the `DELETED` lifecycle state is never
materialized as part of the status or metrics, and it is only audited and sent
to the listener plugins. It would be good if we could send an update first, at
least on the metrics side, to mark this state as reached before removal.
I know that is not fully in the scope of the PR, just wanted to point it
out. We can tackle it as part of a separate ticket.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -265,24 +266,32 @@ protected JobUpgrade
getJobUpgrade(FlinkResourceContext<CR> ctx, Configuration d
return JobUpgrade.savepoint(true);
}
- if (ReconciliationUtils.isJobCancelling(status)) {
- LOG.info("Cancellation is in progress. Waiting for cancelled
state.");
- return JobUpgrade.pendingCancellation();
- }
-
boolean running = ReconciliationUtils.isJobRunning(status);
boolean versionChanged =
flinkVersionChanged(
ReconciliationUtils.getDeployedSpec(resource),
resource.getSpec());
+ if (ReconciliationUtils.isJobCancelling(status)) {
+ boolean fallbackAllowed =
+ !versionChanged
+ && (upgradeMode == UpgradeMode.LAST_STATE
+ || lastStateFallbackEnabled(deployConfig));
+ if (!fallbackAllowed || !cancellationTimedOut(ctx)) {
+ LOG.info("Cancellation is in progress. Waiting for cancelled
state.");
+ return JobUpgrade.pendingCancellation();
+ }
+ LOG.warn(
+ "Job cancellation did not complete within {}",
+ TimeUtils.formatWithHighestUnit(
+
ctx.getOperatorConfig().getFlinkShutdownClusterTimeout()));
Review Comment:
Same here about the timeout in use.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java:
##########
@@ -289,7 +291,58 @@ protected abstract boolean reconcileSpecChange(
@Override
public DeleteControl cleanup(FlinkResourceContext<CR> ctx) {
autoscaler.cleanup(ctx.getJobAutoScalerContext());
Review Comment:
I'd also place this autoscaler cleanup inside the try-catch block, because
currently it can bypass the newly added logic.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkSessionJobController.java:
##########
@@ -134,14 +134,19 @@ public DeleteControl cleanup(FlinkSessionJob sessionJob,
Context josdkContext) {
if (canaryResourceManager.handleCanaryResourceDeletion(sessionJob)) {
return DeleteControl.defaultDelete();
}
+ statusRecorder.updateStatusFromCache(sessionJob);
+ if
(sessionJob.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
+ LOG.info("Nothing was deployed yet, nothing to clean up");
+ onCleanupFinished(sessionJob);
+ return DeleteControl.defaultDelete();
+ }
eventRecorder.triggerEvent(
sessionJob,
EventRecorder.Type.Normal,
EventRecorder.Reason.Cleanup,
EventRecorder.Component.Operator,
"Cleaning up FlinkSessionJob",
josdkContext.getClient());
- statusRecorder.updateStatusFromCache(sessionJob);
Review Comment:
Why to have this removal in place? Without this we may risk to act on a
stale status.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/SessionJobReconciler.java:
##########
@@ -192,25 +192,32 @@ public DeleteControl
cleanupInternal(FlinkResourceContext<FlinkSessionJob> ctx)
// Only check for pending cancellation once the cluster is ready, so
that other deletion
// paths (cluster missing, deleting, unhealthy without HA) are not
blocked.
if (ReconciliationUtils.isJobCancelling(status)) {
+ if (cancellationTimedOut(ctx)) {
+ eventRecorder.triggerEvent(
Review Comment:
Looks like this event is triggered only for session jobs. It will be great
to have the application jobs in sync with this implementation.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java:
##########
@@ -94,14 +94,19 @@ public DeleteControl cleanup(FlinkDeployment flinkApp,
Context<FlinkDeployment>
if (canaryResourceManager.handleCanaryResourceDeletion(flinkApp)) {
return DeleteControl.defaultDelete();
}
+ statusRecorder.updateStatusFromCache(flinkApp);
+ if
(flinkApp.getStatus().getReconciliationStatus().isBeforeFirstDeployment()) {
Review Comment:
Besides having the argument for placing this code block and this check after
the event triggering, I even think that this check should be placed under the
observation flow that happens at line 113, mainly because if the observer sees
the reconciliationStatus.state in UPGRADING with nothing deployed, it clears
the `lastReconciledSpec`. This flow together with the removal that this PR is
making at `ApplicationReconciler` line 440 will throw an exception which will
be wrongly labelled as CleanupFailed, which isn't true.
In my opinion there are 2 paths here to follow:
1. Rollback the `status.getReconciliationStatus().isBeforeFirstDeployment()`
within `ApplicationReconciler` and `SessionJobReconciler` + add the same guard
for `SessionReconciler`. This will align all the reconciler implementations,
and this code block can be fully dropped.
2. Keep the removals and place this check under observe call .
The advantage of option 1 is that the check lives in the layer where it
belongs, where this assumption actually holds. It also ensures that leftovers
specific to each deployment mode are cleaned up in the way that mode requires.
Option 2 relies only on `lastReconciledSpec` being null, which doesn’t
guarantee that resources such as an autoscaler ConfigMap have been removed, nor
does it account for other resources added in the future that could end up as
leftovers.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -331,6 +340,22 @@ protected JobUpgrade
getJobUpgrade(FlinkResourceContext<CR> ctx, Configuration d
return JobUpgrade.unavailable();
}
+ private static boolean lastStateFallbackEnabled(Configuration
deployConfig) {
+ return deployConfig.get(
+
KubernetesOperatorConfigOptions.OPERATOR_JOB_UPGRADE_LAST_STATE_FALLBACK_ENABLED);
+ }
+
+ protected boolean cancellationTimedOut(FlinkResourceContext<CR> ctx) {
+ var updateTime =
ctx.getResource().getStatus().getJobStatus().getUpdateTime();
+ return updateTime != null
+ && clock.instant()
+ .isAfter(
+
Instant.ofEpochMilli(Long.parseLong(updateTime))
+ .plus(
+ ctx.getOperatorConfig()
+
.getFlinkShutdownClusterTimeout()));
Review Comment:
Using this timeout for job-level cancellation doesn’t feel right to me,
since it’s currently used for cluster-level deletion. For the job level, I’d
rather introduce a separate timeout owned by the operator, the same way
`resource.cleanup.timeout` is. I’d also rename the existing one to
cluster.cleanup.timeout and add a sibling `job.cleanup.timeout`. Alternatively,
we could fall back on Flink’s core `task.cancellation.timeout`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]