mielientiev commented on code in PR #1065:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/1065#discussion_r3120155023
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/SessionReconciler.java:
##########
@@ -64,9 +66,38 @@ public SessionReconciler(
@Override
protected boolean readyToReconcile(FlinkResourceContext<FlinkDeployment>
ctx) {
+ var deployment = ctx.getResource();
+ // Only check when there is a pending spec change on the deployment
itself
+ if (!Objects.equals(
+ deployment.getMetadata().getGeneration(),
+ deployment.getStatus().getObservedGeneration())) {
+ var deploymentName = deployment.getMetadata().getName();
+ boolean hasTransitioningSessionJob =
+
ctx.getJosdkContext().getSecondaryResources(FlinkSessionJob.class).stream()
+ .filter(job ->
deploymentName.equals(job.getSpec().getDeploymentName()))
+
.anyMatch(SessionReconciler::isSessionJobTransitioning);
+ if (hasTransitioningSessionJob) {
+ LOG.info(
+ "Deferring session cluster upgrade: associated session
jobs are being upgraded");
+ return false;
+ }
+ }
return true;
}
+ private static boolean isSessionJobTransitioning(FlinkSessionJob job) {
+ var reconStatus = job.getStatus().getReconciliationStatus();
+ // New session jobs state is UPGRADING by default before first
deployment. Treating them as
+ // transitioning would deadlock: the deployment can't
+ // start because it waits for the job, and the job can't start because
it waits for the
+ // cluster.
+ if (reconStatus.isBeforeFirstDeployment()) {
+ return false;
+ }
+ var state = reconStatus.getState();
+ return state == ReconciliationState.UPGRADING || state ==
ReconciliationState.ROLLING_BACK;
Review Comment:
That's a good point! I think there's a possibility when something goes wrong
and session job can be stuck in `upgrading/rolling_back` state and basically
meaning that until the job won't be fixed/deleted, the cluster can be stuck.
What is in your opinion is the better / safer way to fix this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
