spuru9 opened a new pull request, #1202:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1202
## What is the purpose of the change
`SessionReconciler.getNonTerminalJobs` returns `Set.of()` on any exception
while querying the session cluster for running jobs, and `cleanupInternal`
treats an empty set as "no running jobs, safe to delete". As a result, if the
JobManager is unreachable while a session `FlinkDeployment` is being deleted
(with `block-on-unmanaged-jobs` enabled — the default), the operator cannot
enumerate running jobs but deletes the cluster anyway, stopping unmanaged
(CLI-submitted) jobs without a checkpoint — logging only a WARN. The safety
guard fails open exactly when it cannot be evaluated.
This change makes the guard fail closed: an indeterminate result blocks
deletion instead of assuming the cluster is empty.
## Brief change log
- `getNonTerminalJobs` now returns `Optional<Set<JobID>>`:
`Optional.empty()` means "could not determine" (the query failed), distinct
from a present empty set ("determined: no jobs").
- `cleanupInternal` blocks deletion and emits a `CleanupFailed` event on
the indeterminate case, rescheduling so the block lifts automatically once the
JobManager is reachable again — mirroring the existing block-on-session-jobs
branch.
- The `RestClusterClient` is now closed via try-with-resources (previously
leaked on every cleanup attempt).
- The jobs request is bounded by the configured Flink client timeout
instead of an unbounded `get()`.
Related: the fail-open was introduced with the block-on-unmanaged-jobs
feature (FLINK-28648). The new block reschedules and lifts once the cluster is
reachable, so it does not reintroduce the deletion deadlock fixed in
FLINK-39618. Force-deleting a cluster whose JobManager is permanently
unreachable is still possible via the existing opt-out
(`block-on-unmanaged-jobs` / `block-on-session-jobs` = false, FLINK-39432).
## Verifying this change
This change added tests and can be verified as follows:
- `SessionReconcilerTest` covers the new behavior, including
`testCleanupBlocksWhenNonTerminalJobsCannotBeDetermined` (JobManager
unreachable → deletion blocked, finalizer retained).
- Manually verified end-to-end on minikube: with the unpatched operator,
deleting a session cluster while the JobManager is down tears the cluster down
(`Stopping session cluster`); with this change the deletion is blocked with a
`CleanupFailed` event, and completes on its own once the JobManager is
reachable and the cluster is genuinely empty.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`:
no
- Core observer or reconciler logic that is regularly executed: yes
(session cluster cleanup path)
## Documentation
- Does this pull request introduce a new feature? no
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Code)
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
