prashantbh opened a new pull request, #28748: URL: https://github.com/apache/flink/pull/28748
## What is the purpose of the change Fixes a regression from [FLINK-36451](https://issues.apache.org/jira/browse/FLINK-36451) that can permanently erase a component's leader information from the HA backend after a leadership loss and re-acquisition. Mechanism: leadership is granted once per process (a single `issuedLeaderSessionID`), but each component (restserver, dispatcher, jobmanager, resourcemanager) confirms its own leader information asynchronously on `leadershipOperationExecutor`, while leader-information change events are handled synchronously in `notifyLeaderInformationChangeInternal`. After re-acquiring leadership a component is granted but not yet confirmed, so its `confirmedLeaderInformation` is empty. A change event in that window takes the `confirmedLeaderInformation.isEmpty()` branch and publishes empty, which on Kubernetes removes the component's key. The self-correcting rewrite only runs once a confirmation exists, so the erased entry is not restored until a subsequent confirmation succeeds, and remains erased if none does. The change event is typically self-inflicted: all components share one ConfigMap, so when one component confirms (e.g. the dispatcher), that write fires a watch event evaluated against every other component, including one whose confirmation hasn't landed yet and whose stale prior-session entry is then reset to empty. Impact: the JobManager keeps running as a healthy leader while its external entry is gone. The REST endpoint cannot be resolved (restserver), or job submission and TaskManager registration fail (dispatcher/jobmanager), with no self-recovery. ## Brief change log - `DefaultLeaderElectionService#notifyLeaderInformationChangeInternal`: the `confirmedLeaderInformation.isEmpty()` branch now returns instead of publishing empty. A component's entry is never deleted while nothing is confirmed for it: a later (session-fenced) confirmation overwrites any stale value, and when we do not hold the lock the write would no-op anyway. Intentional cleanup is unchanged (`remove` -> `deleteLeaderInformation`). ## Verifying this change Added tests that use `ManuallyTriggeredScheduledExecutorService` to hold a confirmation while a change event is injected; they fail without the fix and pass with it: - `testLeaderInformationChangeDoesNotEraseComponentWithInflightConfirmation`: a change event during an outstanding confirmation must not erase the entry; it self-heals once the confirmation runs. - `testLeaderInformationChangeDoesNotEraseComponentAfterReacquisition`: loss and re-acquisition with a new session; the prior session's entry must not be erased and self-heals to the new session. - Repurposed `testGrantDoesNotBlockNotifyLeaderInformationChange` and `...AllKnownLeaderInformation` to assert non-erasure for both `onLeaderInformationChange` overloads. - Also verified against the leaderelection / leaderretrieval / highavailability suites (including ZooKeeper and `LeaderChangeClusterComponentsTest`) and the `flink-kubernetes` HA unit tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes, fixes HA leader-election behavior affecting JobManager recovery on Kubernetes and ZooKeeper. - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code / Claude Opus 4.8) Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
