[
https://issues.apache.org/jira/browse/FLINK-40114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Bhardwaj updated FLINK-40114:
--------------------------------------
Labels: high-availability (was: )
> Leader information permanently erased when a change event races a pending
> confirmation
> --------------------------------------------------------------------------------------
>
> Key: FLINK-40114
> URL: https://issues.apache.org/jira/browse/FLINK-40114
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.4.0
> Reporter: Prashant Bhardwaj
> Priority: Major
> Labels: high-availability
>
> h3. Summary
> Since FLINK-36451 made leadership confirmation asynchronous, a component's
> leader information can be *permanently deleted* from the HA backend (e.g. the
> Kubernetes leader ConfigMap) after a leadership loss and re-acquisition.
> h3. Root cause
> Confirmation runs asynchronously ({{{}confirmLeadershipAsync{}}} on the
> single-threaded {{{}leadershipOperationExecutor{}}}), while the
> leader-information-change handler ({{{}onLeaderInformationChange{}}} →
> {{{}notifyLeaderInformationChangeInternal{}}}) still runs synchronously on
> the driver's watch thread. After a loss + re-acquisition, a component's local
> confirmation is cleared but its stale entry remains in the HA store. If a
> change event is handled before the re-granted session's confirmation lands,
> the handler takes the [empty-confirmation reset
> branch|https://github.com/apache/flink/blob/f9d81b2e11e5cff7469b523cc73ce34b1d28e5ab/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/DefaultLeaderElectionService.java#L519]
> and publishes empty, which on Kubernetes [removes the
> key|https://github.com/apache/flink/blob/f9d81b2e11e5cff7469b523cc73ce34b1d28e5ab/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesLeaderElectionDriver.java#L161].
> Because the self-correcting path runs only once a confirmation exists, if
> that confirmation never lands — due to a transient getConfigMap() failure or
> session churn — the deletion is permanent. Triggered by any API-server
> disruption (outage, rolling upgrade, or intermittent 5xx/429 under load).
> h3. Impact
> The victim is whichever component's latest-session confirmation fails to land:
> * {{restserver}} erased → REST endpoint cannot be resolved (operator/clients
> cannot reach the cluster).
> * {{{}dispatcher{}}}/{{{}jobmanager{}}} erased → no job submission, no
> TaskManager registration.
> The JobManager keeps running and believes it is a healthy leader, so it is
> silent and does not self-recover.
> h3. Reproduction
> Reproducible deterministically as a unit test by driving
> {{DefaultLeaderElectionService}} with a manually-triggered
> {{leadershipOperationExecutor}} so the confirmation can be held mid-flight: a
> component wins leadership and confirms a session, loses leadership (clearing
> the local confirmation but leaving the stale entry in the store), then wins
> again; while the new session's confirmation is still pending, a
> leader-information-change event carrying the stale entry drops the
> component's key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
