shashank created CAMEL-27296:
--------------------------------

             Summary: camel-kubernetes - a stopped KubernetesClusterView keeps 
reporting the leadership and keeps the lease, so no other pod can lead
                 Key: CAMEL-27296
                 URL: https://issues.apache.org/jira/browse/CAMEL-27296
             Project: Camel
          Issue Type: Bug
          Components: camel-kubernetes
            Reporter: shashank
            Assignee: shashank


h3. What happens

{{KubernetesClusterView.doStop}} only stops the leadership controller (its 
executors) and closes the client. It does not clear {{currentLeader}}, does not 
fire leadership lost, and does not release the lease. A view is stopped when 
its last user releases it ({{MasterConsumer}}, {{ClusteredRoutePolicy}}) or 
when the cluster service stops. Then:

* the stopped view keeps reporting the pod as leader 
({{getLocalMember().isLeader()}}, {{getLeader()}}, 
{{CamelClusterService.isLeader(namespace)}});
* the lease ({{Lease}} holderIdentity, or the leader key of the group in the 
{{ConfigMap}}) still names the pod, and the pod is still ready, so the other 
pods keep treating it as the valid leader: no pod can lead the group until this 
pod becomes unready or goes away;
* when the view is used again (a clustered route added again), the restarted 
view reports the old leadership at once, until its first successful lookup (for 
ever if the API server cannot be reached), also while another pod is the leader.

{{FileLockClusterView.doStop}} releases its lock and fires leadership lost.

h3. How to reproduce

{{KubernetesClusterViewStopTest}} (camel-kubernetes, {{LockTestServer}} mock, 
Lease and ConfigMap): release the view of the leader pod so it stops; the view 
still says leader, the lease still names the pod, and the other pod never 
becomes leader.

Found with a TLA+ model of the Kubernetes leadership state machine.

h3. Fix

{{KubernetesLeadershipController.stop}} waits for a running refresh (up to the 
renew deadline); the view clears {{currentLeader}} and fires leadership lost if 
the pod was the leader; then {{releaseLeadership}} reads the lease again and, 
if it names the pod, removes the pod from it with 
{{optimisticDeleteLeaderInfo}} (409 reads again, up to 3 tries; other failures 
are logged at WARN and the lease is kept, as before). The release runs on its 
own thread and the stop waits for it up to the renew deadline, as the 
Kubernetes client tries a failed request again on its own (by default up to 10 
times), which could otherwise hold the stop for minutes when the API server 
cannot be reached. The lease is only released when the leadership had taken 
effect (not while the pod waits in BECOMING_LEADER for the previous leader). 
Events of a stopped view's controller are ignored. Upgrade guide entry for 4.24.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to