[ 
https://issues.apache.org/jira/browse/CAMEL-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen updated CAMEL-27296:
--------------------------------
    Fix Version/s: 4.24.0

> camel-kubernetes - a stopped KubernetesClusterView keeps reporting the 
> leadership and keeps the lease, so no other pod can lead
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27296
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27296
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-kubernetes
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> h3. What happens
> {{KubernetesClusterView.doStop}} only stops the leadership controller (its 
> executors) and closes the client. It does not clear {{currentLeader}}, does 
> not fire leadership lost, and does not release the lease. A view is stopped 
> when its last user releases it ({{MasterConsumer}}, {{ClusteredRoutePolicy}}) 
> or when the cluster service stops. Then:
> * the stopped view keeps reporting the pod as leader 
> ({{getLocalMember().isLeader()}}, {{getLeader()}}, 
> {{CamelClusterService.isLeader(namespace)}});
> * the lease ({{Lease}} holderIdentity, or the leader key of the group in the 
> {{ConfigMap}}) still names the pod, and the pod is still ready, so the other 
> pods keep treating it as the valid leader: no pod can lead the group until 
> this pod becomes unready or goes away;
> * when the view is used again (a clustered route added again), the restarted 
> view reports the old leadership at once, until its first successful lookup 
> (for ever if the API server cannot be reached), also while another pod is the 
> leader.
> {{FileLockClusterView.doStop}} releases its lock and fires leadership lost.
> h3. How to reproduce
> {{KubernetesClusterViewStopTest}} (camel-kubernetes, {{LockTestServer}} mock, 
> Lease and ConfigMap): release the view of the leader pod so it stops; the 
> view still says leader, the lease still names the pod, and the other pod 
> never becomes leader.
> Found with a TLA+ model of the Kubernetes leadership state machine.
> h3. Fix
> {{KubernetesLeadershipController.stop}} waits for a running refresh (up to 
> the renew deadline); the view clears {{currentLeader}} and fires leadership 
> lost if the pod was the leader; then {{releaseLeadership}} reads the lease 
> again and, if it names the pod, removes the pod from it with 
> {{optimisticDeleteLeaderInfo}} (409 reads again, up to 3 tries; other 
> failures are logged at WARN and the lease is kept, as before). The release 
> runs on its own thread and the stop waits for it up to the renew deadline, as 
> the Kubernetes client tries a failed request again on its own (by default up 
> to 10 times), which could otherwise hold the stop for minutes when the API 
> server cannot be reached. The lease is only released when the leadership had 
> taken effect (not while the pod waits in BECOMING_LEADER for the previous 
> leader). Events of a stopped view's controller are ignored. Upgrade guide 
> entry for 4.24.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to