[
https://issues.apache.org/jira/browse/CAMEL-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-27296:
--------------------------------
Fix Version/s: 4.24.0
> camel-kubernetes - a stopped KubernetesClusterView keeps reporting the
> leadership and keeps the lease, so no other pod can lead
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27296
> URL: https://issues.apache.org/jira/browse/CAMEL-27296
> Project: Camel
> Issue Type: Bug
> Components: camel-kubernetes
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> h3. What happens
> {{KubernetesClusterView.doStop}} only stops the leadership controller (its
> executors) and closes the client. It does not clear {{currentLeader}}, does
> not fire leadership lost, and does not release the lease. A view is stopped
> when its last user releases it ({{MasterConsumer}}, {{ClusteredRoutePolicy}})
> or when the cluster service stops. Then:
> * the stopped view keeps reporting the pod as leader
> ({{getLocalMember().isLeader()}}, {{getLeader()}},
> {{CamelClusterService.isLeader(namespace)}});
> * the lease ({{Lease}} holderIdentity, or the leader key of the group in the
> {{ConfigMap}}) still names the pod, and the pod is still ready, so the other
> pods keep treating it as the valid leader: no pod can lead the group until
> this pod becomes unready or goes away;
> * when the view is used again (a clustered route added again), the restarted
> view reports the old leadership at once, until its first successful lookup
> (for ever if the API server cannot be reached), also while another pod is the
> leader.
> {{FileLockClusterView.doStop}} releases its lock and fires leadership lost.
> h3. How to reproduce
> {{KubernetesClusterViewStopTest}} (camel-kubernetes, {{LockTestServer}} mock,
> Lease and ConfigMap): release the view of the leader pod so it stops; the
> view still says leader, the lease still names the pod, and the other pod
> never becomes leader.
> Found with a TLA+ model of the Kubernetes leadership state machine.
> h3. Fix
> {{KubernetesLeadershipController.stop}} waits for a running refresh (up to
> the renew deadline); the view clears {{currentLeader}} and fires leadership
> lost if the pod was the leader; then {{releaseLeadership}} reads the lease
> again and, if it names the pod, removes the pod from it with
> {{optimisticDeleteLeaderInfo}} (409 reads again, up to 3 tries; other
> failures are logged at WARN and the lease is kept, as before). The release
> runs on its own thread and the stop waits for it up to the renew deadline, as
> the Kubernetes client tries a failed request again on its own (by default up
> to 10 times), which could otherwise hold the stop for minutes when the API
> server cannot be reached. The lease is only released when the leadership had
> taken effect (not while the pod waits in BECOMING_LEADER for the previous
> leader). Events of a stopped view's controller are ignored. Upgrade guide
> entry for 4.24.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)