[
https://issues.apache.org/jira/browse/CAMEL-24457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108878#comment-18108878
]
mustafa kamal edited comment on CAMEL-24457 at 8/27/26 3:25 PM:
----------------------------------------------------------------
Dear Maintainers I found an issue that happens when the application is
gracefully shutdown the application get's into a locked state and never shuts
down from my testing I found the cause to be a dead lock between
SpringApplicationShutdownHook and Curator-LeaderSelector-0
Also another issue I found was that when the ZooKeeperClusterView is being
shutdown, all the Listeners will be removed so there is no meaning in sending
an event
One fix I did that solved the two issue is to guard
fireLeadershipChangedEvent((CamelClusterMember) null);
In ZooKeeperClusterView with
if (isRunAlowed) {
}
This does solve the second issue of firing the Event while all Listeners are
marked for removal
It does solve the first issue but I'm not 100% sure on how it does and if it's
the best solution none the less the issue doesn't appear with this code change.
Another key info about the Dead Lock it doesn't happen to all Applications.
One application with 2 routes doesn't get into the locked state.
Another with 9 routes always gets into that Locked State.
So it also appears to be a time issue.
Please do inform me if you need any code examples, logs, or thread dumps.
was (Author: JIRAUSER314432):
Dear Maintainers I found an issue that happens when the application is
gracefully shutdown the application get's into a locked state and never shuts
down from my testing I found the cause to be a dead lock between
SpringApplicationShutdownHook and Curator-LeaderSelector-0
Also another issue I found was that when the ZooKeeperClusterView is being
shutdown, all the Listeners will be removed so there is no meaning in sending
an event
On fix I did that solved the two issue is to guard
fireLeadershipChangedEvent((CamelClusterMember) null);
In ZooKeeperClusterView with
if (isRunAlowed) {
}
This does solve the second issue of firing the Event while all Listeners are
marked for removal
It does solve the first issue but I'm not 100% sure on how it does and if it's
the best solution none the less the issue doesn't appear with this code change.
Another key info about the Dead Lock it doesn't happen to all Applications.
One application with 2 routes doesn't get into the locked state.
Another with 9 routes always gets into that Locked State.
So it also appears to be a time issue.
Please do inform me if you need any code examples, logs, or thread dumps.
> Camel Zookeeper Cluster Split brain issue when leader is isolated
> -----------------------------------------------------------------
>
> Key: CAMEL-24457
> URL: https://issues.apache.org/jira/browse/CAMEL-24457
> Project: Camel
> Issue Type: Bug
> Components: camel-zookeeper, camel-zookeeper-master
> Affects Versions: 4.18.3
> Environment: 3 Linux RHEL machines
> All 3 has ZK on it
> And 2 Has the the camel application in cluster mode Active/Passive
> Reporter: mustafa kamal
> Priority: Major
> Labels: bug, split-brain
> Fix For: 4.22.1, 4.23.0, 4.18.5
>
>
> ------------------------------------------------------------------------------------------------------------------
> Problem Description:
> ------------------------------------------------------------------------------------------------------------------
> When the ZooKeeper cluster becomes unavailable, the Camel route managed by
> "ClusteredRoutePolicy" never stops, even though the node has lost its
> ZooKeeper session. Beyond the route not stopping, this also introduces a
> split-brain scenario: if one node becomes isolated from ZooKeeper while
> others remain connected, a new leader election runs on the healthy side and a
> second node wins leadership and starts the same route — now two nodes are
> running the same route simultaneously with no coordination, which can cause
> data corruption, duplicate processing, or conflicting writes depending on
> what the route does.
>
> ------------------------------------------------------------------------------------------------------------------
> Root Cause:
> ------------------------------------------------------------------------------------------------------------------
> The root cause is a timing issue between two threads in
> "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()". When
> ZooKeeper goes down, Curator's "ConnectionStateManager" thread fires a
> "SUSPENDED"/"LOST" state change, which causes "LeaderSelectorListenerAdapter"
> to interrupt the thread blocked inside "takeLeadership()". The "BlockingTask"
> handles the interrupt correctly and exits its loop. Execution then reaches
> the leadership-lost event at:
>
> fireLeadershipChangedEvent(getLeader().orElse(null)); //
> ZooKeeperClusterView.java line 155
>
> This fires "ClusteredRoutePolicy.leadershipChanged()", which, regardless of
> the argument passed, always calls back into:
> setLeader(clusterView.getLocalMember().isLeader()); //
> ClusteredRoutePolicy.java line 376
>
> Which resolves to: leaderSelector.hasLeadership() //
> CuratorLocalMember.isLeader(), line 162
> This returns "true" at this point because Curator's guarantees that
> "hasLeadership()" only becomes "false" after "takeLeadership()" returns to
> the "LeaderSelector" internals. The event fires from inside
> "takeLeadership()", so the answer is always "true", "ClusteredRoutePolicy"
> sees no leadership change, and the route keeps running indefinitely.
>
> ------------------------------------------------------------------------------------------------------------------
> Consequences:
> ------------------------------------------------------------------------------------------------------------------
> The consequence is:
> - The [isolated node] keeps running the route because "hasLeadership()" is
> still "true" at the moment the event fires
> - The [healthy side] elects a new leader, which also starts the same route
> - Both nodes now process the same workload simultaneously with no mutual
> exclusion — a classic [split-brain]
>
>
> ------------------------------------------------------------------------------------------------------------------
> Solution:
> ------------------------------------------------------------------------------------------------------------------
> Introduce a "volatile boolean leader" flag inside "ZooKeeperClusterView" that
> is owned and controlled by the view itself, rather than delegating to
> "leaderSelector.hasLeadership()".
> Change "CuratorLocalMember.isLeader()" to return this flag instead.
> In "takeLeadership()", set the flag to "true" before firing the
> leadership-gained event, and in the "finally" block set it to "false" before
> firing the leadership-lost event. This guarantees that when
> "ClusteredRoutePolicy" calls back into "isLeader()" during the event, it
> reads "false", which causes "stopManagedRoutes()" to be called and the route
> to stop correctly before any other node can win the election and start it.
> Additionally, "leaderSelector.autoRequeue()" should be called in "doStart()"
> so that after losing leadership due to a ZooKeeper disconnect, the node
> automatically re-enters the election when ZooKeeper reconnects and the route
> can start again on whichever node wins.
>
> - "ZooKeeperClusterView": add "volatile boolean leader" field
> - "ZooKeeperClusterView.CuratorLocalMember.isLeader()": return "leader" flag
> instead of "leaderSelector.hasLeadership()"
> - "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()": set
> "leader = true" on entry, wrap task in "try/finally", set "leader = false"
> before firing the lost event
> - "ZooKeeperClusterView.doStart()": add "leaderSelector.autoRequeue()" to
> re-enter election after reconnect
>
> ------------------------------------------------------------------------------------------------------------------
> How to reproduce:
> ------------------------------------------------------------------------------------------------------------------
> 1. Start two instances of a Camel application using "ZooKeeperClusterService"
> with "ClusteredRouteController"
> 2. Confirm one instance is leader and its route is running
> 3. Isolate the leader node from all ZooKeeper nodes (e.g. firewall rules or
> kill ZK nodes)
> 4. Observe on the healthy side: a new leader is elected and its route starts
> 5. Observe on the isolated node: the route never stops — both nodes are now
> running the same route simultaneously (split-brain)
> 6. Expected: the isolated node's route stops as soon as its ZooKeeper session
> is lost
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
