[
https://issues.apache.org/jira/browse/CAMEL-24457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-24457:
--------------------------------
Fix Version/s: 4.23.0
> Camel Zookeeper Cluster Split brain issue when leader is isolated
> -----------------------------------------------------------------
>
> Key: CAMEL-24457
> URL: https://issues.apache.org/jira/browse/CAMEL-24457
> Project: Camel
> Issue Type: Bug
> Components: camel-zookeeper, camel-zookeeper-master
> Affects Versions: 4.18.3
> Environment: 3 Linux RHEL machines
> All 3 has ZK on it
> And 2 Has the the camel application in cluster mode Active/Passive
> Reporter: mustafa kamal
> Priority: Major
> Labels: bug, split-brain
> Fix For: 4.23.0
>
>
> ------------------------------------------------------------------------------------------------------------------
> Problem Description:
> ------------------------------------------------------------------------------------------------------------------
> When the ZooKeeper cluster becomes unavailable, the Camel route managed by
> "ClusteredRoutePolicy" never stops, even though the node has lost its
> ZooKeeper session. Beyond the route not stopping, this also introduces a
> split-brain scenario: if one node becomes isolated from ZooKeeper while
> others remain connected, a new leader election runs on the healthy side and a
> second node wins leadership and starts the same route — now two nodes are
> running the same route simultaneously with no coordination, which can cause
> data corruption, duplicate processing, or conflicting writes depending on
> what the route does.
>
> ------------------------------------------------------------------------------------------------------------------
> Root Cause:
> ------------------------------------------------------------------------------------------------------------------
> The root cause is a timing issue between two threads in
> "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()". When
> ZooKeeper goes down, Curator's "ConnectionStateManager" thread fires a
> "SUSPENDED"/"LOST" state change, which causes "LeaderSelectorListenerAdapter"
> to interrupt the thread blocked inside "takeLeadership()". The "BlockingTask"
> handles the interrupt correctly and exits its loop. Execution then reaches
> the leadership-lost event at:
>
> fireLeadershipChangedEvent(getLeader().orElse(null)); //
> ZooKeeperClusterView.java line 155
>
> This fires "ClusteredRoutePolicy.leadershipChanged()", which, regardless of
> the argument passed, always calls back into:
> setLeader(clusterView.getLocalMember().isLeader()); //
> ClusteredRoutePolicy.java line 376
>
> Which resolves to: leaderSelector.hasLeadership() //
> CuratorLocalMember.isLeader(), line 162
> This returns "true" at this point because Curator's guarantees that
> "hasLeadership()" only becomes "false" after "takeLeadership()" returns to
> the "LeaderSelector" internals. The event fires from inside
> "takeLeadership()", so the answer is always "true", "ClusteredRoutePolicy"
> sees no leadership change, and the route keeps running indefinitely.
>
> ------------------------------------------------------------------------------------------------------------------
> Consequences:
> ------------------------------------------------------------------------------------------------------------------
> The consequence is:
> - The [isolated node] keeps running the route because "hasLeadership()" is
> still "true" at the moment the event fires
> - The [healthy side] elects a new leader, which also starts the same route
> - Both nodes now process the same workload simultaneously with no mutual
> exclusion — a classic [split-brain]
>
>
> ------------------------------------------------------------------------------------------------------------------
> Solution:
> ------------------------------------------------------------------------------------------------------------------
> Introduce a "volatile boolean leader" flag inside "ZooKeeperClusterView" that
> is owned and controlled by the view itself, rather than delegating to
> "leaderSelector.hasLeadership()".
> Change "CuratorLocalMember.isLeader()" to return this flag instead.
> In "takeLeadership()", set the flag to "true" before firing the
> leadership-gained event, and in the "finally" block set it to "false" before
> firing the leadership-lost event. This guarantees that when
> "ClusteredRoutePolicy" calls back into "isLeader()" during the event, it
> reads "false", which causes "stopManagedRoutes()" to be called and the route
> to stop correctly before any other node can win the election and start it.
> Additionally, "leaderSelector.autoRequeue()" should be called in "doStart()"
> so that after losing leadership due to a ZooKeeper disconnect, the node
> automatically re-enters the election when ZooKeeper reconnects and the route
> can start again on whichever node wins.
>
> - "ZooKeeperClusterView": add "volatile boolean leader" field
> - "ZooKeeperClusterView.CuratorLocalMember.isLeader()": return "leader" flag
> instead of "leaderSelector.hasLeadership()"
> - "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()": set
> "leader = true" on entry, wrap task in "try/finally", set "leader = false"
> before firing the lost event
> - "ZooKeeperClusterView.doStart()": add "leaderSelector.autoRequeue()" to
> re-enter election after reconnect
>
> ------------------------------------------------------------------------------------------------------------------
> How to reproduce:
> ------------------------------------------------------------------------------------------------------------------
> 1. Start two instances of a Camel application using "ZooKeeperClusterService"
> with "ClusteredRouteController"
> 2. Confirm one instance is leader and its route is running
> 3. Isolate the leader node from all ZooKeeper nodes (e.g. firewall rules or
> kill ZK nodes)
> 4. Observe on the healthy side: a new leader is elected and its route starts
> 5. Observe on the isolated node: the route never stops — both nodes are now
> running the same route simultaneously (split-brain)
> 6. Expected: the isolated node's route stops as soon as its ZooKeeper session
> is lost
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
