shashank created CAMEL-25286:
--------------------------------
Summary: camel-consul - ConsulClusterView stops taking part in the
leader election after a failed query or when Consul invalidates its session,
and can keep the leadership while another node takes it
Key: CAMEL-25286
URL: https://issues.apache.org/jira/browse/CAMEL-25286
Project: Camel
Issue Type: Bug
Components: camel-consul
Reporter: shashank
{{ConsulClusterView}} (used by {{ConsulClusterService}}, {{camel-master}} and
{{ClusteredRoutePolicy}}) holds the leadership with a lock on
{{<rootPath>/<namespace>}} owned by a Consul session. A {{Watcher}} watches
that key with a chain of blocking queries; each query also renews the session.
Three paths end the election for the node until the cluster service is
restarted:
h3. 1. A failed query: the watch ends and the node can stay leader while
another node takes the lock
{code:java}
public void onFailure(Throwable throwable) {
LOGGER.debug("{}", throwable.getMessage(), throwable);
if (sessionId.get() != null) {
keyValueClient.releaseLock(configuration.getRootPath(),
sessionId.get()); // wrong key, synchronous
}
localMember.setMaster(false);
watch();
}
{code}
* It releases the lock of the root path ({{/camel}}), not of the namespace path
that the session holds (the same mistake was fixed in {{getLeader()}} in 2021,
PR #4937).
* {{releaseLock}} is a synchronous HTTP call. When the query failed because
Consul cannot be reached (the agent restarts, a network error), that call
throws a {{ConsulException}}, so {{setMaster(false)}} and {{watch()}} never
run: no more queries and no more session renewals. A leader keeps its
leadership locally (its clustered routes keep running); its session expires
after the TTL, Consul releases the lock, another node acquires it, and both
nodes run the clustered routes. A node that was not leader never takes part
again.
* The same happens when handling an answer fails ({{acquireLock}} and the
leadership event call Consul synchronously from {{onComplete}}), and
{{setMaster(false)}} itself calls Consul ({{getLeader()}}) to fire the
leadership event, so the event was lost while Consul was down.
h3. 2. An invalidated session is never replaced
When Consul invalidates the session (its TTL expired during a network
partition, the local agent restarted or left so its {{serfHealth}} check
failed, Consul lost its data), every renewal fails with {{404 Session id '...'
not found}} (the stack trace of CAMEL-15423), and {{acquireLock}} returns
{{false}} for ever because {{getSessionInfo}} finds no session. The node can
never be leader again; if all nodes lost their sessions (for example a rolling
restart of the Consul agents) no node is leader any more and the clustered
routes stay stopped everywhere.
h3. 3. A key that does not exist is ignored
{{onComplete}} only acts when the key exists ({{if (value.isPresent())}}).
After Consul lost its data the key is gone and no node acquires the lock again;
a node that was leader also keeps its leadership locally.
h3. Reproduction (unit test, no Consul)
{{ConsulClusterViewRecoveryTest}} runs the real {{ConsulClusterService}} and
view against mocked {{SessionClient}} / {{KeyValueClient}} that simulate Consul
(sessions, lock holder, key, reachability; a renewal of an unknown session
throws {{ConsulException}} with code 404 as the client does). On main all 4
tests fail:
* Consul cannot be reached, the query fails: the node stays leader ({{expected:
<false> but was: <true>}}) and never queries again.
* A failed query: the lock released is not the one of the namespace path
(Mockito: {{Argument(s) are different! Wanted: releaseLock("/camel/my-ns",
"session-1")}}, actual {{"/camel"}}).
* Consul invalidated the session: no new session is created ({{expected: <2>
but was: <1>}}) and the node never takes the free lock again.
* Consul lost its data: the node stays leader without a lock ({{expected:
<false> but was: <true>}}) and never creates the key again.
h3. Proposed fix
* {{onFailure}}: give the leadership up first, release the lock of the
namespace path (ignoring a failure), and query again after
{{sessionRefreshInterval}} (at least one second) on a single-thread scheduler
of the view, so that an agent that cannot be reached is not queried in a loop.
* Renewal: a session that Consul does not know any more (404, or no session
returned) is replaced by a new one (under the session lock, only while the view
runs), after giving the leadership up; other renewal failures are logged at
debug and retried with the next query.
* {{onComplete}}: a missing key is handled like a free key (try to acquire it);
an exception while handling the answer gives the leadership up instead of
ending the watch. The index is stored before handling the answer.
* The leadership event is fired with no leader when the current leader cannot
be read.
A failed query while Consul can be reached now really releases the lock (no
{{lock-delay}} on an explicit release), so the leadership can move to another
node; the node had already given it up locally on main. While the Consul
cluster has no leader the release fails too and the node takes its own lock
back with the next answer. File lock and ZooKeeper cluster views also give the
leadership up on the first failure.
No new option. With the fix the 4 new tests pass and the module's unit tests
pass (8); the cluster ITs need Docker (not run).
Affected: 4.14.x, 4.18.x and main (same code).
Duplicate check (2026-10-03): JIRA text "ConsulClusterView" /
"ConsulClusterService" / "consul" + "leader" / "consul" + "session" (14
issues): only CAMEL-15423 (2020, closed Incomplete, same symptom, no fix),
CAMEL-22523 (test automation), CAMEL-25275 (consumers, open, ours). GitHub pull
requests "ConsulClusterView": only #25842 (ZooKeeper cluster shutdown) and
#4937 (getLeader path, 2021).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
