shashank created CAMEL-25315:
--------------------------------
Summary: camel-zookeeper-master - a master whose consumer failed
to start never consumes, and a consumer started while the leadership is lost or
the route stops keeps running
Key: CAMEL-25315
URL: https://issues.apache.org/jira/browse/CAMEL-25315
Project: Camel
Issue Type: Bug
Components: camel-zookeeper-master
Reporter: shashank
{{MasterConsumer}} starts the delegate consumer from the {{CHANGED}} group
event (the group's own thread, {{onLockOwned}}) and stops it from
{{DISCONNECTED}} (called directly on the Curator connection-state thread, and
by {{ZooKeeperGroup.close()}} on the closing thread) and from {{doStop}}.
Nothing synchronises them, and {{onLockOwned}} only acts {{if (delegate ==
null)}}:
{code:java}
if (delegate == null) {
try {
ServiceHelper.startService(endpoint.getConsumerEndpoint());
delegate = endpoint.getConsumerEndpoint().createConsumer(processor);
...
groupListener.updateState(thisNodeState);
ServiceHelper.startService(delegate);
} catch (Exception e) {
LOG.error("Failed to start master consumer for: {}", endpoint, e);
}
}
{code}
* *A failed start is never retried.* When the delegate cannot start at the
moment the node is elected (JMS broker, FTP or database server not reachable
yet), the failed consumer stays in {{delegate}}, so every later leadership
event skips the start. The node keeps the ZooKeeper leadership, the other nodes
stay standby, and nobody consumes until this node loses its connection or the
route is restarted. The only trace is one ERROR log line.
* *A consumer started after a disconnect keeps running on a node that is not
the master.* If the connection is lost ({{SUSPENDED}}/{{LOST}} ->
{{DISCONNECTED}} -> {{stopConsumer()}}) between the null check and the
assignment of the new delegate (endpoint start, {{createConsumer}}), the stop
finds nothing, and the delegate is then assigned and started. Another node
becomes the master: two nodes consume. When this node reconnects as a standby
nothing stops it (standby events do nothing).
* *The same after a route stop:* {{doStop}} stops nothing, and {{close()}} only
calls {{DISCONNECTED}} again after waiting up to 5 seconds for the group
thread, so a start that takes longer leaves a consumer running after the route
(or the CamelContext) stopped.
camel-master's {{MasterConsumer}} handles these cases (leadership lock, start
attempts with a backoff, a start that finishes after the leadership was lost is
stopped).
h3. Reproduction
{{MasterConsumerLeadershipTest}} (camel-zookeeper-master, no ZooKeeper: a
{{ManagedGroupFactory}} in the registry returns a fake group whose events the
test fires; the delegate endpoint can fail a start or hold {{createConsumer}}
on a latch), on main:
{noformat}
testFailedStartIsRetried: the first start fails
The master must start its consumer after a failed start ==> expected: <1> but
was: <0> within 20 seconds.
testDisconnectedWhileCreatingConsumer: DISCONNECTED while the group thread is
in createConsumer
A node that lost the leadership must not consume ==> expected: <0> but was:
<1>
testStoppedWhileCreatingConsumer: stopRoute while the group thread is in
createConsumer
A stopped route must not consume ==> expected: <0> but was: <1>
{noformat}
No sleeps: latches, Awaitility.
The defect was found with a TLA+ model of the consumer (group thread, Curator
connection thread, stopping thread; the null check, the creation, the start and
the stop as separate steps; {{close()}} waiting at most 5 s): "a master that
received a leadership event consumes" is violated in 3 steps with one failed
start, and "no delegate runs on a stopped route or a node that is not the
master" in 5 steps (event, connection lost, stopConsumer, create, start) and in
7 steps for the route stop. A first version of the model let the start read a
stale delegate; the Java reads the field again ({{startService(delegate)}}),
which the corrected model follows. With the fix all properties hold, with up to
3 events, 2 failed starts, a disconnect and a stop, and TLC's deadlock check
on; mutations of the fix without the generation check (a disconnect during the
start leaves a zombie) or without the retry violate them. The model does not
cover reconnecting as master after a disconnect.
h3. Proposed fix
A {{leadershipLock}} and a generation counter (incremented by
{{onDisconnected}} and {{doStop}}): a start is marked under the lock after
checking again that the group is connected and this node is the master
({{ZooKeeperGroup}} clears {{connected}} before it calls {{DISCONNECTED}}); the
consumer is created and started without holding the lock (it can take long); it
is published under the lock only if the generation did not change, otherwise it
is stopped. A failed start is retried after 5 seconds by a scheduled task of
the same generation (cancelled by a disconnect or stop), and the started state
is published to the group once per leadership term, so a retry does not trigger
another group event (a new state with a random container id would). This
follows camel-master's {{MasterConsumer}} (CAMEL-24583, CAMEL-24626) in its
locking and lost-leadership checks, with two differences:
camel-zookeeper-master has no {{backOffDelay}}/{{backOffMaxAttempts}} options,
so the delay is fixed at camel-master's default of 5 seconds and the retries go
on while this node is the master (camel-master gives up after 10 attempts and
stays leader without consuming); and each attempt creates a new delegate
consumer (a failed {{start()}} already stopped it), where camel-master reuses
one. Each failed attempt is logged at WARN with its stack trace.
With the fix the new tests and the camel-zookeeper-master unit tests (14) pass;
the integration tests need Docker and were not run.
Affected: 4.14.x, 4.18.x and main (same code, GitHub contents API);
{{MasterConsumer}} unchanged since 2021.
Duplicate check (2026-10-04): JIRA component camel-zookeeper-master and text
"zookeeper-master"/"zookeepermaster" (53 issues; CAMEL-24457 and CAMEL-24545
are about the camel-zookeeper cluster service, CAMEL-17226 namespaces; none
about the master consumer's start or races). No open PR touches the component.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
