shashank created CAMEL-26798:
--------------------------------
Summary: camel-hazelcast - HazelcastRoutePolicy: a consumer whose
start fails at takeover is never started again, and a takeover can start the
consumer of a stopping route
Key: CAMEL-26798
URL: https://issues.apache.org/jira/browse/CAMEL-26798
Project: Camel
Issue Type: Bug
Components: camel-hazelcast
Reporter: shashank
Assignee: shashank
When a node takes the leadership, {{HazelcastRoutePolicy}} starts the consumers
it stopped while the node was a follower. Its leadership thread then holds the
Hazelcast lock and does nothing else until the policy stops. Four problems in
that start:
*A failed start is never retried.* When the start of a consumer fails (for
example a broker that is not reachable yet at takeover), the exception is
logged and nothing tries again: the node holds the lock, so no other node can
take over, and it does not consume. The cluster has a leader that never
consumes until a restart. A failure also skips the consumers of the other
routes that use the same policy. This is the same defect as CAMEL-26540
({{MasterRoutePolicy}}) and CAMEL-25315 ({{MasterConsumer}}), which were fixed
by retrying the start while the node is the leader.
*A takeover can start the consumer of a route that is being stopped or
suspended.* A route stopped on a follower: the route service stops the
consumer, and only then calls the policy's {{onStop}}, which forgets the route.
A leadership taken in between starts the consumer again, so the route ends
stopped with a running consumer; when the route is started again it has two
consumers. The same happens with a route being suspended ({{onSuspend}}): it
ends suspended with a running consumer.
*A leadership taken while a route starts can be lost.* {{onStart}} reads the
leader flag without a lock and then stops the consumer and remembers the route.
When the leadership is taken between the two, the leadership thread finds no
consumer to start, and the route then stops its consumer: the node holds the
lock and does not consume, and the other nodes wait for the lock.
*A stop during a takeover waits.* The consumers are started under the lifecycle
lock of the policy, which {{stop()}} holds while it waits for the leadership
thread, so a stop of the CamelContext while the node takes the leadership waits
until the consumers started (or for the timeout of the executor shutdown).
Expected: a leader starts every consumer the policy stopped, trying again a
start that failed while it is the leader; it does not start the consumer of a
route that is stopping, stopped or suspending; and a leadership taken while a
route starts is not missed.
Suggested fix: start each consumer on its own and forget the route only once
its consumer started; let the leadership thread try again the failed starts
while it holds the lock; skip a route whose consumer may not be started
({{isResumeOrStartConsumerAllowed}}) and check again after the start; start the
consumers without holding a lock that the stop or the route callbacks need;
guard the leader flag and the routes of the policy with a lock of the policy
(not the lifecycle lock of the service, which the stop holds while it waits for
the leadership thread).
Found with Specula (TLA+ model checked against traces of the real code) and
with a TLA+ model of the policy, and confirmed with tests on the real classes
(embedded Hazelcast member).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
