[
https://issues.apache.org/jira/browse/CAMEL-26798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26798 started by shashank.
----------------------------------------
> camel-hazelcast - HazelcastRoutePolicy: a consumer whose start fails at
> takeover is never started again, and a takeover can start the consumer of a
> stopping route
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26798
> URL: https://issues.apache.org/jira/browse/CAMEL-26798
> Project: Camel
> Issue Type: Bug
> Components: camel-hazelcast
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> When a node takes the leadership, {{HazelcastRoutePolicy}} starts the
> consumers it stopped while the node was a follower. Its leadership thread
> then holds the Hazelcast lock and does nothing else until the policy stops.
> Four problems in that start:
> *A failed start is never retried.* When the start of a consumer fails (for
> example a broker that is not reachable yet at takeover), the exception is
> logged and nothing tries again: the node holds the lock, so no other node can
> take over, and it does not consume. The cluster has a leader that never
> consumes until a restart. A failure also skips the consumers of the other
> routes that use the same policy. This is the same defect as CAMEL-26540
> ({{MasterRoutePolicy}}) and CAMEL-25315 ({{MasterConsumer}}), which were
> fixed by retrying the start while the node is the leader.
> *A takeover can start the consumer of a route that is being stopped or
> suspended.* A route stopped on a follower: the route service stops the
> consumer, and only then calls the policy's {{onStop}}, which forgets the
> route. A leadership taken in between starts the consumer again, so the route
> ends stopped with a running consumer; when the route is started again it has
> two consumers. The same happens with a route being suspended ({{onSuspend}}):
> it ends suspended with a running consumer.
> *A leadership taken while a route starts can be lost.* {{onStart}} reads the
> leader flag without a lock and then stops the consumer and remembers the
> route. When the leadership is taken between the two, the leadership thread
> finds no consumer to start, and the route then stops its consumer: the node
> holds the lock and does not consume, and the other nodes wait for the lock.
> *A stop during a takeover waits.* The consumers are started under the
> lifecycle lock of the policy, which {{stop()}} holds while it waits for the
> leadership thread, so a stop of the CamelContext while the node takes the
> leadership waits until the consumers started (or for the timeout of the
> executor shutdown).
> Expected: a leader starts every consumer the policy stopped, trying again a
> start that failed while it is the leader; it does not start the consumer of a
> route that is stopping, stopped or suspending; and a leadership taken while a
> route starts is not missed.
> Suggested fix: start each consumer on its own and forget the route only once
> its consumer started; let the leadership thread try again the failed starts
> while it holds the lock; skip a route whose consumer may not be started
> ({{isResumeOrStartConsumerAllowed}}) and check again after the start; start
> the consumers without holding a lock that the stop or the route callbacks
> need; guard the leader flag and the routes of the policy with a lock of the
> policy (not the lifecycle lock of the service, which the stop holds while it
> waits for the leadership thread).
> Found with Specula (TLA+ model checked against traces of the real code) and
> with a TLA+ model of the policy, and confirmed with tests on the real classes
> (embedded Hazelcast member).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
