shashank created CAMEL-25843:
--------------------------------
Summary: camel-hazelcast - a reliable topic consumer that falls
behind the ringbuffer is removed by Hazelcast and receives nothing again
Key: CAMEL-25843
URL: https://issues.apache.org/jira/browse/CAMEL-25843
Project: Camel
Issue Type: Bug
Components: camel-hazelcast
Reporter: shashank
Assignee: shashank
With {{reliable=true}} the hazelcast-topic consumer
({{HazelcastTopicConsumer}}, main 20cdb309f7d0) listens to a reliable topic
with a plain {{MessageListener}} ({{CamelMessageListener}}). A reliable topic
keeps its messages in a ringbuffer of limited capacity (10000 by default,
optionally with a time to live). When a listener falls behind by more than
that, the messages it has not read yet are overwritten. Hazelcast then removes
a listener that is not loss tolerant, for good: {{MessageRunner.accept}}
(Hazelcast 5.4.0) calls {{cancel()}} when the read result shows lost messages
and {{isLossTolerable}} is false, and a plain {{MessageListener}} is wrapped in
{{ReliableMessageListenerAdapter}}, whose {{isLossTolerant()}} returns false.
So one burst that the route cannot keep up with (the route runs on Hazelcast's
listener thread; a slow processor, a downstream system that slows down) ends
the consumer: the route stays started, receives nothing again until it is
restarted, and Camel logs nothing (Hazelcast logs one warning through its own
logging). The overwritten messages are lost either way; the issue is that every
message after them is lost too.
*Fix*: the reliable topic consumer registers {{CamelReliableMessageListener}},
a {{ReliableMessageListener}} that is loss tolerant: it logs a WARN with the
number of messages lost ("The consumer of the reliable topic news lost 21
messages: they were overwritten in the ringbuffer before they were received")
and goes on with the oldest message still in the ringbuffer. It starts with the
messages published from then on ({{retrieveInitialSequence}} -1), as the plain
listener did. The plain topic is unchanged.
*Test*: new {{HazelcastReliableTopicConsumerLossTest}} (an embedded member,
ringbuffer capacity 10): the route is busy with a first message while 30 more
are published, then one more message is published. On main it is never
received: {{mock://news Received message count. Expected: <1> but was: <0>}}.
With the fix it is, and the WARN reports 21 lost messages. camel-hazelcast unit
tests with the fix: 243 run, 0 failures, 1 error
({{HazelcastUserConfigSerializationFilterWarningTest.warnsForUserClientConfigWithoutFilter}},
a client that cannot reach a cluster in the test environment; it errors the
same way without the change).
*How it was found*: variant analysis of CAMEL-25815 (pattern: the server side
ends a consumer's subscription and the consumer never subscribes again), then
the JUnit test.
*Affected*: main, camel-4.18.x and camel-4.14.x (the same plain listener,
Hazelcast 5.4.0; read, not run).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
