shashank created CAMEL-25840:
--------------------------------
Summary: camel-hazelcast - the SEDA consumer takes and loses an
item when its route stops: a poll still waiting takes the next item offered,
and its exchange is rejected
Key: CAMEL-25840
URL: https://issues.apache.org/jira/browse/CAMEL-25840
Project: Camel
Issue Type: Bug
Components: camel-hazelcast
Reporter: shashank
Assignee: shashank
{{HazelcastSedaConsumer.doStop}} (main 20cdb309f7d0) shuts its executor down
without waiting ({{ExecutorServiceManager.shutdown}}) and then stops the
processor ({{super.doStop()}}). Its threads usually wait in
{{queue.poll(pollTimeout)}} (1 second by default) at that moment, and they only
look at {{isRunAllowed()}} when the poll returns. A poll takes the item from
the queue, so an item offered in that window is taken by the stopping consumer,
its exchange fails at once with {{RouteStoppingException}} ("The exchange
cannot continue: its route is being stopped or reloaded") and, without
{{transacted}} (the default), the item is gone: it is in neither the queue nor
the route. Each consumer thread ({{concurrentConsumers}}) can lose one item per
stop. On a busy queue that happens at nearly every stop of the route: a route
stop or restart, a {{master:}} or route policy failover, the shutdown of the
application. The consumer logs one WARN ("Error processing exchange") for each
lost item; nothing retries it. With {{transacted=true}} the transaction rolls
back and the item goes back to the queue.
The graceful shutdown does not help: the shutdown strategy stops the consumer
first and only then waits for in-flight exchanges, and the exchange of this
late poll is created after that.
*Fix*: {{doStop}} waits for the consumer threads to end their current poll and
route what it returned (a graceful executor shutdown, bounded by the context's
{{shutdownAwaitTermination}}) before it stops the processor. The scheduler of
the delay after an error is shut down after the threads, since they use it. A
stop takes at most {{pollTimeout}} longer.
*Test*: new {{HazelcastSedaConsumerStopTest}} (an embedded Hazelcast member):
stop the idle route while its thread waits in the poll, then offer an item. On
main the item is taken: {{The item offered after the route stopped was taken
from the queue ==> expected: <1> but was: <0>}}, and the log shows the
{{RouteStoppingException}}. With the fix the item stays in the queue and the
next start routes it. camel-hazelcast unit tests with the fix: 243 run, 0
failures, 1 error
({{HazelcastUserConfigSerializationFilterWarningTest.warnsForUserClientConfigWithoutFilter}},
a client that cannot reach a cluster in the test environment; it errors the
same way without the change).
*How it was found*: variant analysis of CAMEL-25790 (pattern: a consumer with
its own loops, and what its stop does to them), found next to CAMEL-25837 (the
same window in the hazelcast-queue consumer in poll mode), then the JUnit test.
*Affected*: main, camel-4.18.x and camel-4.14.x (the same {{doStop}}; read, not
run). This change touches only {{doStop}}; CAMEL-25816 changes the loop of the
same class, and the two merge without conflict.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
