shashank created CAMEL-25816:
--------------------------------
Summary: camel-hazelcast - a transacted hazelcast-seda consumer
whose route continues on another thread never commits: processed messages come
back when the application leaves the cluster, failed ones are not retried
Key: CAMEL-25816
URL: https://issues.apache.org/jira/browse/CAMEL-25816
Project: Camel
Issue Type: Bug
Components: camel-hazelcast
Reporter: shashank
Assignee: shashank
{{HazelcastSedaConsumer}} (main 936d6f4517a0, {{run()}} lines 86-180, callback
at 116-135) begins a Hazelcast transaction on its polling thread when
{{transacted=true}}, polls the message inside it, and hands the exchange to the
asynchronous routing engine ({{processor.process(exchange, callback)}}). The
{{done}} callback commits the transaction, or rolls it back when the exchange
failed or was marked rollback only.
A Hazelcast transaction belongs to the thread that began it:
{{TransactionImpl.commit()}} and {{rollback()}} check the thread and throw
{{IllegalStateException: Transaction cannot span multiple threads!}} on any
other thread (Hazelcast 5.4.0). When the route continues on another thread, the
callback runs there. That is the case with an asynchronous producer, for
example camel-kafka (the Kafka producer continues routing on its worker pool),
netty or vertx-http, and with {{threads()}} or an asynchronous delay. Then:
* the commit fails; the reactive executor logs {{WARN Error executing reactive
work due to Transaction cannot span multiple threads!. This exception is
ignored.}} and the message stays reserved by a transaction that never ends;
* a failed exchange is not rolled back either, so the failed message is not
taken again (the documented transacted behaviour), it just disappears from the
queue;
* the next poll on the consumer thread fails with {{Nested transactions are not
allowed!}} (the open transaction is still bound to the thread), is reported to
the exception handler and waits {{onErrorDelay}};
* when the application leaves the Hazelcast cluster (a restart or a redeploy),
the other members roll back the transactions it left open, so every message the
route processed since it started is put back into the queue and consumed again.
The option says the messages "will only be removed if the transaction commits,
which happens when the processing is complete"; here they are never removed,
and are consumed twice.
*Fix*: with {{transacted=true}} the consumer thread processes the exchange
synchronously (it waits for the exchange to complete, also when the route
continued on another thread) and commits or rolls back the transaction itself;
the non-transacted path is unchanged. A transacted consumer thread then takes
the next message only when the previous one is complete
({{concurrentConsumers}} gives more threads). Option description and upgrade
note (4.24) updated.
*Tests*: new {{HazelcastSedaTransactedAsyncRouteTest}} (2 tests, two embedded
Hazelcast members on the loopback interface, a route with {{threads(1)}}).
{{testCompletedExchangesAreCommitted}} routes three messages, stops the route
and shuts down the application's member; on main the remaining member gets them
back: {{The processed messages were not removed from the queue ==> expected:
<0> but was: <3>}}. {{testFailedExchangeIsRolledBack}} fails the first attempt;
on main the message is never taken again: {{mock://failed Received message
count. Expected: <1> but was: <0>}}. Both pass with the fix (3 runs).
camel-hazelcast unit tests with the fix: 244 run, 0 failures, 1 error in
{{HazelcastUserConfigSerializationFilterWarningTest.warnsForUserClientConfigWithoutFilter}}
(a Hazelcast client cannot connect to the member in the test environment; it
fails the same way on main).
*How it was found*: reading the transacted consumer for transaction state held
across threads, then a probe with an embedded member (the WARN above, and the
three processed messages back in the queue after the member left), then the
JUnit test above. Not run against a separate Hazelcast cluster.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
(the same callback; read, not run).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
