shashank created CAMEL-26970:
--------------------------------
Summary: camel-azure-servicebus - an exchange that completes
during a graceful stop cannot settle its message: deferShutdown closes the
receiver (SDK V2 stack)
Key: CAMEL-26970
URL: https://issues.apache.org/jira/browse/CAMEL-26970
Project: Camel
Issue Type: Bug
Components: camel-azure-servicebus
Reporter: shashank
Assignee: shashank
Found by the Service Bus DST world of CAMEL-26956 (fork branch
{{camel-azure-servicebus-dst-world-v2}}, PR opens with the 11-16 batch): its
default set accepts this case, {{-Ddst.servicebus.strictClose=true}} fails 601
of 1000 seeds. Confirmed outside the DST harness against the SDK classes (javap
of azure-messaging-servicebus 7.17.16) and with a plain unit test (below).
h3. What happens
CAMEL-23260 made {{ServiceBusConsumer}} {{ShutdownAware}} so that exchanges in
flight at a graceful stop can still settle their message. Its {{deferShutdown}}
calls {{client.stop()}} with the comment "stop accepting new messages but keep
the connection open so that in-flight exchanges can still complete/abandon
messages". With the SDK version Camel uses that is not what {{stop()}} does:
* {{ServiceBusProcessorClient.stop()}} calls {{ServiceBusProcessor.stop()}}
when the processor runs on the V2 stack ({{processorV2 != null}});
* the V2 stack is the default for a non-session processor:
{{ServiceBusClientBuilder.V2StackSupport.isNonSessionAsyncReceiveEnabled}} is
true unless {{com.azure.messaging.servicebus.nonSession.asyncReceive.v2}} opts
out;
* {{ServiceBusProcessor.stop()}} is {{close()}}, which disposes the
{{RollingMessagePump}}; the pump is a {{Mono.using}} whose cleanup closes its
{{ServiceBusReceiverAsyncClient}};
* {{ServiceBusReceiverAsyncClient.updateDisposition}} on a disposed receiver
returns {{IllegalStateException}} "Cannot perform operation '...' on a disposed
receiver".
So an exchange that completes after {{deferShutdown}} gets that exception from
{{messageContext.complete()}} (logged as "Exception occurred during
onCompletion"), the message is not settled, and Service Bus delivers it again:
the graceful stop processes it twice. A failed exchange cannot abandon or
dead-letter its message either. This is the log of CAMEL-23260 itself ("Cannot
perform operation 'abandoned' on a disposed receiver"), which the deferral did
not cure on the V2 stack.
h3. Expected
The intent written in {{deferShutdown}} and in CAMEL-23260: during a graceful
stop the consumer stops taking new messages and the exchanges in flight still
settle theirs, before the stop closes the client. Service Bus ("Settle messages
before you close the receiver or connection", dead-letter queue page):
"Settling a message after you close the receiver throws an exception, and the
message isn't settled." and "If you close the receiver or its connection before
you settle a message (complete, abandon, defer, or dead-letter), the settlement
doesn't reach the service, so the message stays locked until the lock expires.
The service then redelivers the message, which increases its delivery count."
h3. Reproduction
New
{{ServiceBusConsumerTest.gracefulStopKeepsTheClientOpenUntilTheExchangesInFlightAreSettled}},
with a processor client mock that behaves as the 7.17 V2 client (stop() and
close() dispose the receiver; a settlement after that throws the
IllegalStateException above): a message is in flight, {{deferShutdown}} runs,
the exchange completes. On main {{complete()}} throws; expected the message
completed and the client closed only by the stop.
h3. Fix
In peek-lock mode {{deferShutdown}} no longer stops the processor client; it
marks the consumer as stopping. A message the client delivers meanwhile is held
in {{processMessage}}, neither routed nor settled, until the consumer stops; as
the client delivers the next message only when {{processMessage}} returns, at
most one message per concurrent call is held ({{maxConcurrentCalls}}, per
session with sessions). The stop closes the client and then releases them;
Service Bus delivers a held message again once its lock expires (its delivery
count goes up by one, as for the exchanges in flight on main). The trade is a
few messages delivered again later without having been routed, instead of every
exchange in flight at the stop being routed twice. In receive-and-delete mode
nothing needs settling and {{deferShutdown}} still stops the client. Branch
{{camel-azure-servicebus-settle-during-graceful-stop}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)