shashank created CAMEL-26970:
--------------------------------

             Summary: camel-azure-servicebus - an exchange that completes 
during a graceful stop cannot settle its message: deferShutdown closes the 
receiver (SDK V2 stack)
                 Key: CAMEL-26970
                 URL: https://issues.apache.org/jira/browse/CAMEL-26970
             Project: Camel
          Issue Type: Bug
          Components: camel-azure-servicebus
            Reporter: shashank
            Assignee: shashank


Found by the Service Bus DST world of CAMEL-26956 (fork branch 
{{camel-azure-servicebus-dst-world-v2}}, PR opens with the 11-16 batch): its 
default set accepts this case, {{-Ddst.servicebus.strictClose=true}} fails 601 
of 1000 seeds. Confirmed outside the DST harness against the SDK classes (javap 
of azure-messaging-servicebus 7.17.16) and with a plain unit test (below).

h3. What happens

CAMEL-23260 made {{ServiceBusConsumer}} {{ShutdownAware}} so that exchanges in 
flight at a graceful stop can still settle their message. Its {{deferShutdown}} 
calls {{client.stop()}} with the comment "stop accepting new messages but keep 
the connection open so that in-flight exchanges can still complete/abandon 
messages". With the SDK version Camel uses that is not what {{stop()}} does:
* {{ServiceBusProcessorClient.stop()}} calls {{ServiceBusProcessor.stop()}} 
when the processor runs on the V2 stack ({{processorV2 != null}});
* the V2 stack is the default for a non-session processor: 
{{ServiceBusClientBuilder.V2StackSupport.isNonSessionAsyncReceiveEnabled}} is 
true unless {{com.azure.messaging.servicebus.nonSession.asyncReceive.v2}} opts 
out;
* {{ServiceBusProcessor.stop()}} is {{close()}}, which disposes the 
{{RollingMessagePump}}; the pump is a {{Mono.using}} whose cleanup closes its 
{{ServiceBusReceiverAsyncClient}};
* {{ServiceBusReceiverAsyncClient.updateDisposition}} on a disposed receiver 
returns {{IllegalStateException}} "Cannot perform operation '...' on a disposed 
receiver".

So an exchange that completes after {{deferShutdown}} gets that exception from 
{{messageContext.complete()}} (logged as "Exception occurred during 
onCompletion"), the message is not settled, and Service Bus delivers it again: 
the graceful stop processes it twice. A failed exchange cannot abandon or 
dead-letter its message either. This is the log of CAMEL-23260 itself ("Cannot 
perform operation 'abandoned' on a disposed receiver"), which the deferral did 
not cure on the V2 stack.

h3. Expected

The intent written in {{deferShutdown}} and in CAMEL-23260: during a graceful 
stop the consumer stops taking new messages and the exchanges in flight still 
settle theirs, before the stop closes the client. Service Bus ("Settle messages 
before you close the receiver or connection", dead-letter queue page): 
"Settling a message after you close the receiver throws an exception, and the 
message isn't settled." and "If you close the receiver or its connection before 
you settle a message (complete, abandon, defer, or dead-letter), the settlement 
doesn't reach the service, so the message stays locked until the lock expires. 
The service then redelivers the message, which increases its delivery count."

h3. Reproduction

New 
{{ServiceBusConsumerTest.gracefulStopKeepsTheClientOpenUntilTheExchangesInFlightAreSettled}},
 with a processor client mock that behaves as the 7.17 V2 client (stop() and 
close() dispose the receiver; a settlement after that throws the 
IllegalStateException above): a message is in flight, {{deferShutdown}} runs, 
the exchange completes. On main {{complete()}} throws; expected the message 
completed and the client closed only by the stop.

h3. Fix

In peek-lock mode {{deferShutdown}} no longer stops the processor client; it 
marks the consumer as stopping. A message the client delivers meanwhile is held 
in {{processMessage}}, neither routed nor settled, until the consumer stops; as 
the client delivers the next message only when {{processMessage}} returns, at 
most one message per concurrent call is held ({{maxConcurrentCalls}}, per 
session with sessions). The stop closes the client and then releases them; 
Service Bus delivers a held message again once its lock expires (its delivery 
count goes up by one, as for the exchanges in flight on main). The trade is a 
few messages delivered again later without having been routed, instead of every 
exchange in flight at the stop being routed twice. In receive-and-delete mode 
nothing needs settling and {{deferShutdown}} still stops the client. Branch 
{{camel-azure-servicebus-settle-during-graceful-stop}}.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to