This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 184121cf5bca CAMEL-26956: camel-azure-servicebus - a DST fake world
for the Service Bus consumer (#30130)
184121cf5bca is described below
commit 184121cf5bca9fdaefc32a6bd6ca62d3ff09da10
Author: allthingssecurity <[email protected]>
AuthorDate: Tue Nov 17 14:28:45 2026 +0530
CAMEL-26956: camel-azure-servicebus - a DST fake world for the Service Bus
consumer (#30130)
…e Bus consumer
Adds a camel-test-dst world for the camel-azure-servicebus consumer in
peek-lock mode, in the dst package of its tests: ServiceBusConsumerDstTest
(1000 seeds), ServiceBusWorld and FakeServiceBus. The real consumer and its
lock renewer run against a fake queue behind the processor and receiver
clients of a mocked ServiceBusClientFactory set on the endpoint, written
from the Service Bus documentation on locks, settlement and dead-lettering
and from azure-messaging-servicebus 7.17: locks for the lock duration of the
queue, redelivery and a higher delivery count after an expiry or an abandon,
dead-lettering past MaxDeliveryCount, complete, abandon and dead-letter that
fail once the lock is lost, renewals that take a while and fail when they
come too late, a processor whose stop() closes its receiver (the V2 stack,
the default of the SDK), after which a settlement throws, and the lock
renewal of the processor client while a processMessage call runs (up to
maxAutoLockRenewDuration, ended when the call returns or the receiver is
closed).
The route has an asynchronous step, so only the lock renewer of the
consumer keeps the locks of the routed messages. The seed picks the lock
duration, the max delivery count, maxConcurrentCalls,
maxAutoLockRenewDuration (60 or 300 seconds), enableDeadLettering and the
fault rates, and injects failed and slow renewals, failed settlements,
failed processing, processing longer than the lock duration, and route
restarts while messages are processed. The operator stops the route as
DefaultShutdownStrategy does, with the wait of the strategy as tasks of
the operator, so that a processMessage call that waits until the consumer
stops does not sit above the stop on the one stack; a stop that times out
forces the consumer to stop, and the exchanges still in flight fail;
their settlements then find the client closed (by design), so
strictClose counts only the settlements inside the graceful wait.
Invariants: no lock is lost and no message is delivered again while the
consumer holds it (a routed delivery whose exchange is not done), unless a
renewal failed, the route was stopped or maxAutoLockRenewDuration passed;
the message of a completed exchange is completed, unless its settlement
failed or was excused; a message is completed only after its exchange
completed, and abandoned or dead-lettered only after it failed (or was
in flight when a stop timed out); once the faults cleared every message is
completed or dead-lettered; the consumer
settles a delivery at most once and renews only what it holds, through an
open client, and routes or settles a delivered message before
processMessage returns, unless the route is being stopped.
Two behaviours are accepted by default and counted with a property: the
consumer stops renewing a message after one renewal error
(-Ddst.servicebus.strictRenew=true, 28 of 1000 seeds fail; CAMEL-26969),
and the settlement of an exchange that completes during the graceful stop
of the route, after the stop() of the processor closed its receiver,
fails, so the message is delivered again
(-Ddst.servicebus.strictClose=true, 96 of 1000 seeds fail; CAMEL-26970).
Self-check: on the code before CAMEL-25701 (4f879148b118 reverted) 76 of
2000 seeds fail; main passes 5000 of 5000 seeds.
The module's pom loads the byte code agent of camel-test-dst, the test log
is capped, and the catalog lists the world.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../camel/catalog/components/azure-servicebus.json | 3 +
.../org/apache/camel/catalog/docs/test-dst.adoc | 10 +
.../camel-azure/camel-azure-servicebus/pom.xml | 26 +
.../azure/servicebus/azure-servicebus.json | 3 +
.../azure/servicebus/dst/FakeServiceBus.java | 467 +++++++++++++++++
.../servicebus/dst/ServiceBusConsumerDstTest.java | 37 ++
.../azure/servicebus/dst/ServiceBusWorld.java | 582 +++++++++++++++++++++
.../src/test/resources/log4j2.properties | 9 +-
.../camel-test-dst/src/main/docs/test-dst.adoc | 10 +
9 files changed, 1146 insertions(+), 1 deletion(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
index a79da4aee71f..1c868480c984 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
@@ -119,5 +119,8 @@
"credentialType": { "index": 28, "kind": "parameter", "displayName":
"Credential Type", "group": "security", "label": "security", "required": false,
"type": "enum", "javaType":
"org.apache.camel.component.azure.common.CredentialType", "enum": [
"AZURE_IDENTITY", "CONNECTION_STRING", "TOKEN_CREDENTIAL" ], "deprecated":
false, "autowired": false, "secret": false, "defaultValue":
"CONNECTION_STRING", "configurationClass":
"org.apache.camel.component.azure.servicebus.ServiceBusConfigurat [...]
"fullyQualifiedNamespace": { "index": 29, "kind": "parameter",
"displayName": "Fully Qualified Namespace", "group": "security", "label":
"security", "required": false, "type": "string", "javaType":
"java.lang.String", "deprecated": false, "autowired": false, "secret": false,
"configurationClass":
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration",
"configurationField": "configuration", "description": "Fully Qualified
Namespace of the service bus" },
"tokenCredential": { "index": 30, "kind": "parameter", "displayName":
"Token Credential", "group": "security", "label": "security", "required":
false, "type": "object", "javaType":
"com.azure.core.credential.TokenCredential", "deprecated": false, "autowired":
true, "secret": true, "security": "secret", "configurationClass":
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration",
"configurationField": "configuration", "description": "A TokenCredential for
Azure AD authe [...]
+ },
+ "dstWorlds": {
+ "ServiceBusWorld": { "javaType":
"org.apache.camel.component.azure.servicebus.dst.ServiceBusWorld", "fake":
"FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count,
dead-letter queue) behind the processor and receiver clients of
ServiceBusClientFactory", "covers": [ "the lock renewer of the consumer
(PEEK_LOCK, maxAutoLockRenewDuration) with an asynchronous route", "processing
that takes longer than the lock duration", "complete, abandon and dead-letter,
also after [...]
}
}
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
index 43249aaa404e..c834da4f739a 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
@@ -91,6 +91,16 @@ a|* checkpointing by checkpointBatchSize and
checkpointBatchTimeout
* an asynchronous downstream step
|CAMEL-25683, CAMEL-25738, CAMEL-25781
+|camel-azure-servicebus
+|`ServiceBusWorld` (`ServiceBusConsumerDstTest`)
+|FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count,
dead-letter queue) behind the processor and receiver clients of
ServiceBusClientFactory
+a|* the lock renewer of the consumer (PEEK_LOCK, maxAutoLockRenewDuration)
with an asynchronous route
+* processing that takes longer than the lock duration
+* complete, abandon and dead-letter, also after the lock was lost
+* maxConcurrentCalls of one to three (with the asynchronous route it bounds
the processMessage calls)
+* route stop and start while messages are processed
+|CAMEL-25701, CAMEL-26956, CAMEL-26969, CAMEL-26970, CAMEL-27066
+
|camel-google-pubsub
|`PubsubWorld` (`PubsubConsumerDstTest`)
|FakePubsub: a subscription (Pull, Acknowledge, ModifyAckDeadline, ack
deadlines, dead letter policy) behind the SubscriberStub of synchronousPull
diff --git a/components/camel-azure/camel-azure-servicebus/pom.xml
b/components/camel-azure/camel-azure-servicebus/pom.xml
index 9687ce9b3207..08eac037e281 100644
--- a/components/camel-azure/camel-azure-servicebus/pom.xml
+++ b/components/camel-azure/camel-azure-servicebus/pom.xml
@@ -36,6 +36,8 @@
<!-- Floci AZ is not available on these platforms -->
<skipITs.ppc64le>true</skipITs.ppc64le>
<skipITs.s390x>true</skipITs.s390x>
+ <!-- the DST tests load the byte code agent of camel-test-dst when the
JVM starts (see test-dst.adoc) -->
+
<camel.surefire.fork.additional-vmargs>"-javaagent:${net.bytebuddy:byte-buddy-agent:jar}"</camel.surefire.fork.additional-vmargs>
</properties>
<dependencies>
@@ -93,5 +95,29 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <!-- deterministic simulation test of the consumer and its lock
renewer (dst package) -->
+ <dependency>
+ <groupId>org.apache.camel</groupId>
+ <artifactId>camel-test-dst</artifactId>
+ <scope>test</scope>
+ </dependency>
</dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <!-- sets the property net.bytebuddy:byte-buddy-agent:jar to
the path of the agent of the DST tests -->
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>dst-agent-path</id>
+ <goals>
+ <goal>properties</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
</project>
diff --git
a/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
b/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
index a79da4aee71f..1c868480c984 100644
---
a/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
+++
b/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
@@ -119,5 +119,8 @@
"credentialType": { "index": 28, "kind": "parameter", "displayName":
"Credential Type", "group": "security", "label": "security", "required": false,
"type": "enum", "javaType":
"org.apache.camel.component.azure.common.CredentialType", "enum": [
"AZURE_IDENTITY", "CONNECTION_STRING", "TOKEN_CREDENTIAL" ], "deprecated":
false, "autowired": false, "secret": false, "defaultValue":
"CONNECTION_STRING", "configurationClass":
"org.apache.camel.component.azure.servicebus.ServiceBusConfigurat [...]
"fullyQualifiedNamespace": { "index": 29, "kind": "parameter",
"displayName": "Fully Qualified Namespace", "group": "security", "label":
"security", "required": false, "type": "string", "javaType":
"java.lang.String", "deprecated": false, "autowired": false, "secret": false,
"configurationClass":
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration",
"configurationField": "configuration", "description": "Fully Qualified
Namespace of the service bus" },
"tokenCredential": { "index": 30, "kind": "parameter", "displayName":
"Token Credential", "group": "security", "label": "security", "required":
false, "type": "object", "javaType":
"com.azure.core.credential.TokenCredential", "deprecated": false, "autowired":
true, "secret": true, "security": "secret", "configurationClass":
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration",
"configurationField": "configuration", "description": "A TokenCredential for
Azure AD authe [...]
+ },
+ "dstWorlds": {
+ "ServiceBusWorld": { "javaType":
"org.apache.camel.component.azure.servicebus.dst.ServiceBusWorld", "fake":
"FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count,
dead-letter queue) behind the processor and receiver clients of
ServiceBusClientFactory", "covers": [ "the lock renewer of the consumer
(PEEK_LOCK, maxAutoLockRenewDuration) with an asynchronous route", "processing
that takes longer than the lock duration", "complete, abandon and dead-letter,
also after [...]
}
}
diff --git
a/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/FakeServiceBus.java
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/FakeServiceBus.java
new file mode 100644
index 000000000000..e89ca458827c
--- /dev/null
+++
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/FakeServiceBus.java
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.servicebus.dst;
+
+import java.time.Instant;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+
+import com.azure.core.amqp.exception.AmqpErrorCondition;
+import com.azure.core.amqp.exception.AmqpException;
+import com.azure.core.util.BinaryData;
+import com.azure.messaging.servicebus.ServiceBusErrorContext;
+import com.azure.messaging.servicebus.ServiceBusErrorSource;
+import com.azure.messaging.servicebus.ServiceBusException;
+import com.azure.messaging.servicebus.ServiceBusProcessorClient;
+import com.azure.messaging.servicebus.ServiceBusReceivedMessage;
+import com.azure.messaging.servicebus.ServiceBusReceivedMessageContext;
+import com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.camel.test.dst.DstLedger;
+import org.apache.camel.test.dst.DstTask;
+import reactor.core.publisher.Mono;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * A fake Azure Service Bus queue in peek-lock mode, and the clients
camel-azure-servicebus gets from its
+ * ServiceBusClientFactory (Mockito mocks of the final SDK classes, whose
behaviour is the queue below). Written from
+ * the Service Bus documentation, not from the Camel consumer:
+ * <ul>
+ * <li>[L] Peek-lock
(https://learn.microsoft.com/azure/service-bus-messaging/message-transfers-locks-settlement):
a
+ * received message is locked for the lock duration of the queue; "When the
lock on a message is explicitly released or
+ * when the lock expires, the message goes back to the front of the retrieval
order for redelivery"; "Whenever a message
+ * is delivered under a peek-lock, but is either explicitly abandoned or the
lock has expired, the delivery count on the
+ * message is incremented. When the delivery count exceeds the limit, the
message is moved to the DLQ"
+ *
(https://learn.microsoft.com/azure/service-bus-messaging/service-bus-dead-letter-queues).</li>
+ * <li>[S] Settlement: complete removes the message, abandon releases the lock
(the message is available again),
+ * deadLetter moves it to the dead-letter queue; "The Complete, DeadLetter, or
RenewLock operations might fail due to
+ * network issues, if the held lock has expired" (assumed: the Java SDK
reports it as a ServiceBusException with reason
+ * MESSAGE_LOCK_LOST). "Settling a message after you close the receiver throws
an exception, and the message isn't
+ * settled" (dead-letter page): in azure-messaging-servicebus 7.17 the
processor of a queue runs the V2 stack by
+ * default, where stop() closes the processor and its receiver, so a
settlement after stop() or close() throws
+ * IllegalStateException. A settlement of a delivery that was settled already
is a misuse (the SDK rejects it).</li>
+ * <li>[R] RenewMessageLock: "The client that owns the lock can extend it"
(assumed: also the receiver client the
+ * consumer creates for its renewals, which is what the consumer relies on);
the call takes a while, and the lock is
+ * extended by the lock duration from when the server handles it, which fails
when the lock was lost by then. A renewal
+ * through a closed receiver client is a misuse.</li>
+ * <li>[P] The processor client (ServiceBusProcessorClient, auto-complete
disabled by the factory): it receives messages
+ * and calls processMessage with up to maxConcurrentCalls calls at a time, and
receives the next message for a call only
+ * when the call returned; stop and close end the receiving (and close the
receiver, see [S]).</li>
+ * <li>[A] The lock renewal of the processor client while processMessage runs
+ *
(https://learn.microsoft.com/java/api/com.azure.messaging.servicebus.servicebusclientbuilder.servicebusprocessorclientbuilder):
+ * "By default, a ServiceBusProcessorClient is configured with auto-completion
and auto-lock renewal capabilities";
+ * maxAutoLockRenewDuration "Sets the amount of time to continue auto-renewing
the lock. Setting Duration#ZERO or null
+ * disables auto-renewal"; "To keep the message locked, the client will have
to continuously renew the message lock
+ * before its expiration. maxAutoLockRenewDuration controls how long the
background renewal task runs" (the factory
+ * passes the maxAutoLockRenewDuration of the endpoint). In
azure-messaging-servicebus 7.17 (javap of 7.17.16) the V2
+ * MessagePump begins the renewal of a message before it calls processMessage
and disposes it when the call returns;
+ * LockRenewalOperation renews when the lock has min(half of the time left, 10
seconds) left, until
+ * maxAutoLockRenewDuration has passed, and ends at the first failed renewal.
The renewal goes through the receiver of
+ * the processor, so it ends when stop() or close() disposed it.</li>
+ * </ul>
+ * Not modelled: sessions, prefetch, deferral, scheduled messages, time to
live, duplicate detection, topics and
+ * subscriptions (a queue only), faults and latency of the renewals of [A]
(the faults are injected on the renewals of
+ * the consumer), connection loss and the processError callback.
+ * <p>
+ * Timing (one-stack rule): a receive takes the message at the end of its
latency; settlement takes effect at once (it
+ * is made from completion callbacks); a renewal does not block (the lock
renewer of the consumer and the renewal of [A]
+ * are background tasks), the latency of a renewal of the consumer is drawn
when it is called and decides whether it
+ * comes in time.
+ */
+final class FakeServiceBus {
+
+ static final String QUEUE = "dst";
+ /** [A] LockRenewalOperation renews when the lock has min(half of the time
left, this) left. */
+ static final long MAX_RENEWAL_BUFFER_MILLIS = 10_000;
+
+ enum State {
+ AVAILABLE,
+ LOCKED,
+ COMPLETED,
+ DEAD_LETTERED
+ }
+
+ static final class Msg {
+ final String id;
+ State state = State.AVAILABLE;
+ int deliveryCount;
+ String lockToken;
+ long lockedUntil;
+
+ Msg(String id) {
+ this.id = id;
+ }
+ }
+
+ private final Dst dst;
+ private final DstLedger ledger;
+ final Map<String, Msg> messages = new LinkedHashMap<>();
+ final List<String> violations = new ArrayList<>();
+ long lockDurationMillis = 30_000;
+ int maxDeliveryCount = 10;
+ double renewFailRate;
+ double settleFailRate;
+ /** The longest latency of a renewal while the faults are on (decided when
the renewal is called). */
+ long renewLatencyMaxMillis;
+ boolean faults = true;
+ /** Called with the delivery ("id#n") when a message is delivered. */
+ Consumer<String> onDelivery = d -> {
+ };
+ /**
+ * Called with the delivery and the expiry of its lock (or -1 when it
holds no lock) when an injected renewal fault
+ * hits it.
+ */
+ BiConsumer<String, Long> onRenewFault = (d, lockedUntil) -> {
+ };
+ /** Called with the delivery when a renewal (of the consumer or of the
processor client) extended its lock. */
+ Consumer<String> onRenewed = d -> {
+ };
+ /** Called with "delivery STATE" when the consumer settles a delivery,
before the outcome is known. */
+ Consumer<String> onSettle = d -> {
+ };
+ /**
+ * Called with "delivery outcome" after it: COMPLETED, ABANDONED,
DEAD_LETTERED, fault, lockLost, closedByStop
+ * (after stop() and before close()) or closed (after close()).
+ */
+ Consumer<String> onSettled = d -> {
+ };
+ /** Called when a processor client is stopped or closed. */
+ Runnable onProcessorClosed = () -> {
+ };
+ /** Called with the delivery when its lock expires. */
+ Consumer<String> onLockExpired = d -> {
+ };
+ private final Set<String> settledDeliveries = new HashSet<>();
+ private int tokens;
+ private int receivers;
+
+ FakeServiceBus(Dst dst, DstLedger ledger) {
+ this.dst = dst;
+ this.ledger = ledger;
+ }
+
+ void send(String id) {
+ messages.put(id, new Msg(id));
+ ledger.record("sent", "q", id);
+ }
+
+ /** [L] locks that expired. */
+ void expire() {
+ for (Msg m : messages.values()) {
+ if (m.state == State.LOCKED && dst.now() >= m.lockedUntil) {
+ ledger.record("lockExpired", "q", m.id + "#" +
m.deliveryCount);
+ onLockExpired.accept(m.id + "#" + m.deliveryCount);
+ available(m, "lock expired");
+ }
+ }
+ }
+
+ private void available(Msg m, String why) {
+ m.lockToken = null;
+ if (m.deliveryCount >= maxDeliveryCount) {
+ m.state = State.DEAD_LETTERED;
+ ledger.record("deadLettered", "q", m.id + "
(MaxDeliveryCountExceeded)");
+ } else {
+ m.state = State.AVAILABLE;
+ }
+ dst.trace("fake: " + m.id + " " + why + " -> " + m.state);
+ }
+
+ boolean settled() {
+ expire();
+ return messages.values().stream()
+ .allMatch(m -> m.state == State.COMPLETED || m.state ==
State.DEAD_LETTERED);
+ }
+
+ String unsettled() {
+ return messages.values().stream().filter(m -> m.state !=
State.COMPLETED && m.state != State.DEAD_LETTERED)
+ .map(m -> m.id + "=" + m.state + "/d" +
m.deliveryCount).toList().toString();
+ }
+
+ private static ServiceBusException lockLost(String token,
ServiceBusErrorSource source) {
+ return new ServiceBusException(
+ new AmqpException(
+ false, AmqpErrorCondition.MESSAGE_LOCK_LOST,
+ "dst: the lock " + token + " is lost or expired",
null),
+ source);
+ }
+
+ private static ServiceBusException transientFailure(ServiceBusErrorSource
source) {
+ return new ServiceBusException(
+ new AmqpException(true, AmqpErrorCondition.SERVER_BUSY_ERROR,
"dst: the server is busy", null), source);
+ }
+
+ /** The expiry of the lock with this token, or -1 when no message holds it
(without expiring the locks first). */
+ private long lockedUntil(String token) {
+ for (Msg m : messages.values()) {
+ if (m.state == State.LOCKED && token.equals(m.lockToken)) {
+ return m.lockedUntil;
+ }
+ }
+ return -1;
+ }
+
+ private Msg locked(String token) {
+ expire();
+ for (Msg m : messages.values()) {
+ if (m.state == State.LOCKED && token.equals(m.lockToken)) {
+ return m;
+ }
+ }
+ return null;
+ }
+
+ /** ServiceBusClientFactory.createServiceBusProcessorClient: [P] a
processor client on this queue. */
+ ServiceBusProcessorClient newProcessorClient(
+ int maxConcurrentCalls, long maxAutoLockRenewMillis,
+ Consumer<ServiceBusReceivedMessageContext> processMessage,
Consumer<ServiceBusErrorContext> processError) {
+ ServiceBusProcessorClient client =
mock(ServiceBusProcessorClient.class);
+ String name = "processor-" + receivers++;
+ DstExecutor calls = dst.executor(name + "-calls", maxConcurrentCalls);
+ DstExecutor receiver = dst.executor(name + "-receive", 1);
+ DstExecutor lockRenewal = dst.executor(name + "-lock-renewal",
Integer.MAX_VALUE);
+ boolean[] running = new boolean[1];
+ // 0: open, 1: stopped (stop() closed the receiver), 2: closed
+ int[] closed = new int[1];
+ int[] inCalls = new int[1];
+ Runnable[] receive = new Runnable[1];
+ receive[0] = () -> {
+ if (!running[0]) {
+ return;
+ }
+ expire();
+ Msg next = inCalls[0] < maxConcurrentCalls
+ ? messages.values().stream().filter(m -> m.state ==
State.AVAILABLE).findFirst().orElse(null)
+ : null;
+ if (next == null) {
+ receiver.schedule(name + " receive", 100, receive[0]);
+ return;
+ }
+ // reads take effect at the end of their latency: the receive
takes the message after it
+ dst.block(dst.between(5, 50), name + " receive");
+ expire();
+ if (!running[0] || next.state != State.AVAILABLE) {
+ receiver.schedule(name + " receive", 0, receive[0]);
+ return;
+ }
+ next.state = State.LOCKED;
+ next.deliveryCount++;
+ next.lockToken = "lock-" + tokens++;
+ next.lockedUntil = dst.now() + lockDurationMillis;
+ String delivery = next.id + "#" + next.deliveryCount;
+ ledger.record("delivered", "q", delivery);
+ onDelivery.accept(delivery);
+ ServiceBusReceivedMessageContext context = context(next,
next.lockToken, delivery, closed);
+ inCalls[0]++;
+ String token = next.lockToken;
+ calls.schedule(name + " processMessage " + delivery, 0, () -> {
+ // [A] the renewal of the lock while the call runs, disposed
when it returns
+ DstTask<?>[] renewal = new DstTask<?>[1];
+ if (maxAutoLockRenewMillis > 0) {
+ Msg m = locked(token);
+ if (m != null) {
+ renewLockWhileProcessed(lockRenewal, renewal, token,
delivery, m.lockedUntil,
+ dst.now() + maxAutoLockRenewMillis, closed);
+ }
+ }
+ try {
+ processMessage.accept(context);
+ } finally {
+ if (renewal[0] != null) {
+ renewal[0].cancel(false);
+ }
+ inCalls[0]--;
+ }
+ });
+ receiver.schedule(name + " receive", 0, receive[0]);
+ };
+ doAnswer(i -> {
+ if (closed[0] != 0) {
+ violations.add(name + " started after it was closed");
+ return null;
+ }
+ if (!running[0]) {
+ running[0] = true;
+ dst.trace("fake: " + name + " start");
+ receiver.schedule(name + " receive", 0, receive[0]);
+ }
+ return null;
+ }).when(client).start();
+ doAnswer(i -> {
+ // [P] [S] V2 stack: stop() closes the processor and its receiver
+ running[0] = false;
+ closed[0] = Math.max(closed[0], 1);
+ dst.trace("fake: " + name + " stop (closes the receiver)");
+ onProcessorClosed.run();
+ return null;
+ }).when(client).stop();
+ doAnswer(i -> {
+ running[0] = false;
+ closed[0] = 2;
+ dst.trace("fake: " + name + " close");
+ onProcessorClosed.run();
+ return null;
+ }).when(client).close();
+ when(client.isRunning()).thenAnswer(i -> running[0]);
+ return client;
+ }
+
+ /**
+ * [A] The next renewal of the processor client for a message whose
processMessage call runs: when the lock it knows
+ * of has min(half of the time left, 10 seconds) left; it ends once
maxAutoLockRenewDuration has passed, when the
+ * receiver of the processor is closed, or at the first failed renewal
(the lock was lost).
+ */
+ private void renewLockWhileProcessed(
+ DstExecutor lockRenewal, DstTask<?>[] renewal, String token,
String delivery, long lockedUntil, long until,
+ int[] closed) {
+ long left = lockedUntil - dst.now();
+ long delay = left < 400 ? 0 : left - Math.min(left / 2,
MAX_RENEWAL_BUFFER_MILLIS);
+ renewal[0] = lockRenewal.schedule("processor lock renewal of " +
delivery, delay, () -> {
+ renewal[0] = null;
+ if (closed[0] != 0 || dst.now() >= until) {
+ return;
+ }
+ Msg m = locked(token);
+ if (m == null) {
+ ledger.record("processorRenewLockLost", "q", delivery);
+ return;
+ }
+ m.lockedUntil = dst.now() + lockDurationMillis;
+ ledger.record("processorRenewed", "q", delivery);
+ onRenewed.accept(delivery);
+ renewLockWhileProcessed(lockRenewal, renewal, token, delivery,
m.lockedUntil, until, closed);
+ });
+ }
+
+ private ServiceBusReceivedMessageContext context(Msg m, String token,
String delivery, int[] closed) {
+ ServiceBusReceivedMessage message =
mock(ServiceBusReceivedMessage.class);
+ when(message.getBody()).thenReturn(BinaryData.fromString(m.id));
+ when(message.getMessageId()).thenReturn(m.id);
+ when(message.getLockToken()).thenReturn(token);
+ when(message.getDeliveryCount()).thenReturn((long) m.deliveryCount);
+ when(message.getLockedUntil())
+
.thenReturn(OffsetDateTime.ofInstant(Instant.ofEpochMilli(Dst.EPOCH_MILLIS +
m.lockedUntil), ZoneOffset.UTC));
+ when(message.getApplicationProperties()).thenReturn(Map.of());
+ ServiceBusReceivedMessageContext context =
mock(ServiceBusReceivedMessageContext.class);
+ when(context.getMessage()).thenReturn(message);
+ doAnswer(i -> {
+ settle(token, delivery, State.COMPLETED,
ServiceBusErrorSource.COMPLETE, closed[0]);
+ return null;
+ }).when(context).complete();
+ doAnswer(i -> {
+ settle(token, delivery, State.AVAILABLE,
ServiceBusErrorSource.ABANDON, closed[0]);
+ return null;
+ }).when(context).abandon();
+ doAnswer(i -> {
+ settle(token, delivery, State.DEAD_LETTERED,
ServiceBusErrorSource.UNKNOWN, closed[0]);
+ return null;
+ }).when(context).deadLetter();
+ doAnswer(i -> {
+ settle(token, delivery, State.DEAD_LETTERED,
ServiceBusErrorSource.UNKNOWN, closed[0]);
+ return null;
+ }).when(context).deadLetter(any());
+ return context;
+ }
+
+ /** [S] a settlement with the lock token of a delivery. */
+ private void settle(String token, String delivery, State to,
ServiceBusErrorSource source, int closed) {
+ onSettle.accept(delivery + " " + to);
+ // the SDK checks a disposed receiver first, and marks a message
settled only once a disposition succeeded
+ if (closed != 0) {
+ ledger.record("settleAfterClose", "q", delivery + " " + to);
+ onSettled.accept(delivery + (closed == 1 ? " closedByStop" : "
closed"));
+ throw new IllegalStateException("Cannot perform operation on a
disposed receiver.");
+ }
+ if (settledDeliveries.contains(delivery)) {
+ violations.add(delivery + " settled again (" + to + ")");
+ throw new IllegalArgumentException("The message has either been
deleted or already settled.");
+ }
+ if (faults && dst.chance(settleFailRate)) {
+ ledger.record("settleFault", "q", delivery);
+ onSettled.accept(delivery + " fault");
+ throw Dst.injected(transientFailure(source));
+ }
+ Msg m = locked(token);
+ if (m == null) {
+ ledger.record("lockLost", "q", delivery + " " + to);
+ onSettled.accept(delivery + " lockLost");
+ throw lockLost(token, source);
+ }
+ ledger.record(to.name().toLowerCase(), "q", delivery);
+ settledDeliveries.add(delivery);
+ onSettled.accept(delivery + " " + (to == State.AVAILABLE ? "ABANDONED"
: to.name()));
+ if (to == State.AVAILABLE) {
+ available(m, "abandoned");
+ } else {
+ m.state = to;
+ m.lockToken = null;
+ }
+ }
+
+ /** ServiceBusClientFactory.createServiceBusReceiverAsyncClient: [R] the
lock renewals of the consumer. */
+ ServiceBusReceiverAsyncClient newRenewalClient() {
+ ServiceBusReceiverAsyncClient client =
mock(ServiceBusReceiverAsyncClient.class);
+ boolean[] closed = new boolean[1];
+ doAnswer(i -> {
+ closed[0] = true;
+ return null;
+ }).when(client).close();
+
when(client.renewMessageLock(any(ServiceBusReceivedMessage.class))).thenAnswer(i
-> {
+ ServiceBusReceivedMessage message = i.getArgument(0);
+ String token = message.getLockToken();
+ String delivery = message.getMessageId() + "#" +
message.getDeliveryCount();
+ if (closed[0]) {
+ violations.add("a renewal of " + delivery + " through a closed
receiver client");
+ return Mono.error(new IllegalStateException("Cannot perform
operation on a disposed receiver."));
+ }
+ if (settledDeliveries.contains(delivery)) {
+ violations.add("a renewal of " + delivery + " after it was
settled");
+ }
+ if (faults && dst.chance(renewFailRate)) {
+ ledger.record("renewFault", "q", delivery);
+ onRenewFault.accept(delivery, lockedUntil(token));
+ return
Mono.error(Dst.injected(transientFailure(ServiceBusErrorSource.RENEW_LOCK)));
+ }
+ // the server handles the renewal after its latency: a lock that
expires before that is lost
+ long latency = faults && renewLatencyMaxMillis > 0 ?
dst.between(1, renewLatencyMaxMillis) : 1;
+ Msg m = locked(token);
+ if (m == null || dst.now() + latency >= m.lockedUntil) {
+ ledger.record("renewLockLost", "q", delivery);
+ return Mono.error(lockLost(token,
ServiceBusErrorSource.RENEW_LOCK));
+ }
+ m.lockedUntil = dst.now() + latency + lockDurationMillis;
+ ledger.record("renewed", "q", delivery);
+ onRenewed.accept(delivery);
+ return Mono.just(
+
OffsetDateTime.ofInstant(Instant.ofEpochMilli(Dst.EPOCH_MILLIS +
m.lockedUntil), ZoneOffset.UTC));
+ });
+ return client;
+ }
+}
diff --git
a/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusConsumerDstTest.java
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusConsumerDstTest.java
new file mode 100644
index 000000000000..87db721adf59
--- /dev/null
+++
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusConsumerDstTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.servicebus.dst;
+
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstInstrument;
+import org.apache.camel.test.dst.DstTest;
+
+/**
+ * Deterministic simulation test of the camel-azure-servicebus consumer in
peek-lock mode against a fake queue, with an
+ * asynchronous route slower than the lock duration, failed lock renewals and
settlements, and route stops while
+ * messages are processed: no message is delivered again while the consumer
holds it, a message is settled as its
+ * exchange ended, and once the faults cleared every message is completed or
dead-lettered. A failing seed is replayed
+ * with {@code -Ddst.seed=<seed>}; {@code -Ddst.seeds=5000} runs more seeds.
+ */
+@DstInstrument("org.apache.camel.component.azure.servicebus")
+class ServiceBusConsumerDstTest {
+
+ @DstTest(seeds = 1000)
+ void noMessageIsDeliveredAgainWhileTheConsumerHoldsIt(Dst dst) throws
Exception {
+ new ServiceBusWorld(dst).run();
+ }
+}
diff --git
a/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusWorld.java
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusWorld.java
new file mode 100644
index 000000000000..543e3fb45cf4
--- /dev/null
+++
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusWorld.java
@@ -0,0 +1,582 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.servicebus.dst;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Consumer;
+
+import com.azure.messaging.servicebus.ServiceBusErrorContext;
+import com.azure.messaging.servicebus.ServiceBusReceivedMessageContext;
+import org.apache.camel.AsyncCallback;
+import org.apache.camel.Exchange;
+import org.apache.camel.Route;
+import org.apache.camel.ServiceStatus;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.azure.servicebus.ServiceBusComponent;
+import org.apache.camel.component.azure.servicebus.ServiceBusConfiguration;
+import org.apache.camel.component.azure.servicebus.ServiceBusConstants;
+import org.apache.camel.component.azure.servicebus.ServiceBusEndpoint;
+import
org.apache.camel.component.azure.servicebus.client.ServiceBusClientFactory;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.apache.camel.spi.ShutdownAware;
+import org.apache.camel.support.AsyncProcessorSupport;
+import org.apache.camel.support.service.ServiceHelper;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstCamel;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.camel.test.dst.DstLedger;
+import org.apache.camel.test.dst.DstWorld;
+import org.apache.camel.test.dst.Invariants;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * The world around a camel-azure-servicebus consumer in peek-lock mode: a
fake queue ({@link FakeServiceBus}), a
+ * sender, a route with an asynchronous step (a downstream call) that is
sometimes longer than the lock duration and
+ * sometimes fails, and an operator that stops and starts the route while
messages are processed. Faults: lock renewals
+ * fail or take long, settlements fail.
+ * <p>
+ * The route is asynchronous, so processMessage returns at once and the lock
of a routed message is kept only by the
+ * lock renewer of the consumer (maxAutoLockRenewDuration), which renews every
message whose lock expires before the run
+ * after the next one of its period task (every 10 seconds, or a second
later). A message the consumer holds in
+ * processMessage without routing it keeps its lock through the renewal of the
processor client while the call runs (see
+ * FakeServiceBus [A]); the invariants count only the routed deliveries as
held by the consumer. The seed picks the lock
+ * duration of the queue, the max delivery count, maxConcurrentCalls (with the
asynchronous route it bounds only the
+ * concurrent processMessage calls), maxAutoLockRenewDuration (60 or 300
seconds), enableDeadLettering, the latency of
+ * the renewals and the fault rates.
+ * <p>
+ * Properties: no lock is lost, and no message is delivered again, while the
consumer holds the message, unless an
+ * injected renewal of its lock failed less than one run of the lock renewer
before the lock expired, the route was
+ * stopped while it was processed (once a stop timed out the consumer stops
renewing, and the exchange goes on), or
+ * maxAutoLockRenewDuration passed. The consumer renews a lock again after a
transient renewal error (CAMEL-26969), but
+ * only at its next run (CAMEL-27066): the lock renewer runs every 10 seconds
or a second later, so a transient renewal
+ * error in the last 11 seconds of a lock (plus the latency of the renewal)
can still lose it. An earlier failed renewal
+ * is no excuse, so a renewer that gave up after one error would fail. {@code
-Ddst.servicebus.strictRenew=true} does
+ * not excuse a failed renewal at all, and still fails on some seeds until
CAMEL-27066 is fixed. The message of a
+ * completed exchange is completed, unless its settlement failed, found its
lock lost under one of those excuses, or
+ * came after the close() of a stop that timed out while the exchange was in
flight (by design); a graceful stop keeps
+ * the processor client open for the exchanges in flight (the V2 processor of
the SDK closes its receiver on stop(), see
+ * FakeServiceBus; CAMEL-26970), so a settlement inside the graceful wait must
find it open; a message is completed only
+ * after its exchange completed, and abandoned or dead-lettered only after it
failed (or was in flight when a stop timed
+ * out, see below); once the faults cleared every message is completed or
dead-lettered; the consumer settles each
+ * delivery at most once and renews only what it holds, through an open client.
+ * <p>
+ * The operator stops the route as DefaultShutdownStrategy does (its
ShutdownTask), with the wait of the strategy as
+ * tasks of the operator (see {@link #gracefulStop}): a processMessage call
that waits until the consumer stops would
+ * otherwise sit above the stop on the one stack, and could not see the stop.
When the stop times out, the consumer is
+ * forced to stop as the strategy does, and the exchanges still in flight fail
(their route rejects them).
+ * <p>
+ * Not explored: sessions, receiveAndDelete mode, a processorClient given by
the user (the consumer does not renew its
+ * locks), a synchronous route (a processMessage call that lasts as long as
the exchange), maxAutoLockRenewDuration=0
+ * (no renewal by the consumer or by the processor client), topics.
+ */
+@DstWorld(component = "azure-servicebus",
+ fake = "FakeServiceBus: a queue in peek-lock mode (lock duration,
delivery count, dead-letter queue) behind"
+ + " the processor and receiver clients of
ServiceBusClientFactory",
+ covers = {
+ "the lock renewer of the consumer (PEEK_LOCK,
maxAutoLockRenewDuration) with an asynchronous route",
+ "processing that takes longer than the lock duration",
+ "complete, abandon and dead-letter, also after the lock was
lost",
+ "maxConcurrentCalls of one to three (with the asynchronous
route it bounds the processMessage calls)",
+ "route stop and start while messages are processed" },
+ faults = {
+ "a lock renewal fails", "a lock renewal takes long", "a
settlement fails", "processing fails",
+ "processing is slower than the lock duration" },
+ invariants = {
+ "no lock is lost and no message is delivered again while the
consumer holds it, unless a renewal"
+ + " failed less than one renewer run before the lock
expired, the route was stopped or"
+ + " maxAutoLockRenewDuration passed",
+ "the message of a completed exchange is completed, unless
its settlement failed, its lock was lost"
+ + " as excused
above, or a forced stop closed the client first",
+ "a message is completed only after its exchange completed",
+ "a message is abandoned or dead-lettered only after its
exchange failed",
+ "every message is completed or dead-lettered once the faults
cleared",
+ "the consumer settles a delivery at most once and renews
only what it holds, through an open client" },
+ issues = { "CAMEL-25701", "CAMEL-26956", "CAMEL-26969",
"CAMEL-26970", "CAMEL-27066" })
+final class ServiceBusWorld {
+
+ static final String ROUTE = "servicebus";
+ /**
+ * Excuses a lost lock only when the renewal found it lost, not after an
injected transient renewal failure: the
+ * consumer retries a failed renewal only at its next run (CAMEL-27066),
so this fails on some seeds until it is
+ * fixed.
+ */
+ static final boolean STRICT_RENEW =
Boolean.getBoolean("dst.servicebus.strictRenew");
+ /** The longest time between two runs of the lock renewer of the consumer:
every 10 seconds, or a second later. */
+ static final long RENEWER_RUN_MILLIS = 11_000;
+
+ private final Dst dst;
+ private final DstLedger ledger;
+ private final FakeServiceBus bus;
+ private final int maxConcurrentCalls;
+ private final boolean deadLettering;
+ private final double processFailRate;
+ private final long maxAutoLockRenewDuration;
+ private final List<String> violations = new ArrayList<>();
+ private final List<String> lostWhileHeld = new ArrayList<>();
+ /** The time each delivery was routed (the lock renewer of the consumer
counts maxAutoLockRenewDuration from it). */
+ private final Map<String, Long> routedAt = new HashMap<>();
+ /**
+ * The outcome of the settlement of each delivery (COMPLETED, ABANDONED,
DEAD_LETTERED, fault, lockLost,
+ * closedByStop, closed).
+ */
+ private final Map<String, String> outcomes = new HashMap<>();
+ private final List<String> notCompleted = new ArrayList<>();
+ private final List<String> redeliveredWhileHeld = new ArrayList<>();
+ private final List<String> wrongSettlements = new ArrayList<>();
+ /**
+ * The deliveries ("id#n") whose exchange is in progress, by message id:
routed and not done yet. A delivery the
+ * consumer holds in processMessage without routing it is not counted (the
processor client renews its lock).
+ */
+ private final Map<String, String> held = new HashMap<>();
+ /**
+ * Every delivery whose exchange is in progress (a message delivered again
while an earlier delivery of it is still
+ * in progress has two).
+ */
+ private final Set<String> inProgress = new HashSet<>();
+ /**
+ * The deliveries whose last renewal failed less than one run of the lock
renewer (RENEWER_RUN_MILLIS, plus the
+ * latency of a renewal) before their lock expired: the retry at the next
run comes too late (CAMEL-27066). A later
+ * renewal that extended the lock removes the delivery again.
+ */
+ private final Set<String> renewFaulted = new HashSet<>();
+ /** The deliveries in progress when the route was stopped: the consumer
stops renewing their locks. */
+ private final Set<String> heldAtStop = new HashSet<>();
+ /**
+ * The deliveries in progress when a graceful stop timed out and the
consumer was forced to stop: stopping the
+ * consumer stops the processor of its route, which then rejects the
exchange (it fails, and its message is
+ * abandoned or dead-lettered, through the closed client), even when the
downstream call answered.
+ */
+ private final Set<String> forcedAtStop = new HashSet<>();
+ /** The deliveries of forcedAtStop abandoned or dead-lettered: that
settlement must find the client closed. */
+ private final Set<String> forcedSettlements = new HashSet<>();
+ /** The deliveries routed, or settled (an abandon when routing failed), by
the consumer. */
+ private final Set<String> routedOrSettled = new HashSet<>();
+ /**
+ * Whether the operator stopped the route and has not started it again
(from deferShutdown until the restart): with
+ * the CAMEL-26970 fix a call that comes after the stop closed the client
returns without routing, as the consumer
+ * stays in its stopping state until it is started again.
+ */
+ private boolean stopping;
+ private final Set<String> completed = new HashSet<>();
+ private final Set<String> failed = new HashSet<>();
+ private boolean faults = true;
+ /** Exchanges whose asynchronous step has not answered yet. */
+ private int inFlight;
+ private int sendsLeft;
+ private int opsLeft;
+ private boolean finished;
+ private String finalStatus;
+ private boolean finalSettled;
+
+ ServiceBusWorld(Dst dst) {
+ this.dst = dst;
+ this.ledger = new DstLedger(dst);
+ this.bus = new FakeServiceBus(dst, ledger);
+ bus.lockDurationMillis = dst.pick(List.of(30_000L, 60_000L));
+ bus.maxDeliveryCount = dst.pick(List.of(3, 10));
+ bus.renewFailRate = dst.pick(List.of(0.0, 0.05, 0.2));
+ bus.settleFailRate = dst.pick(List.of(0.0, 0.05));
+ bus.renewLatencyMaxMillis = dst.pick(List.of(0L, 2000L));
+ this.maxAutoLockRenewDuration = dst.pick(List.of(60_000L, 300_000L));
+ this.maxConcurrentCalls = dst.pick(List.of(1, 2, 3));
+ this.deadLettering = dst.chance(0.3);
+ this.processFailRate = dst.pick(List.of(0.0, 0.05, 0.2));
+ bus.onDelivery = this::delivered;
+ bus.onRenewFault = this::renewFault;
+ bus.onRenewed = renewFaulted::remove;
+ bus.onSettle = this::settling;
+ bus.onSettled = this::settled;
+ bus.onLockExpired = this::lockExpired;
+ // what is in flight when the processor client stops: the stop happens
inside the route stop, after the
+ // operator took its own snapshot, and a message can be delivered in
between
+ bus.onProcessorClosed = () -> heldAtStop.addAll(inProgress);
+ dst.trace("world: " + config());
+ }
+
+ private String config() {
+ return "lockDuration=" + bus.lockDurationMillis + " maxDeliveryCount="
+ bus.maxDeliveryCount
+ + " maxConcurrentCalls=" + maxConcurrentCalls + "
maxAutoLockRenewDuration=" + maxAutoLockRenewDuration
+ + " enableDeadLettering=" + deadLettering + " renewFail=" +
bus.renewFailRate + " renewLatencyMax="
+ + bus.renewLatencyMaxMillis + " settleFail=" +
bus.settleFailRate + " processFail=" + processFailRate;
+ }
+
+ @SuppressWarnings("unchecked")
+ void run() throws Exception {
+ DefaultCamelContext context = DstCamel.newContext(dst);
+ ServiceBusClientFactory factory = mock(ServiceBusClientFactory.class);
+
when(factory.createServiceBusProcessorClient(any(ServiceBusConfiguration.class),
any(Consumer.class),
+ any(Consumer.class))).thenAnswer(
+ i -> bus.newProcessorClient(
+ i.<ServiceBusConfiguration>
getArgument(0).getMaxConcurrentCalls(),
+ i.<ServiceBusConfiguration>
getArgument(0).getMaxAutoLockRenewDuration(),
+
checkedProcessMessage(i.<Consumer<ServiceBusReceivedMessageContext>>
getArgument(1)),
+ i.<Consumer<ServiceBusErrorContext>>
getArgument(2)));
+
when(factory.createServiceBusReceiverAsyncClient(any(ServiceBusConfiguration.class)))
+ .thenAnswer(i -> bus.newRenewalClient());
+ context.addComponent("azure-servicebus", new ServiceBusComponent());
+ String uri = "azure-servicebus:" + FakeServiceBus.QUEUE
+ +
"?connectionString=RAW(Endpoint=sb://dst.servicebus.windows.net/;SharedAccessKeyName=dst;"
+ +
"SharedAccessKey=dst)&serviceBusReceiveMode=PEEK_LOCK&maxConcurrentCalls=" +
maxConcurrentCalls
+ + "&enableDeadLettering=" + deadLettering +
"&maxAutoLockRenewDuration=" + maxAutoLockRenewDuration;
+ DstExecutor downstream = dst.executor("downstream", Integer.MAX_VALUE);
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ // the injected processing failures are expected: no error
handler log for each of them
+ errorHandler(defaultErrorHandler().logExhausted(false));
+ from(uri).routeId(ROUTE).autoStartup(false)
+ .process(new AsyncProcessorSupport() {
+ @Override
+ public boolean process(Exchange exchange,
AsyncCallback callback) {
+ return downstream(exchange, callback,
downstream);
+ }
+ });
+ }
+ });
+ ((ServiceBusEndpoint)
context.getEndpoint(uri)).setServiceBusClientFactory(factory);
+ context.start();
+ DstExecutor operator = dst.executor("operator", 1);
+ try {
+ operator.schedule("start route", 0, () -> startRoute(context));
+ planSends();
+ planOperations(context, operator);
+ operator.schedule("finisher", 1000, () -> finisher(context,
operator, -1));
+ dst.runUntil(() -> finished, "the finisher");
+ dst.check(
+ Invariants.of("the consumer settles a delivery at most
once and renews only what it holds, through an"
+ + " open client",
+ () -> bus.violations.isEmpty() &&
violations.isEmpty()
+ ? null : bus.violations + " " + violations
+ " (" + config() + ")"),
+ Invariants.of("no lock is lost and no message is delivered
again while the consumer holds it,"
+ + " unless a renewal failed less than one
renewer run before the lock expired, the"
+ + " route was stopped or
maxAutoLockRenewDuration passed",
+ () -> redeliveredWhileHeld.isEmpty() &&
lostWhileHeld.isEmpty()
+ ? null : redeliveredWhileHeld + " " +
lostWhileHeld + " (" + config() + ")"),
+ Invariants.of("the message of a completed exchange is
completed, unless its settlement failed, its"
+ + " lock was lost as excused above, or a
forced stop closed the client first",
+ () -> notCompleted.isEmpty() ? null : notCompleted
+ " (" + config() + ")"),
+ Invariants.of("a message is completed only after its
exchange completed",
+ () -> wrongSettlements.stream().noneMatch(s ->
s.contains("COMPLETED"))
+ ? null : wrongSettlements + " (" +
config() + ")"),
+ Invariants.of("a message is abandoned or dead-lettered
only after its exchange failed",
+ () -> wrongSettlements.stream().allMatch(s ->
s.contains("COMPLETED"))
+ ? null : wrongSettlements + " (" +
config() + ")"),
+ Invariants.of("every message is completed or dead-lettered
once the faults cleared",
+ () -> finalSettled && "Started".equals(finalStatus)
+ ? null
+ : "route " + finalStatus + ", unsettled "
+ bus.unsettled() + " (" + config() + ")"));
+ } finally {
+ dst.dropPending();
+ // with an exchange still in flight (its answer was dropped) the
shutdown would wait for it for ever on the
+ // one stack
+ if (inFlight == 0) {
+ context.stop();
+ }
+ }
+ }
+
+ /**
+ * The processMessage callback of the consumer, checked: unless the
operator stopped the route and has not started
+ * it again, a delivered message is routed or settled by the time the call
returns (the route is synchronous up to
+ * its downstream call). Once the route is stopped the consumer may hold
the message in the call, unrouted, until it
+ * stops, and return without routing it until the route is started again.
+ */
+ private Consumer<ServiceBusReceivedMessageContext> checkedProcessMessage(
+ Consumer<ServiceBusReceivedMessageContext> processMessage) {
+ return context -> {
+ boolean duringStop = stopping;
+ String delivery = context.getMessage().getMessageId() + "#" +
context.getMessage().getDeliveryCount();
+ processMessage.accept(context);
+ if (!duringStop && !routedOrSettled.contains(delivery)) {
+ violations.add(delivery + " neither routed nor settled when
processMessage returned at t=" + dst.now());
+ }
+ };
+ }
+
+ /** The asynchronous step of the route: a downstream call that answers
after a latency, and may fail. */
+ private boolean downstream(Exchange exchange, AsyncCallback callback,
DstExecutor downstream) {
+ String id =
exchange.getMessage().getHeader(ServiceBusConstants.MESSAGE_ID, String.class);
+ Long count =
exchange.getMessage().getHeader(ServiceBusConstants.DELIVERY_COUNT, Long.class);
+ String delivery = id + "#" + count;
+ ledger.record("routed", "q", delivery);
+ // from now on the consumer holds the message, until its exchange is
done
+ held.put(id, delivery);
+ inProgress.add(delivery);
+ routedOrSettled.add(delivery);
+ routedAt.put(delivery, dst.now());
+ long latency = faults && dst.chance(0.2)
+ ? dst.between(bus.lockDurationMillis / 2, 3 *
bus.lockDurationMillis)
+ : dst.between(10, 500);
+ boolean fail = faults && dst.chance(processFailRate);
+ inFlight++;
+ downstream.schedule("answer for " + delivery, latency, () -> {
+ inFlight--;
+ if (fail) {
+ failed.add(delivery);
+ ledger.record("failed", "q", delivery);
+ exchange.setException(Dst.injected(new
IllegalStateException("dst: processing failed for " + delivery)));
+ } else {
+ completed.add(delivery);
+ ledger.record("completed", "q", delivery);
+ }
+ held.remove(id, delivery);
+ inProgress.remove(delivery);
+ callback.done(false);
+ });
+ return false;
+ }
+
+ /** The queue delivered a message: it must not be one whose exchange is in
progress. */
+ private void delivered(String delivery) {
+ String id = delivery.substring(0, delivery.indexOf('#'));
+ String previous = held.get(id);
+ if (previous != null && !excused(previous)) {
+ redeliveredWhileHeld.add(delivery + " at t=" + dst.now() + " while
" + previous + " is processed");
+ }
+ }
+
+ /**
+ * An injected renewal failure excuses the loss of the lock only when the
next run of the lock renewer comes too
+ * late to renew it: the renewer runs every 10 seconds or a second later
(the period task scheduler ticks every
+ * second), and the renewal itself takes up to renewLatencyMaxMillis.
+ */
+ private void renewFault(String delivery, long lockedUntil) {
+ if (lockedUntil < 0) {
+ // the lock was lost already: its loss was judged when it expired
+ return;
+ }
+ if (lockedUntil - dst.now() <= RENEWER_RUN_MILLIS +
bus.renewLatencyMaxMillis) {
+ renewFaulted.add(delivery);
+ } else {
+ renewFaulted.remove(delivery);
+ }
+ }
+
+ /**
+ * Why the consumer may lose the lock of a delivery it holds: an injected
renewal failure in the last run of the
+ * lock renewer before the lock expired (the consumer renews again only at
its next run, CAMEL-27066), a route stop
+ * while it was processed (the consumer stops renewing once the stop timed
out), or a hold longer than
+ * maxAutoLockRenewDuration.
+ */
+ private boolean excused(String delivery) {
+ Long at = routedAt.get(delivery);
+ return !STRICT_RENEW && renewFaulted.contains(delivery) ||
heldAtStop.contains(delivery)
+ || at != null && dst.now() >= at + maxAutoLockRenewDuration;
+ }
+
+ /** The queue let the lock of a delivery expire: the consumer must renew
the locks of what it holds. */
+ private void lockExpired(String delivery) {
+ String id = delivery.substring(0, delivery.indexOf('#'));
+ if (delivery.equals(held.get(id)) && !excused(delivery)) {
+ lostWhileHeld.add("the lock of " + delivery + " expired at t=" +
dst.now() + " while it is processed");
+ }
+ }
+
+ private void settled(String outcome) {
+ int space = outcome.indexOf(' ');
+ String delivery = outcome.substring(0, space);
+ String result = outcome.substring(space + 1);
+ if (forcedSettlements.remove(delivery) && !"closed".equals(result)) {
+ violations.add(delivery + " in flight at a forced stop was settled
with outcome " + result
+ + " (only through the closed client is expected)");
+ }
+ // whether a lost lock is excused is decided when the settlement finds
it lost
+ outcomes.put(delivery, "lockLost".equals(result) && excused(delivery)
? "lockLost (excused)" : result);
+ }
+
+ /** A completed exchange must complete its message, unless the settlement
failed for an excused reason. */
+ private void checkCompletions() {
+ for (String delivery : completed.stream().sorted().toList()) {
+ String outcome = outcomes.get(delivery);
+ boolean ok = "COMPLETED".equals(outcome) || "fault".equals(outcome)
+ || "lockLost (excused)".equals(outcome)
+ // a settlement after the consumer's own close() is one of
an exchange that was in flight when the
+ // stop timed out (by design); a graceful stop keeps the
client open, so "closedByStop" is not excused
+ || "closed".equals(outcome);
+ if (!ok) {
+ notCompleted.add(delivery + " completed but its settlement was
" + outcome);
+ }
+ }
+ }
+
+ private void settling(String settlement) {
+ String delivery = settlement.substring(0, settlement.indexOf(' '));
+ boolean complete = settlement.endsWith("COMPLETED");
+ routedOrSettled.add(delivery);
+ if (!complete && !failed.contains(delivery) &&
forcedAtStop.contains(delivery)) {
+ forcedSettlements.add(delivery);
+ } else if (complete && !completed.contains(delivery) || !complete &&
!failed.contains(delivery)) {
+ wrongSettlements.add(settlement + " at t=" + dst.now());
+ }
+ }
+
+ private void finisher(DefaultCamelContext context, DstExecutor operator,
long deadline) {
+ if (deadline == -1 && (sendsLeft > 0 || opsLeft > 0)) {
+ operator.schedule("finisher", 500, () -> finisher(context,
operator, -1));
+ return;
+ }
+ if (deadline == -1) {
+ long slow = dst.between(0, 10_000);
+ dst.trace("world: operations done, faults for another " + slow + "
ms");
+ operator.schedule("finisher", slow, () -> finisher(context,
operator, -2));
+ return;
+ }
+ if (deadline == -2) {
+ faults = false;
+ bus.faults = false;
+ dst.trace("world: faults cleared");
+ long d = dst.now() + 60 * 60_000L;
+ operator.schedule("finisher", 500, () -> finisher(context,
operator, d));
+ return;
+ }
+ ServiceStatus status =
context.getRouteController().getRouteStatus(ROUTE);
+ boolean settled = bus.settled();
+ if ((!settled || inFlight > 0 || status != ServiceStatus.Started) &&
dst.now() < deadline) {
+ operator.schedule("finisher", 500, () -> finisher(context,
operator, deadline));
+ return;
+ }
+ finalStatus = String.valueOf(status);
+ finalSettled = settled;
+ if (inFlight > 0) {
+ violations.add(inFlight + " exchanges still in flight at the
deadline");
+ }
+ checkCompletions();
+ finished = true;
+ }
+
+ private void startRoute(DefaultCamelContext context) {
+ try {
+ context.getRouteController().startRoute(ROUTE);
+ ledger.record("routeStarted", "r", ROUTE);
+ } catch (Exception e) {
+ violations.add("start failed: " + e);
+ }
+ }
+
+ private void planSends() {
+ DstExecutor sender = dst.executor("sender", 1);
+ long t = dst.between(0, 500);
+ int n = 0;
+ int bursts = (int) dst.between(1, 4);
+ for (int b = 0; b < bursts; b++) {
+ int size = (int) dst.between(1, 12);
+ for (int i = 0; i < size; i++) {
+ String id = "m" + n++;
+ sendsLeft++;
+ sender.schedule("send " + id, t, () -> {
+ bus.send(id);
+ sendsLeft--;
+ });
+ t += dst.between(0, 100);
+ }
+ t += dst.between(0, 30_000);
+ }
+ }
+
+ /**
+ * Stops the route while messages may be processed (the consumer stops
receiving, the shutdown strategy waits for
+ * the exchanges in flight, whose asynchronous answers are tasks of their
own), then starts it again.
+ */
+ private void planOperations(DefaultCamelContext context, DstExecutor
operator) {
+ int ops = (int) dst.between(0, 3);
+ long t = dst.between(500, 5000);
+ for (int i = 0; i < ops; i++) {
+ opsLeft++;
+ operator.schedule("stop route", t, () -> {
+ if (!stopping &&
context.getRouteController().getRouteStatus(ROUTE) == ServiceStatus.Started) {
+ heldAtStop.addAll(inProgress);
+ // only the start after this stop ends the stopping state
(not the start of a skipped stop)
+ gracefulStop(context, operator, () -> restart(context,
operator, true));
+ } else {
+ restart(context, operator, false);
+ }
+ });
+ t += dst.between(5000, 60_000);
+ }
+ }
+
+ private void restart(DefaultCamelContext context, DstExecutor operator,
boolean afterStop) {
+ operator.schedule("start route", dst.between(100, 10_000), () -> {
+ if (afterStop) {
+ stopping = false;
+ }
+ startRoute(context);
+ opsLeft--;
+ });
+ }
+
+ /**
+ * The graceful stop of the route by DefaultShutdownStrategy as it reaches
the consumer, with the wait of the
+ * strategy as tasks of the operator: deferShutdown(shutdownRunningTask of
the route), then a check every second
+ * (the loop delay of the strategy) until the route has no exchange in
flight and the consumer none pending, for up
+ * to the timeout of the strategy; then stopRoute, which finds nothing to
wait for and stops the consumer. When the
+ * timeout passes first, the strategy forces the consumer to stop
(shutdownNowOnTimeout, the default), which this
+ * does with ServiceHelper.stopService before stopRoute. Here stopRoute is
a second graceful shutdown, which the
+ * real strategy does not make (it stops the route services at once): it
waits for the exchanges still in flight (on
+ * the one stack its own timeout is noticed only when that wait returns),
which delays the route stop and the
+ * restart. Nothing is delivered meanwhile, as the stopped consumer has
closed its processor client, and the
+ * settlements of those exchanges find the client closed as they would
after the real forced stop. Settlements after
+ * a forced stop are "closed" (by design); a settlement inside the
graceful wait must find the client open.
+ * <p>
+ * Why not stopRoute at once: the consumer may hold a message delivered
during the graceful stop in processMessage
+ * until it stops. On the one stack that call would run above the wait of
the strategy, and the stop below it could
+ * not go on until the call returned.
+ * <p>
+ * This and {@link #awaitInflightThenStop} mirror {@code
DefaultShutdownStrategy.ShutdownTask} (deferShutdown, the
+ * wait for the inflight exchanges and the pending exchanges of the
consumer, the forced stop on timeout): check
+ * them against it when the strategy changes.
+ */
+ private void gracefulStop(DefaultCamelContext context, DstExecutor
operator, Runnable then) {
+ Route route = context.getRoute(ROUTE);
+ ShutdownAware consumer = (ShutdownAware) route.getConsumer();
+ stopping = true;
+ consumer.deferShutdown(route.getShutdownRunningTask());
+ ledger.record("routeStopping", "r", ROUTE);
+ long timeout =
context.getShutdownStrategy().getTimeUnit().toMillis(context.getShutdownStrategy().getTimeout());
+ awaitInflightThenStop(context, consumer, operator, dst.now() +
timeout, then);
+ }
+
+ private void awaitInflightThenStop(
+ DefaultCamelContext context, ShutdownAware consumer, DstExecutor
operator, long deadline, Runnable then) {
+ int pending = context.getInflightRepository().size(ROUTE) +
Math.max(0, consumer.getPendingExchangesSize());
+ if (pending > 0 && dst.now() < deadline) {
+ operator.schedule("graceful stop: wait for the exchanges in
flight", 1000,
+ () -> awaitInflightThenStop(context, consumer, operator,
deadline, then));
+ return;
+ }
+ try {
+ if (pending > 0) {
+ dst.trace("world: graceful stop timed out with " + pending + "
exchanges, the consumer is stopped now");
+ forcedAtStop.addAll(inProgress);
+ ServiceHelper.stopService(consumer);
+ }
+ context.getRouteController().stopRoute(ROUTE);
+ ledger.record("routeStopped", "r", ROUTE);
+ } catch (Exception e) {
+ violations.add("stop failed: " + e);
+ }
+ then.run();
+ }
+}
diff --git
a/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
b/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
index 91c0226629df..d3f9d7de7b11 100644
---
a/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
+++
b/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
@@ -14,9 +14,16 @@
## See the License for the specific language governing permissions and
## limitations under the License.
## ---------------------------------------------------------------------------
-appender.file.type = File
+# capped: the DST test runs the consumer for a thousand seeds
+appender.file.type = RollingFile
appender.file.name = file
appender.file.fileName = target/camel-servicebus-test.log
+appender.file.filePattern = target/camel-servicebus-test.%i.log
+appender.file.policies.type = Policies
+appender.file.policies.size.type = SizeBasedTriggeringPolicy
+appender.file.policies.size.size = 50MB
+appender.file.strategy.type = DefaultRolloverStrategy
+appender.file.strategy.max = 1
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d [%-15.15t] %-5p %-30.30c{1} - %m%n
appender.out.type = Console
diff --git a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
index 43249aaa404e..c834da4f739a 100644
--- a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
+++ b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
@@ -91,6 +91,16 @@ a|* checkpointing by checkpointBatchSize and
checkpointBatchTimeout
* an asynchronous downstream step
|CAMEL-25683, CAMEL-25738, CAMEL-25781
+|camel-azure-servicebus
+|`ServiceBusWorld` (`ServiceBusConsumerDstTest`)
+|FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count,
dead-letter queue) behind the processor and receiver clients of
ServiceBusClientFactory
+a|* the lock renewer of the consumer (PEEK_LOCK, maxAutoLockRenewDuration)
with an asynchronous route
+* processing that takes longer than the lock duration
+* complete, abandon and dead-letter, also after the lock was lost
+* maxConcurrentCalls of one to three (with the asynchronous route it bounds
the processMessage calls)
+* route stop and start while messages are processed
+|CAMEL-25701, CAMEL-26956, CAMEL-26969, CAMEL-26970, CAMEL-27066
+
|camel-google-pubsub
|`PubsubWorld` (`PubsubConsumerDstTest`)
|FakePubsub: a subscription (Pull, Acknowledge, ModifyAckDeadline, ack
deadlines, dead letter policy) behind the SubscriberStub of synchronousPull