This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git


The following commit(s) were added to refs/heads/main by this push:
     new a5bc56a0e01b CAMEL-26974: camel-zookeeper-master - a DST fake world 
for the master consumer and the route policy (#30137)
a5bc56a0e01b is described below

commit a5bc56a0e01bef4e710d657b2817751ecc107a3f
Author: allthingssecurity <[email protected]>
AuthorDate: Tue Nov 17 14:28:56 2026 +0530

    CAMEL-26974: camel-zookeeper-master - a DST fake world for the master 
consumer and the route policy (#30137)
    
    … consumer and the route policy
    
    Adds a camel-test-dst world for the camel-zookeeper-master consumer and
    MasterRoutePolicy, in the dst package of its tests: ZooKeeperMasterDstTest
    (1000 seeds), ZooKeeperMasterWorld and FakeZooKeeper. The real master
    consumer or route policy (the seed picks one) and the real ZooKeeperGroup
    run against a fake ZooKeeper ensemble behind the CuratorFramework shared as
    the curator bean, written from the ZooKeeper Programmer's Guide, the leader
    election recipe and the Curator documentation: sessions, ephemeral
    sequential znodes, one-time watches held while the client is disconnected,
    session expiry, the connection states SUSPENDED, LOST and RECONNECTED, the
    wait for the connection and the retry of an operation (also of a write
    whose answer was lost), and guaranteed deletes. The other nodes of the
    group are sessions of the fake that join, leave and die. The consumer that
    only the master runs is a test consumer whose start takes a while and
    sometimes fails.
    
    The groups are made by a ManagedGroupFactory bean, as
    DefaultManagedGroupFactory makes them, to stand in for the thread
    interrupt of the executor's shutdownNow that the harness does not do: the
    fake fails the calls of the thread of a closed group, and the world offers
    the closed group a refresh so that its main loop ends.
    
    Invariants: this node runs its consumer only while it is the master; once
    the faults cleared the master runs its consumer; a stopped consumer or a
    removed route leaves no listener on the shared CuratorFramework, and no
    znode of this node is left in the group after the final stop; the shared
    CuratorFramework and group factory are not closed; the client is used as
    ZooKeeper and Curator allow.
    
    Not explored: several master endpoints or policies of one node in a group,
    suspend and resume, the CuratorFramework the component creates itself, a
    stop while the thread of the group waits in a call of ZooKeeper other than
    a create. Left out of the default set while they are looked at separately,
    each added back by a system property: a route stopped while the client is
    disconnected or while the group waits for the answer of a create, and a
    route of the policy started from outside when it is started already.
    
    Self-check: the world fails on the code before CAMEL-25315, CAMEL-26540,
    CAMEL-26802 and CAMEL-26887, and with the change of CAMEL-26870 taken out;
    main passes 5000 of 5000 seeds.
    
    The module's pom loads the byte code agent of camel-test-dst, the test log
    is capped, and the catalog lists the world.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
 .../camel/catalog/components/zookeeper-master.json |    3 +
 .../org/apache/camel/catalog/docs/test-dst.adoc    |   19 +
 .../camel-test-dst/src/main/docs/test-dst.adoc     |   19 +
 components/camel-zookeeper-master/pom.xml          |   27 +-
 .../zookeepermaster/zookeeper-master.json          |    3 +
 .../zookeepermaster/dst/FakeZooKeeper.java         | 1000 ++++++++++++++++++++
 .../dst/ZooKeeperMasterDstTest.java                |   37 +
 .../zookeepermaster/dst/ZooKeeperMasterWorld.java  |  734 ++++++++++++++
 .../src/test/resources/log4j2.properties           |    9 +-
 9 files changed, 1849 insertions(+), 2 deletions(-)

diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/zookeeper-master.json
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/zookeeper-master.json
index be5022259750..8d3bfd7b49a2 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/zookeeper-master.json
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/zookeeper-master.json
@@ -42,5 +42,8 @@
     "bridgeErrorHandler": { "index": 2, "kind": "parameter", "displayName": 
"Bridge Error Handler", "group": "consumer (advanced)", "label": 
"consumer,advanced", "required": false, "type": "boolean", "javaType": 
"boolean", "deprecated": false, "autowired": false, "secret": false, 
"defaultValue": false, "description": "Allows for bridging the consumer to the 
Camel routing Error Handler, which mean any exceptions (if possible) occurred 
while the Camel consumer is trying to pickup incoming  [...]
     "exceptionHandler": { "index": 3, "kind": "parameter", "displayName": 
"Exception Handler", "group": "consumer (advanced)", "label": 
"consumer,advanced", "required": false, "type": "object", "javaType": 
"org.apache.camel.spi.ExceptionHandler", "optionalPrefix": "consumer.", 
"deprecated": false, "autowired": false, "secret": false, "description": "To 
let the consumer use a custom ExceptionHandler. Notice if the option 
bridgeErrorHandler is enabled then this option is not in use. By def [...]
     "exchangePattern": { "index": 4, "kind": "parameter", "displayName": 
"Exchange Pattern", "group": "consumer (advanced)", "label": 
"consumer,advanced", "required": false, "type": "enum", "javaType": 
"org.apache.camel.ExchangePattern", "enum": [ "InOnly", "InOut" ], 
"deprecated": false, "autowired": false, "secret": false, "description": "Sets 
the exchange pattern when the consumer creates an exchange." }
+  },
+  "dstWorlds": {
+    "ZooKeeperMasterWorld": { "javaType": 
"org.apache.camel.component.zookeepermaster.dst.ZooKeeperMasterWorld", "fake": 
"FakeZooKeeper: a ZooKeeper ensemble (sessions, ephemeral sequential znodes, 
one-time watches) with the other nodes of the group as sessions, behind a 
CuratorFramework shared as the curator bean", "covers": [ "the master consumer 
(zookeeper-master endpoint) and the MasterRoutePolicy, on the real 
ZooKeeperGroup", "connection losses shorter and longer than the session ti [...]
   }
 }
diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
index c834da4f739a..d6ee0ac24ad9 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
@@ -171,6 +171,16 @@ a|* the polling consumer with maxResults and naturalOrder
 * the pages of conversations.history (oldest and cursor)
 * route suspend and resume while a batch is routed
 |CAMEL-25684, CAMEL-25738
+
+|camel-zookeeper-master
+|`ZooKeeperMasterWorld` (`ZooKeeperMasterDstTest`)
+|FakeZooKeeper: a ZooKeeper ensemble (sessions, ephemeral sequential znodes, 
one-time watches) with the other nodes of the group as sessions, behind a 
CuratorFramework shared as the curator bean
+a|* the master consumer (zookeeper-master endpoint) and the MasterRoutePolicy, 
on the real ZooKeeperGroup
+* connection losses shorter and longer than the session timeout (SUSPENDED, 
RECONNECTED, LOST)
+* other nodes that join, leave and die
+* a start of the consumer that fails and its retry; with the master consumer, 
a start that takes long
+* route stop and start, and a route of the policy removed and added again
+|CAMEL-25315, CAMEL-26540, CAMEL-26802, CAMEL-26870, CAMEL-26887, CAMEL-27037, 
CAMEL-27044
 |===
 // dst worlds: END
 
@@ -641,6 +651,15 @@ properties, each adding back a case that the default set 
leaves out: `dst.paho.s
 client whose connect hangs while the callback thread is still in 
messageArrived, CAMEL-27043) and
 `dst.paho.stopWhileConnecting` (`true` lets the operator stop the route while 
the client connects, CAMEL-27041).
 
+For camel-zookeeper-master (`-pl components/camel-zookeeper-master`, 
`ZooKeeperMasterDstTest`) the past fixes are
+`MasterConsumer` of e670025a49a4^ (CAMEL-25315), `MasterRoutePolicy` of 
706beec6d5ab^ (CAMEL-26540), 136ffd929263
+(CAMEL-26802) and 28f3170dcc6a (CAMEL-26887) reverted, and the 
`unregisterAsListener()` of 5ecab86a58ba (CAMEL-26870)
+taken out by hand. Its world has three properties, each adding back a case 
that the default set leaves out:
+`dst.zookeeper.stopWhileSuspended` (a route stopped while the client is 
disconnected, CAMEL-27037),
+`dst.zookeeper.stopDuringCreate` (a route stopped while the group waits for 
the answer of a create, or a create made
+after the close interrupted the thread of the group, CAMEL-27037) and 
`dst.zookeeper.startStartedRoute` (a started
+route of the policy started again, CAMEL-27044).
+
 == Limits
 
 * Interleaving is per task: the harness does not preempt a task in the middle 
of a method.
diff --git a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc 
b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
index c834da4f739a..d6ee0ac24ad9 100644
--- a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
+++ b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
@@ -171,6 +171,16 @@ a|* the polling consumer with maxResults and naturalOrder
 * the pages of conversations.history (oldest and cursor)
 * route suspend and resume while a batch is routed
 |CAMEL-25684, CAMEL-25738
+
+|camel-zookeeper-master
+|`ZooKeeperMasterWorld` (`ZooKeeperMasterDstTest`)
+|FakeZooKeeper: a ZooKeeper ensemble (sessions, ephemeral sequential znodes, 
one-time watches) with the other nodes of the group as sessions, behind a 
CuratorFramework shared as the curator bean
+a|* the master consumer (zookeeper-master endpoint) and the MasterRoutePolicy, 
on the real ZooKeeperGroup
+* connection losses shorter and longer than the session timeout (SUSPENDED, 
RECONNECTED, LOST)
+* other nodes that join, leave and die
+* a start of the consumer that fails and its retry; with the master consumer, 
a start that takes long
+* route stop and start, and a route of the policy removed and added again
+|CAMEL-25315, CAMEL-26540, CAMEL-26802, CAMEL-26870, CAMEL-26887, CAMEL-27037, 
CAMEL-27044
 |===
 // dst worlds: END
 
@@ -641,6 +651,15 @@ properties, each adding back a case that the default set 
leaves out: `dst.paho.s
 client whose connect hangs while the callback thread is still in 
messageArrived, CAMEL-27043) and
 `dst.paho.stopWhileConnecting` (`true` lets the operator stop the route while 
the client connects, CAMEL-27041).
 
+For camel-zookeeper-master (`-pl components/camel-zookeeper-master`, 
`ZooKeeperMasterDstTest`) the past fixes are
+`MasterConsumer` of e670025a49a4^ (CAMEL-25315), `MasterRoutePolicy` of 
706beec6d5ab^ (CAMEL-26540), 136ffd929263
+(CAMEL-26802) and 28f3170dcc6a (CAMEL-26887) reverted, and the 
`unregisterAsListener()` of 5ecab86a58ba (CAMEL-26870)
+taken out by hand. Its world has three properties, each adding back a case 
that the default set leaves out:
+`dst.zookeeper.stopWhileSuspended` (a route stopped while the client is 
disconnected, CAMEL-27037),
+`dst.zookeeper.stopDuringCreate` (a route stopped while the group waits for 
the answer of a create, or a create made
+after the close interrupted the thread of the group, CAMEL-27037) and 
`dst.zookeeper.startStartedRoute` (a started
+route of the policy started again, CAMEL-27044).
+
 == Limits
 
 * Interleaving is per task: the harness does not preempt a task in the middle 
of a method.
diff --git a/components/camel-zookeeper-master/pom.xml 
b/components/camel-zookeeper-master/pom.xml
index 6ef9eb82e874..c6b1cb69bb4a 100644
--- a/components/camel-zookeeper-master/pom.xml
+++ b/components/camel-zookeeper-master/pom.xml
@@ -36,7 +36,8 @@
         <!-- Zookeeper container is not available on this platform -->
         <skipITs.aarch64>true</skipITs.aarch64>
         <skipITs.s390x>true</skipITs.s390x>
-        <camel.surefire.fork.additional-vmargs>-XX:+ExitOnOutOfMemoryError 
-Xmx2G</camel.surefire.fork.additional-vmargs>
+        <!-- the DST tests load the byte code agent of camel-test-dst when the 
JVM starts (see test-dst.adoc) -->
+        <camel.surefire.fork.additional-vmargs>-XX:+ExitOnOutOfMemoryError 
-Xmx2G 
"-javaagent:${net.bytebuddy:byte-buddy-agent:jar}"</camel.surefire.fork.additional-vmargs>
     </properties>
 
     <dependencies>
@@ -132,9 +133,33 @@
             <version>${project.version}</version>
             <scope>test</scope>
         </dependency>
+        <!-- deterministic simulation test of the master consumer and the 
route policy (dst package) -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test-dst</artifactId>
+            <scope>test</scope>
+        </dependency>
 
     </dependencies>
 
+    <build>
+        <plugins>
+            <plugin>
+                <!-- sets the property net.bytebuddy:byte-buddy-agent:jar to 
the path of the agent of the DST tests -->
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-dependency-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>dst-agent-path</id>
+                        <goals>
+                            <goal>properties</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+
     <profiles>
         <profile>
             <id>full</id>
diff --git 
a/components/camel-zookeeper-master/src/generated/resources/META-INF/org/apache/camel/component/zookeepermaster/zookeeper-master.json
 
b/components/camel-zookeeper-master/src/generated/resources/META-INF/org/apache/camel/component/zookeepermaster/zookeeper-master.json
index be5022259750..8d3bfd7b49a2 100644
--- 
a/components/camel-zookeeper-master/src/generated/resources/META-INF/org/apache/camel/component/zookeepermaster/zookeeper-master.json
+++ 
b/components/camel-zookeeper-master/src/generated/resources/META-INF/org/apache/camel/component/zookeepermaster/zookeeper-master.json
@@ -42,5 +42,8 @@
     "bridgeErrorHandler": { "index": 2, "kind": "parameter", "displayName": 
"Bridge Error Handler", "group": "consumer (advanced)", "label": 
"consumer,advanced", "required": false, "type": "boolean", "javaType": 
"boolean", "deprecated": false, "autowired": false, "secret": false, 
"defaultValue": false, "description": "Allows for bridging the consumer to the 
Camel routing Error Handler, which mean any exceptions (if possible) occurred 
while the Camel consumer is trying to pickup incoming  [...]
     "exceptionHandler": { "index": 3, "kind": "parameter", "displayName": 
"Exception Handler", "group": "consumer (advanced)", "label": 
"consumer,advanced", "required": false, "type": "object", "javaType": 
"org.apache.camel.spi.ExceptionHandler", "optionalPrefix": "consumer.", 
"deprecated": false, "autowired": false, "secret": false, "description": "To 
let the consumer use a custom ExceptionHandler. Notice if the option 
bridgeErrorHandler is enabled then this option is not in use. By def [...]
     "exchangePattern": { "index": 4, "kind": "parameter", "displayName": 
"Exchange Pattern", "group": "consumer (advanced)", "label": 
"consumer,advanced", "required": false, "type": "enum", "javaType": 
"org.apache.camel.ExchangePattern", "enum": [ "InOnly", "InOut" ], 
"deprecated": false, "autowired": false, "secret": false, "description": "Sets 
the exchange pattern when the consumer creates an exchange." }
+  },
+  "dstWorlds": {
+    "ZooKeeperMasterWorld": { "javaType": 
"org.apache.camel.component.zookeepermaster.dst.ZooKeeperMasterWorld", "fake": 
"FakeZooKeeper: a ZooKeeper ensemble (sessions, ephemeral sequential znodes, 
one-time watches) with the other nodes of the group as sessions, behind a 
CuratorFramework shared as the curator bean", "covers": [ "the master consumer 
(zookeeper-master endpoint) and the MasterRoutePolicy, on the real 
ZooKeeperGroup", "connection losses shorter and longer than the session ti [...]
   }
 }
diff --git 
a/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/FakeZooKeeper.java
 
b/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/FakeZooKeeper.java
new file mode 100644
index 000000000000..a19e39a836d6
--- /dev/null
+++ 
b/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/FakeZooKeeper.java
@@ -0,0 +1,1000 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.zookeepermaster.dst;
+
+import java.lang.reflect.InvocationHandler;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Proxy;
+import java.lang.reflect.Type;
+import java.lang.reflect.TypeVariable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.Callable;
+import java.util.concurrent.Executor;
+
+import org.apache.camel.component.zookeepermaster.CamelNodeState;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.curator.CuratorZookeeperClient;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.listen.Listenable;
+import org.apache.curator.framework.state.ConnectionState;
+import org.apache.curator.framework.state.ConnectionStateListener;
+import org.apache.curator.retry.RetryOneTime;
+import org.apache.curator.utils.EnsurePath;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.WatchedEvent;
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.data.Stat;
+import tools.jackson.databind.DeserializationFeature;
+import tools.jackson.databind.ObjectMapper;
+import tools.jackson.databind.json.JsonMapper;
+
+/**
+ * A fake ZooKeeper ensemble, the {@link CuratorFramework} of this node on it, 
and the other nodes of the group (peers)
+ * as sessions of their own. Written from the ZooKeeper Programmer's Guide 
(3.9) and the Curator documentation (5.x);
+ * the rules it follows are given (in the words of the documents where 
possible, not verbatim) where they are
+ * implemented and marked:
+ * <ul>
+ * <li>[Z] ZooKeeper Programmer's Guide, "ZooKeeper Sessions", "ZooKeeper 
Watches", "Ephemeral Nodes", "Sequence
+ * Nodes";</li>
+ * <li>[R] ZooKeeper Recipes, "Leader Election";</li>
+ * <li>[C] Curator, "Error Handling" (connection states), the Javadoc of 
{@code ConnectionState}, {@code RetryLoop} and
+ * {@code DeleteBuilder.guaranteed()};</li>
+ * <li>[A] an assumption, where the documents leave the behaviour open; the 
seed picks between the allowed choices.</li>
+ * </ul>
+ * <p>
+ * Modelled: persistent, ephemeral and ephemeral sequential znodes; one-time 
child and data watches, held while the
+ * client is disconnected and delivered after it reconnected to the same 
session; a session that expires when the client
+ * cannot reach the ensemble for longer than the session timeout, with its 
ephemeral znodes; the connection states
+ * SUSPENDED, LOST and RECONNECTED, delivered on a connection state thread; 
watch events on an event thread; the wait of
+ * an operation for the connection (up to the connection timeout) and its 
foreground retry after a connection loss (the
+ * retry policy of the client); a create that the ensemble applied but whose 
answer was lost (a retry then makes a
+ * second znode); guaranteed deletes, done again in the background after a 
reconnect, and a guaranteed delete in the
+ * background (without a callback), done once the client is connected.
+ * <p>
+ * Not modelled: ACLs, containers and TTL znodes, background (asynchronous) 
operations other than a guaranteed delete
+ * without a callback, transactions, the state events of type None that 
ZooKeeper gives every watcher (the watchers of
+ * the group ignore them), read-only mode, the order of the children in a 
getChildren answer (sorted here), and
+ * SessionExpiredException of an operation, which Curator retries (the client 
here has its new session as soon as the
+ * connection is back, so no operation sees the expired one).
+ * <p>
+ * The client notices a connection loss after a seeded delay below two thirds 
of the session timeout (as the ZooKeeper
+ * client does with its read timeout), or at once when one of its calls fails 
on it (a reset connection).
+ * <p>
+ * Timing (the one-stack rule of the harness): writes take effect at the start 
of their latency and reads at the end;
+ * watch and connection state events are tasks of their own threads and never 
block.
+ */
+final class FakeZooKeeper {
+
+    private static final ObjectMapper MAPPER = 
JsonMapper.builderWithJackson2Defaults()
+            
.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES).build();
+
+    private final Dst dst;
+    final List<String> violations = new ArrayList<>();
+    boolean faults = true;
+    /** The negotiated session timeout. */
+    long sessionTimeout = 6000;
+    /** Foreground retries of an operation that failed with a connection loss, 
and the sleep between two. */
+    int retries = 1;
+    long retrySleep = 1000;
+    /** [C] CuratorFrameworkFactory: the connection timeout defaults to 15 
seconds. */
+    long connectionTimeout = 15_000;
+    double slowRate;
+    double responseLostRate;
+
+    private final DstExecutor eventThread;
+    private final DstExecutor stateThread;
+    private final Map<String, Znode> tree = new TreeMap<>();
+    private long zxid;
+    private long nextSession = 1;
+
+    // the session of this node's client, and what the client believes about 
its connection
+    private long session;
+    private boolean network = true;
+    private boolean clientConnected = true;
+    private boolean sessionExpired;
+    private boolean lostPosted;
+    private int outage;
+    private final List<ConnectionStateListener> listeners = new ArrayList<>();
+    private final Map<String, Set<Watcher>> childWatches = new HashMap<>();
+    private final Map<String, Set<Watcher>> dataWatches = new HashMap<>();
+    private final List<Runnable> held = new ArrayList<>();
+    private final Set<String> failedDeletes = new LinkedHashSet<>();
+    private boolean closed;
+    private final CuratorFramework client;
+    private final FakeZookeeperClient zookeeperClient;
+
+    // the other nodes of the group: container -> session
+    private final Map<String, Long> peers = new TreeMap<>();
+
+    private static final class Znode {
+        byte[] data;
+        int version;
+        int cversion;
+        long owner;
+        long czxid;
+        long mzxid;
+        long ctime;
+        long mtime;
+    }
+
+    FakeZooKeeper(Dst dst) {
+        this.dst = dst;
+        this.eventThread = dst.executor("zookeeper-event-thread", 1);
+        this.stateThread = dst.executor("curator-connection-state", 1);
+        this.session = nextSession++;
+        tree.put("/", new Znode());
+        this.client = (CuratorFramework) 
Proxy.newProxyInstance(CuratorFramework.class.getClassLoader(),
+                new Class<?>[] { CuratorFramework.class }, this::invokeClient);
+        this.zookeeperClient = new FakeZookeeperClient();
+    }
+
+    /** The CuratorFramework of this node, as an application shares it (bound 
as the curator bean). */
+    CuratorFramework client() {
+        return client;
+    }
+
+    boolean isClosed() {
+        return closed;
+    }
+
+    int listenerCount() {
+        return listeners.size();
+    }
+
+    boolean isNetworkUp() {
+        return network;
+    }
+
+    boolean isClientConnected() {
+        return clientConnected;
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the operator: connection losses and peers
+
+    /**
+     * The client loses its connection to the ensemble for {@code millis}. [Z] 
The client will see a disconnected event
+     * when it is disconnected from the ensemble; [Z] Expirations happens when 
the cluster does not hear from the client
+     * within the specified session timeout period; [C] SUSPENDED: There has 
been a loss of connection; LOST: The
+     * connection is confirmed to be lost ... Curator will set the LOST state 
when it believes that the ZooKeeper
+     * session has expired (after the negotiated session timeout without a 
reconnect, Curator injects the expiry itself,
+     * so the old session is not resumed after that).
+     */
+    void disconnect(long millis) {
+        if (!network) {
+            return;
+        }
+        network = false;
+        int k = ++outage;
+        dst.trace("zk: connection lost for " + millis + " ms (session " + 
session + ")");
+        // [A] the client notices the loss within two thirds of the session 
timeout (the read timeout of the client), or
+        // at once when one of its calls fails on it
+        long detect = dst.between(10, Math.max(10, sessionTimeout * 2 / 3));
+        stateThread.schedule("detect connection loss", detect, () -> 
notice(k));
+        // the ensemble expires the session when it does not hear from the 
client within the session timeout
+        eventThread.schedule("session expiry on the ensemble", sessionTimeout, 
() -> {
+            if (outage == k && !network) {
+                expireLocalSession();
+            }
+        });
+        dst.executor("network", 1).schedule("connection back", millis, () -> 
reconnect(k));
+    }
+
+    /**
+     * The client notices the connection loss: [Z] a ConnectionLoss comes with 
the Disconnected event of the client ([C]
+     * SUSPENDED).
+     */
+    private void notice(int k) {
+        if (outage == k && !network && clientConnected) {
+            clientConnected = false;
+            post(ConnectionState.SUSPENDED);
+            stateThread.schedule("session timeout of the client", 
sessionTimeout, () -> {
+                if (outage == k && !network && !lostPosted) {
+                    lostPosted = true;
+                    post(ConnectionState.LOST);
+                }
+            });
+        }
+    }
+
+    private void reconnect(int k) {
+        if (outage != k || network) {
+            return;
+        }
+        network = true;
+        if (sessionExpired || lostPosted) {
+            // [Z] the client will be told that the session has expired; [C] a 
new session is made, RECONNECTED
+            if (!sessionExpired) {
+                expireLocalSession();
+            }
+            if (!lostPosted) {
+                clientConnected = false;
+                post(ConnectionState.LOST);
+            }
+            session = nextSession++;
+            sessionExpired = false;
+            lostPosted = false;
+            held.clear();
+            failedDeletes.clear();
+            clientConnected = true;
+            dst.trace("zk: reconnected with a new session " + session);
+            post(ConnectionState.RECONNECTED);
+            return;
+        }
+        dst.trace("zk: reconnected to session " + session);
+        if (!clientConnected) {
+            clientConnected = true;
+            post(ConnectionState.RECONNECTED);
+        }
+        // [Z] When a client reconnects, any previously registered watches 
will be reregistered and triggered if
+        // needed"
+        List<Runnable> events = new ArrayList<>(held);
+        held.clear();
+        events.forEach(Runnable::run);
+        // [C] guaranteed(): Curator will record failed node deletions and 
attempt to delete them in the background
+        // until successful"
+        for (String path : new ArrayList<>(failedDeletes)) {
+            retryDelete(path);
+        }
+    }
+
+    private void retryDelete(String path) {
+        long s = session;
+        eventThread.schedule("guaranteed delete of " + path, latency(), () -> {
+            if (network && session == s && failedDeletes.remove(path) && 
tree.containsKey(path)) {
+                dst.trace("zk: guaranteed delete of " + path + " done in the 
background");
+                remove(path);
+            }
+        });
+    }
+
+    private void expireLocalSession() {
+        long expired = session;
+        sessionExpired = true;
+        dst.trace("zk: session " + expired + " expired");
+        // [Z] the watches of the session are lost; ephemeral nodes ... are 
removed when the session ends
+        childWatches.clear();
+        dataWatches.clear();
+        held.clear();
+        removeEphemeralsOf(expired);
+    }
+
+    private void post(ConnectionState state) {
+        dst.trace("zk: connection state " + state);
+        stateThread.execute(() -> {
+            // [C] ConnectionStateManager calls the listeners registered when 
it dispatches the state
+            List<ConnectionStateListener> snapshot = new 
ArrayList<>(listeners);
+            dst.trace("zk: connection state " + state + " to " + 
snapshot.size() + " listeners");
+            for (ConnectionStateListener l : snapshot) {
+                try {
+                    l.stateChanged(client, state);
+                } catch (RuntimeException e) {
+                    // Curator logs it
+                    dst.trace("zk: listener failed: " + e);
+                }
+            }
+        });
+    }
+
+    /** A peer joins the group: its session makes an ephemeral sequential 
znode, as ZooKeeperGroup does. */
+    void peerJoins(String groupPath, String container) {
+        if (peers.containsKey(container)) {
+            return;
+        }
+        long s = nextSession++;
+        peers.put(container, s);
+        ensureParents(groupPath + "/0");
+        CamelNodeState state = new CamelNodeState("dst", container);
+        String path = createNode(groupPath + "/0", encode(state), 
CreateMode.EPHEMERAL_SEQUENTIAL, s);
+        dst.trace("zk: peer " + container + " joins as " + path);
+    }
+
+    /** A peer leaves the group: it closes its session, so its znode is 
removed at once. */
+    void peerLeaves(String container) {
+        Long s = peers.remove(container);
+        if (s != null) {
+            dst.trace("zk: peer " + container + " leaves");
+            removeEphemeralsOf(s);
+        }
+    }
+
+    /** A peer dies: the ensemble removes its znode once its session timed 
out. */
+    void peerDies(String container) {
+        Long s = peers.remove(container);
+        if (s != null) {
+            dst.trace("zk: peer " + container + " dies");
+            deadPeerSessions.add(s);
+            eventThread.schedule("session expiry of " + container, 
sessionTimeout, () -> removeEphemeralsOf(s));
+        }
+    }
+
+    /** The guaranteed deletes that the client still has to do again in the 
background. */
+    Set<String> pendingDeletes() {
+        return failedDeletes;
+    }
+
+    Set<String> peers() {
+        return peers.keySet();
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // what the members of the group see
+
+    /** The znodes under the group path that the sessions of this node own 
(the current or an expired one). */
+    List<String> localZnodes(String groupPath) {
+        List<String> answer = new ArrayList<>();
+        for (Map.Entry<String, Znode> e : tree.entrySet()) {
+            if (isChild(groupPath, e.getKey()) && e.getValue().owner != 0 && 
!peers.containsValue(e.getValue().owner)
+                    && !isDeadPeer(e.getValue().owner)) {
+                answer.add(e.getKey());
+            }
+        }
+        return answer;
+    }
+
+    private final Set<Long> deadPeerSessions = new HashSet<>();
+
+    private boolean isDeadPeer(long owner) {
+        return deadPeerSessions.contains(owner);
+    }
+
+    /**
+     * The znode of the master, as every member of the group computes it from 
the znodes: one znode per container (the
+     * one with the greatest path), and of those the one with the lowest 
sequence number ([R] the client with the
+     * smallest sequence number is the leader). Null when the group has no 
members.
+     */
+    String master(String groupPath) {
+        Map<String, String> latest = new HashMap<>();
+        for (Map.Entry<String, Znode> e : tree.entrySet()) {
+            if (isChild(groupPath, e.getKey())) {
+                String container = container(e.getValue().data);
+                String cur = latest.get(container);
+                if (cur == null || cur.compareTo(e.getKey()) < 0) {
+                    latest.put(container, e.getKey());
+                }
+            }
+        }
+        return latest.values().stream().sorted().findFirst().orElse(null);
+    }
+
+    /** The container of the peer that owns the znode, or null when it is not 
a peer's. */
+    String peerOf(String path) {
+        Znode z = path != null ? tree.get(path) : null;
+        if (z == null) {
+            return null;
+        }
+        for (Map.Entry<String, Long> e : peers.entrySet()) {
+            if (e.getValue() == z.owner) {
+                return e.getKey();
+            }
+        }
+        return isDeadPeer(z.owner) ? "dead peer" : null;
+    }
+
+    private static boolean isChild(String parent, String path) {
+        return path.startsWith(parent + "/") && path.indexOf('/', 
parent.length() + 1) < 0;
+    }
+
+    private static String container(byte[] data) {
+        try {
+            return MAPPER.readValue(data, CamelNodeState.class).getContainer();
+        } catch (RuntimeException e) {
+            return "undecodable";
+        }
+    }
+
+    private static byte[] encode(CamelNodeState state) {
+        return 
MAPPER.writeValueAsString(state).getBytes(StandardCharsets.UTF_8);
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the ensemble
+
+    private long latency() {
+        if (faults && dst.chance(slowRate)) {
+            return dst.between(200, 2000);
+        }
+        return dst.between(1, 20);
+    }
+
+    private void ensureParents(String path) {
+        int i = path.indexOf('/', 1);
+        while (i > 0) {
+            String p = path.substring(0, i);
+            if (!tree.containsKey(p)) {
+                createNode(p, new byte[0], CreateMode.PERSISTENT, 0);
+            }
+            i = path.indexOf('/', i + 1);
+        }
+    }
+
+    private static String parentOf(String path) {
+        int i = path.lastIndexOf('/');
+        return i <= 0 ? "/" : path.substring(0, i);
+    }
+
+    private String createNode(String path, byte[] data, CreateMode mode, long 
owner) {
+        Znode parent = tree.get(parentOf(path));
+        String name = path;
+        if (mode.isSequential()) {
+            // [Z] a monotonically increasing counter will be appended to the 
name ... formatted as %010d, the
+            // counter of the parent
+            name = path + String.format("%010d", parent.cversion);
+        }
+        Znode z = new Znode();
+        z.data = data;
+        z.owner = mode.isEphemeral() ? owner : 0;
+        z.czxid = ++zxid;
+        z.mzxid = z.czxid;
+        z.ctime = dst.currentTimeMillis();
+        z.mtime = z.ctime;
+        tree.put(name, z);
+        parent.cversion++;
+        fireChildWatches(parentOf(name));
+        return name;
+    }
+
+    private void remove(String path) {
+        if (tree.remove(path) != null) {
+            // [Z] a delete does not change the cversion of the parent 
(PrepRequestProcessor and DataTree.deleteNode of
+            // ZooKeeper 3.9 set only its pzxid); only a create does, so a 
delete does not move the sequence counter
+            fireDataWatches(path, Watcher.Event.EventType.NodeDeleted);
+            fireChildWatches(parentOf(path));
+        }
+    }
+
+    private void removeEphemeralsOf(long owner) {
+        for (String path : new ArrayList<>(tree.keySet())) {
+            Znode z = tree.get(path);
+            if (z != null && z.owner == owner) {
+                remove(path);
+            }
+        }
+    }
+
+    // [Z] Watches are one time triggers; a watch object ... will only be 
triggered once for a given notification
+    private void fireChildWatches(String path) {
+        Set<Watcher> ws = childWatches.remove(path);
+        if (ws != null) {
+            deliver(ws, Watcher.Event.EventType.NodeChildrenChanged, path);
+        }
+    }
+
+    private void fireDataWatches(String path, Watcher.Event.EventType type) {
+        Set<Watcher> ws = dataWatches.remove(path);
+        if (ws != null) {
+            deliver(ws, type, path);
+        }
+        if (type == Watcher.Event.EventType.NodeDeleted) {
+            // [Z] a delete triggers the child watches of the deleted znode too
+            Set<Watcher> cs = childWatches.remove(path);
+            if (cs != null) {
+                deliver(cs, type, path);
+            }
+        }
+    }
+
+    private void deliver(Set<Watcher> watchers, Watcher.Event.EventType type, 
String path) {
+        long s = session;
+        Runnable r = () -> {
+            for (Watcher w : watchers) {
+                eventThread.execute(() -> {
+                    if (session == s && !sessionExpired) {
+                        w.process(new WatchedEvent(type, 
Watcher.Event.KeeperState.SyncConnected, path));
+                    }
+                });
+            }
+        };
+        if (network && clientConnected) {
+            r.run();
+        } else {
+            // [Z] Watches will not be received while disconnected from a 
server
+            held.add(r);
+        }
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the operations of the client
+
+    // the threads of the closed groups (executor labels), which a real close 
interrupts
+    private final Set<String> interrupted = new HashSet<>();
+    /**
+     * Whether a write that an interrupted thread makes while the client is 
connected is applied, as ZooKeeper does (see
+     * {@link #interrupt}). A group whose thread is interrupted by its close 
can then make its znode after it left the
+     * group, the case of {@code -Ddst.zookeeper.stopDuringCreate}; without it 
such a write fails before it is sent.
+     */
+    boolean writesAfterInterrupt;
+
+    /**
+     * The thread with this executor label is interrupted, as 
ExecutorService.shutdownNow does when a group closes. A
+     * ZooKeeper call that it waits in, or makes from now on, throws: the real 
one throws InterruptedException (Curator
+     * sets the interrupt flag again, so the next call throws too). The fake 
throws an IllegalStateException instead, as
+     * an InterruptedException would make the code under test set the 
interrupt flag of the one thread of the harness. A
+     * write is applied by the ensemble when it was sent before the interrupt, 
or, with {@link #writesAfterInterrupt},
+     * when it is made after the interrupt while the client is connected 
(ClientCnxn.submitRequest queues the packet
+     * before it waits); only its answer is lost. Curator queues a guaranteed 
delete again on an InterruptedException;
+     * the fake's IllegalStateException does not show that.
+     */
+    void interrupt(String executorLabel) {
+        interrupted.add(executorLabel);
+    }
+
+    private void checkInterrupted(String what, boolean applied) {
+        if (interrupted.contains(dst.currentExecutor())) {
+            dst.trace("zk: " + what + " is interrupted" + (applied ? " 
(applied, its answer is lost)" : ""));
+            throw new IllegalStateException("dst: " + what + " of an 
interrupted thread (a closed group)");
+        }
+    }
+
+    // the calls in flight per thread (executor label), and the creates among 
them
+    private final Map<String, Integer> inFlight = new HashMap<>();
+    private final Map<String, Integer> createsInFlight = new HashMap<>();
+
+    /**
+     * Whether a thread whose executor label starts with the prefix waits in a 
call of the client other than a create.
+     */
+    boolean inCallOtherThanCreate(String threadPrefix) {
+        return inFlight.entrySet().stream().anyMatch(
+                e -> e.getKey().startsWith(threadPrefix) && e.getValue() > 
createsInFlight.getOrDefault(e.getKey(), 0));
+    }
+
+    /** Whether a thread whose executor label starts with the prefix waits in 
a create. */
+    boolean inCreate(String threadPrefix) {
+        return createsInFlight.entrySet().stream().anyMatch(e -> 
e.getKey().startsWith(threadPrefix) && e.getValue() > 0);
+    }
+
+    private <T> T op(String what, boolean write, Callable<T> apply) throws 
Exception {
+        String thread = dst.currentExecutor();
+        boolean create = what.startsWith("create ");
+        inFlight.merge(thread, 1, Integer::sum);
+        if (create) {
+            createsInFlight.merge(thread, 1, Integer::sum);
+        }
+        try {
+            return call(what, write, apply);
+        } finally {
+            inFlight.merge(thread, -1, Integer::sum);
+            if (create) {
+                createsInFlight.merge(thread, -1, Integer::sum);
+            }
+        }
+    }
+
+    private <T> T call(String what, boolean write, Callable<T> apply) throws 
Exception {
+        if (writesAfterInterrupt && write && 
interrupted.contains(dst.currentExecutor()) && !closed && clientConnected
+                && network) {
+            // [Z] a write made after the interrupt still reaches the 
ensemble: ClientCnxn queues the packet before it
+            // waits, and only the wait is interrupted
+            try {
+                apply.call();
+            } catch (KeeperException e) {
+                // the answer of the ensemble is lost with the interrupt
+                dst.trace("zk: " + what + " of an interrupted thread failed: " 
+ e.code());
+            }
+            checkInterrupted(what, true);
+        }
+        checkInterrupted(what, false);
+        if (closed) {
+            violations.add(what + " on a closed client");
+            throw new IllegalStateException("Expected state [STARTED] was 
[STOPPED]");
+        }
+        if (!clientConnected) {
+            // [C] RetryLoop.callWithRetry first waits until the client is 
connected, up to the connection timeout, once
+            // before its retries
+            dst.awaitUntil(() -> clientConnected, connectionTimeout, what + " 
waits for the connection");
+            checkInterrupted(what, false);
+        }
+        int attempt = 0;
+        while (true) {
+            try {
+                return attempt(what, write, apply);
+            } catch (KeeperException.ConnectionLossException e) {
+                checkInterrupted(what, false);
+                // [C] RetryLoop: a connection loss is retried as the retry 
policy allows
+                if (attempt++ < retries) {
+                    dst.block(retrySleep, "retry of " + what);
+                    // [C] SleepingRetry.allowRetry: a sleep that is 
interrupted ends the retries (the thread of a closed
+                    // group does not make the next attempt)
+                    checkInterrupted(what, false);
+                    continue;
+                }
+                throw e;
+            }
+        }
+    }
+
+    private <T> T attempt(String what, boolean write, Callable<T> apply) 
throws Exception {
+        long l = latency();
+        if (!network) {
+            dst.block(l, what + " (no connection)");
+            throw connectionLoss();
+        }
+        if (write) {
+            int k = outage;
+            T answer = apply.call();
+            dst.block(l, what);
+            checkInterrupted(what, true);
+            // [Z] the client will not know whether the operation succeeded: 
the change is made, and the connection is
+            // lost before its answer came; the client reconnects soon after
+            if (outage == k && network && faults && 
dst.chance(responseLostRate)) {
+                dst.trace("zk: the connection is lost before the answer of " + 
what);
+                disconnect(dst.between(10, 1000));
+            }
+            if (outage != k || !network) {
+                dst.trace("zk: the answer of " + what + " is lost");
+                throw connectionLoss();
+            }
+            return answer;
+        }
+        dst.block(l, what);
+        checkInterrupted(what, false);
+        if (!network) {
+            throw connectionLoss();
+        }
+        return apply.call();
+    }
+
+    private KeeperException.ConnectionLossException connectionLoss() {
+        notice(outage);
+        return new KeeperException.ConnectionLossException();
+    }
+
+    private Object execute(Builder b, Object[] args) throws Exception {
+        String path = (String) args[0];
+        switch (b.kind) {
+            case "create": {
+                byte[] data = args.length > 1 ? (byte[]) args[1] : new byte[0];
+                CreateMode mode = b.mode != null ? b.mode : 
CreateMode.PERSISTENT;
+                return op("create " + path, true, () -> {
+                    if (!tree.containsKey(parentOf(path))) {
+                        if (!b.creatingParents) {
+                            throw new 
KeeperException.NoNodeException(parentOf(path));
+                        }
+                        ensureParents(path);
+                    }
+                    if (!mode.isSequential() && tree.containsKey(path)) {
+                        throw new KeeperException.NodeExistsException(path);
+                    }
+                    String created = createNode(path, data, mode, session);
+                    dst.trace("zk: created " + created + " (session " + 
session + ")");
+                    return created;
+                });
+            }
+            case "setData": {
+                byte[] data = (byte[]) args[1];
+                return op("setData " + path, true, () -> {
+                    Znode z = tree.get(path);
+                    if (z == null) {
+                        throw new KeeperException.NoNodeException(path);
+                    }
+                    z.data = data;
+                    z.version++;
+                    z.mzxid = ++zxid;
+                    z.mtime = dst.currentTimeMillis();
+                    fireDataWatches(path, 
Watcher.Event.EventType.NodeDataChanged);
+                    return stat(z, new Stat());
+                });
+            }
+            case "delete": {
+                if (b.background) {
+                    if (!b.guaranteed) {
+                        violations.add("not modelled: a delete in the 
background that is not guaranteed");
+                        throw new UnsupportedOperationException("not modelled: 
delete().inBackground()");
+                    }
+                    // [C] a background operation is done on the background 
thread of Curator once the client is
+                    // connected, and a guaranteed delete that fails is 
recorded and done again until it succeeds: the
+                    // same as a recorded failed delete here (done after a 
latency while connected, else after the
+                    // reconnect to the same session; when the session 
expired, the ensemble removed the znode)
+                    if (failedDeletes.add(path)) {
+                        retryDelete(path);
+                    }
+                    return null;
+                }
+                try {
+                    return op("delete " + path, true, () -> {
+                        if (!tree.containsKey(path)) {
+                            throw new KeeperException.NoNodeException(path);
+                        }
+                        remove(path);
+                        dst.trace("zk: deleted " + path);
+                        return null;
+                    });
+                } catch (KeeperException.ConnectionLossException e) {
+                    if (b.guaranteed && failedDeletes.add(path)) {
+                        // [C] the failed delete is recorded and done again in 
the background (at the latest after the
+                        // reconnect, below)
+                        retryDelete(path);
+                    }
+                    throw e;
+                }
+            }
+            case "getChildren":
+                return op("getChildren " + path, false, () -> {
+                    if (!tree.containsKey(path)) {
+                        throw new KeeperException.NoNodeException(path);
+                    }
+                    if (b.watcher != null) {
+                        childWatches.computeIfAbsent(path, p -> new 
LinkedHashSet<>()).add(b.watcher);
+                    }
+                    List<String> names = new ArrayList<>();
+                    for (String p : tree.keySet()) {
+                        if (isChild(path, p)) {
+                            names.add(p.substring(path.length() + 1));
+                        }
+                    }
+                    return names;
+                });
+            case "getData":
+                return op("getData " + path, false, () -> {
+                    Znode z = tree.get(path);
+                    if (z == null) {
+                        // [Z] no watch is left for a znode that does not 
exist (getData)
+                        throw new KeeperException.NoNodeException(path);
+                    }
+                    if (b.watcher != null) {
+                        dataWatches.computeIfAbsent(path, p -> new 
LinkedHashSet<>()).add(b.watcher);
+                    }
+                    if (b.stat != null) {
+                        stat(z, b.stat);
+                    }
+                    return z.data;
+                });
+            default:
+                violations.add("not modelled: " + b.kind + " " + b.calls);
+                throw new UnsupportedOperationException("not modelled: " + 
b.kind);
+        }
+    }
+
+    private Stat stat(Znode z, Stat stat) {
+        stat.setVersion(z.version);
+        stat.setCversion(z.cversion);
+        stat.setCzxid(z.czxid);
+        stat.setMzxid(z.mzxid);
+        stat.setCtime(z.ctime);
+        stat.setMtime(z.mtime);
+        stat.setEphemeralOwner(z.owner);
+        stat.setDataLength(z.data != null ? z.data.length : 0);
+        return stat;
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the proxies
+
+    private Object invokeClient(Object proxy, Method method, Object[] args) {
+        switch (method.getName()) {
+            case "create":
+            case "delete":
+            case "getData":
+            case "setData":
+            case "getChildren":
+                return builder(new Builder(method.getName()), 
method.getReturnType());
+            case "getConnectionStateListenable":
+                return listenable;
+            case "getZookeeperClient":
+                return zookeeperClient;
+            case "newNamespaceAwareEnsurePath":
+                return new FakeEnsurePath((String) args[0]);
+            case "clearWatcherReferences":
+                // Curator 5.9 does nothing here 
(CuratorFrameworkImpl.clearWatcherReferences is empty)
+                return null;
+            case "close":
+                violations.add("the shared CuratorFramework was closed");
+                closed = true;
+                return null;
+            case "getState":
+                return closed ? CuratorFrameworkState.STOPPED : 
CuratorFrameworkState.STARTED;
+            case "isStarted":
+                return !closed;
+            case "toString":
+                return "FakeZooKeeper.client";
+            case "hashCode":
+                return System.identityHashCode(proxy);
+            case "equals":
+                return proxy == args[0];
+            default:
+                violations.add("not modelled: CuratorFramework." + 
method.getName());
+                throw new UnsupportedOperationException("not modelled: 
CuratorFramework." + method.getName());
+        }
+    }
+
+    private final Listenable<ConnectionStateListener> listenable = new 
Listenable<>() {
+        @Override
+        public void addListener(ConnectionStateListener listener) {
+            if (!listeners.contains(listener)) {
+                listeners.add(listener);
+            }
+        }
+
+        @Override
+        public void addListener(ConnectionStateListener listener, Executor 
executor) {
+            addListener(listener);
+        }
+
+        @Override
+        public void removeListener(ConnectionStateListener listener) {
+            listeners.remove(listener);
+        }
+    };
+
+    /** The calls of a fluent Curator builder, up to the forPath that runs it. 
*/
+    private static final class Builder {
+        final String kind;
+        final List<String> calls = new ArrayList<>();
+        boolean creatingParents;
+        boolean guaranteed;
+        boolean background;
+        CreateMode mode;
+        Watcher watcher;
+        Stat stat;
+
+        Builder(String kind) {
+            this.kind = kind;
+        }
+    }
+
+    private Object builder(Builder b, Class<?> type) {
+        InvocationHandler h = (proxy, method, args) -> {
+            String name = method.getName();
+            switch (name) {
+                case "forPath":
+                    return execute(b, args);
+                case "creatingParentsIfNeeded":
+                    b.creatingParents = true;
+                    break;
+                case "guaranteed":
+                    b.guaranteed = true;
+                    break;
+                case "inBackground":
+                    if (args != null && args.length > 0) {
+                        violations.add("not modelled: " + b.kind + 
"().inBackground with a callback");
+                        throw new UnsupportedOperationException("not modelled: 
inBackground with a callback");
+                    }
+                    b.background = true;
+                    break;
+                case "withMode":
+                    b.mode = (CreateMode) args[0];
+                    break;
+                case "usingWatcher":
+                    if (args[0] instanceof Watcher w) {
+                        b.watcher = w;
+                    } else {
+                        violations.add("not modelled: a CuratorWatcher");
+                        throw new UnsupportedOperationException("not modelled: 
a CuratorWatcher");
+                    }
+                    break;
+                case "storingStatIn":
+                    b.stat = (Stat) args[0];
+                    break;
+                case "toString":
+                    return b.kind + b.calls;
+                case "hashCode":
+                    return System.identityHashCode(proxy);
+                case "equals":
+                    return proxy == args[0];
+                default:
+                    violations.add("not modelled: " + b.kind + "()." + name);
+                    throw new UnsupportedOperationException("not modelled: " + 
b.kind + "()." + name);
+            }
+            b.calls.add(name);
+            Class<?> next = returnType(method, type);
+            if (next == null || !next.isInterface()) {
+                violations.add("not modelled: " + b.kind + "()." + name + " 
returning " + next);
+                throw new UnsupportedOperationException(name);
+            }
+            return builder(b, next);
+        };
+        return Proxy.newProxyInstance(CuratorFramework.class.getClassLoader(), 
new Class<?>[] { type }, h);
+    }
+
+    /**
+     * The interface a builder method returns: the builders of Curator are 
generic (Watchable&lt;T&gt; returns T), so
+     * the type variable is bound through the generic interfaces of the 
builder the method is called on.
+     */
+    private static Class<?> returnType(Method method, Class<?> on) {
+        Type t = method.getGenericReturnType();
+        if (t instanceof TypeVariable<?> tv) {
+            t = bind(on, new HashMap<>(), tv);
+        }
+        if (t instanceof Class<?> c) {
+            return c;
+        }
+        if (t instanceof ParameterizedType p) {
+            return (Class<?>) p.getRawType();
+        }
+        return null;
+    }
+
+    private static Type bind(Class<?> c, Map<TypeVariable<?>, Type> env, 
TypeVariable<?> target) {
+        for (Type gi : c.getGenericInterfaces()) {
+            Map<TypeVariable<?>, Type> inner = new HashMap<>();
+            Class<?> raw;
+            if (gi instanceof ParameterizedType p) {
+                raw = (Class<?>) p.getRawType();
+                TypeVariable<?>[] vars = raw.getTypeParameters();
+                Type[] actual = p.getActualTypeArguments();
+                for (int i = 0; i < vars.length; i++) {
+                    Type a = actual[i];
+                    if (a instanceof TypeVariable<?> av && 
env.containsKey(av)) {
+                        a = env.get(av);
+                    }
+                    inner.put(vars[i], a);
+                }
+            } else {
+                raw = (Class<?>) gi;
+            }
+            if (inner.containsKey(target)) {
+                return inner.get(target);
+            }
+            Type found = bind(raw, inner, target);
+            if (found != null) {
+                return found;
+            }
+        }
+        return null;
+    }
+
+    /** isConnected is what the client believes: it turns false when the 
client noticed the connection loss. */
+    private final class FakeZookeeperClient extends CuratorZookeeperClient {
+        FakeZookeeperClient() {
+            super("fake:2181", 60000, 15000, null, new RetryOneTime(1000));
+        }
+
+        @Override
+        public boolean isConnected() {
+            return clientConnected && !closed;
+        }
+    }
+
+    /**
+     * Makes the persistent parents of a path, as EnsurePath does, retried as 
the client retries: once, as after a
+     * success EnsurePath does nothing any more (its helper becomes a no-op, 
EnsurePath$1 in Curator 5.9).
+     */
+    private final class FakeEnsurePath extends EnsurePath {
+        private boolean done;
+
+        FakeEnsurePath(String path) {
+            super(path);
+        }
+
+        @Override
+        public void ensure(CuratorZookeeperClient c) throws Exception {
+            if (done) {
+                return;
+            }
+            String path = getPath();
+            op("ensure " + path, true, () -> {
+                if (!tree.containsKey(path)) {
+                    ensureParents(path + "/x");
+                }
+                return null;
+            });
+            done = true;
+        }
+    }
+}
diff --git 
a/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/ZooKeeperMasterDstTest.java
 
b/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/ZooKeeperMasterDstTest.java
new file mode 100644
index 000000000000..b3ef410d7dfd
--- /dev/null
+++ 
b/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/ZooKeeperMasterDstTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.zookeepermaster.dst;
+
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstInstrument;
+import org.apache.camel.test.dst.DstTest;
+
+/**
+ * Deterministic simulation test of the camel-zookeeper-master consumer and 
route policy against a fake ZooKeeper
+ * ensemble, with connection losses, expired sessions, other nodes that join, 
leave and die, consumer starts that take
+ * long or fail, and route stops, starts and removals: this node runs its 
consumer only while it is the master, and once
+ * the faults cleared the master runs its consumer. A failing seed is replayed 
with {@code -Ddst.seed=<seed>};
+ * {@code -Ddst.seeds=5000} runs more seeds.
+ */
+@DstInstrument("org.apache.camel.component.zookeepermaster")
+class ZooKeeperMasterDstTest {
+
+    @DstTest(seeds = 1000)
+    void onlyTheMasterRunsTheConsumer(Dst dst) throws Exception {
+        new ZooKeeperMasterWorld(dst).run();
+    }
+}
diff --git 
a/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/ZooKeeperMasterWorld.java
 
b/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/ZooKeeperMasterWorld.java
new file mode 100644
index 000000000000..6fed06111f0b
--- /dev/null
+++ 
b/components/camel-zookeeper-master/src/test/java/org/apache/camel/component/zookeepermaster/dst/ZooKeeperMasterWorld.java
@@ -0,0 +1,734 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.zookeepermaster.dst;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.Endpoint;
+import org.apache.camel.Processor;
+import org.apache.camel.Producer;
+import org.apache.camel.ServiceStatus;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.zookeepermaster.group.Group;
+import org.apache.camel.component.zookeepermaster.group.GroupListener;
+import org.apache.camel.component.zookeepermaster.group.ManagedGroupFactory;
+import org.apache.camel.component.zookeepermaster.group.NodeState;
+import 
org.apache.camel.component.zookeepermaster.group.internal.ZooKeeperGroup;
+import org.apache.camel.component.zookeepermaster.policy.MasterRoutePolicy;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.apache.camel.support.DefaultComponent;
+import org.apache.camel.support.DefaultConsumer;
+import org.apache.camel.support.DefaultEndpoint;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstCamel;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.camel.test.dst.DstWorld;
+import org.apache.camel.test.dst.Invariants;
+import org.apache.curator.framework.CuratorFramework;
+
+/**
+ * The world around one node of a camel-zookeeper-master group: the real 
master consumer ({@code zookeeper-master:}
+ * endpoint) or the real {@link MasterRoutePolicy}, with the real {@code 
ZooKeeperGroup} of the component, on a fake
+ * ZooKeeper ensemble ({@link FakeZooKeeper}) shared through the {@code 
curator} bean, as an application shares one
+ * CuratorFramework. The other nodes of the group are sessions of the fake 
(peers) that join, leave and die. The
+ * consumer that only the master runs is a test consumer whose start takes a 
while and sometimes fails, as a consumer
+ * that connects to a broker does.
+ * <p>
+ * The seed picks the mode (master consumer or route policy), the session 
timeout, the retry policy of the client, the
+ * start time of the consumer, the fault rates, and the operations: connection 
losses shorter or longer than the session
+ * timeout, peers that join, leave or die, route stops and starts (master 
consumer), routes removed and added again and
+ * route starts from outside (route policy).
+ * <p>
+ * The groups are made by a {@link ManagedGroupFactory} bean, as 
DefaultManagedGroupFactory makes them (a ZooKeeperGroup
+ * on a single thread executor of the ExecutorServiceManager), to stand in for 
the thread interrupt of
+ * {@code ExecutorService.shutdownNow} that the harness does not do. A real 
close of a {@code ZooKeeperGroup} interrupts
+ * its thread: a ZooKeeper call it waits in throws (a write already sent is 
applied), the composite operation stops, and
+ * the wait of the main loop on its queue ends. The fake fails the calls of 
the thread of a closed group, and the world
+ * offers the closed group a refresh (public {@code clearAndRefresh}), after 
which its loop ends. A close therefore
+ * waits the 5 seconds of its {@code awaitTermination} in virtual time (a real 
close waits only while the thread is
+ * busy), so the znode of a closed group is deleted 5 seconds after the close 
began, and a peer takes over that much
+ * later. As the factory is a bean of the registry, the default path 
(DefaultGroupFactoryStrategy and
+ * DefaultManagedGroupFactory) does not run; a bean is also what lets the 
world check that a consumer or a policy does
+ * not close a group factory it did not create.
+ * <p>
+ * Properties: this node runs its consumer only while it is the master 
(allowing the time of a start in flight and of
+ * the events to arrive); once the faults cleared, the master runs its 
consumer, and this node does when it is the
+ * master; a stopped consumer or a removed route leaves no listener on the 
shared CuratorFramework and does not keep the
+ * leadership of the group with a znode of its own; the shared 
CuratorFramework and group factory are not closed by a
+ * consumer or a policy.
+ * <p>
+ * Not explored: several master endpoints or policies of one node in the same 
group (each ZooKeeperGroup main loop runs
+ * for ever, and on the one stack of the harness a second loop nested above 
the first would never let it resume), route
+ * suspend and resume, the CuratorFramework the component creates itself, a 
stop of the route policy on its own (from
+ * JMX), the camel-master cluster services, and a stop while the thread of the 
group waits in a call of ZooKeeper (see
+ * {@code mustWaitToStop}). The route policy starts its consumer at once (see 
{@code maxStart}).
+ * <p>
+ * Left out of the default set, each a known bug (a system property adds 
each): a route stopped while the client is
+ * disconnected leaves the znode of its group behind, CAMEL-27037 ({@code 
-Ddst.zookeeper.stopWhileSuspended=true}); a
+ * route stopped while the group waits for the answer of a create, or makes 
one after the close interrupted its thread,
+ * leaves the znode of that create behind, CAMEL-27037 ({@code 
-Ddst.zookeeper.stopDuringCreate=true}); and a route of
+ * the policy started from outside when it is started already starts its 
consumer again without the onStart of the
+ * policy, CAMEL-27044 ({@code -Ddst.zookeeper.startStartedRoute=true}).
+ */
+@DstWorld(component = "zookeeper-master",
+          fake = "FakeZooKeeper: a ZooKeeper ensemble (sessions, ephemeral 
sequential znodes, one-time watches) with the"
+                 + " other nodes of the group as sessions, behind a 
CuratorFramework shared as the curator bean",
+          covers = {
+                  "the master consumer (zookeeper-master endpoint) and the 
MasterRoutePolicy, on the real ZooKeeperGroup",
+                  "connection losses shorter and longer than the session 
timeout (SUSPENDED, RECONNECTED, LOST)",
+                  "other nodes that join, leave and die",
+                  "a start of the consumer that fails and its retry; with the 
master consumer, a start that takes long",
+                  "route stop and start, and a route of the policy removed and 
added again" },
+          faults = {
+                  "the connection to ZooKeeper is lost", "the session 
expires", "an operation answers slowly",
+                  "the connection is lost before the answer of a write", "the 
start of the consumer fails",
+                  "the start of the consumer takes long" },
+          invariants = {
+                  "the client is used as ZooKeeper and Curator allow",
+                  "this node runs its consumer only while it is the master",
+                  "once the faults cleared, the master runs its consumer",
+                  "a stopped consumer leaves no listener on the shared 
CuratorFramework",
+                  "no znode of this node is left in the group after the final 
stop",
+                  "the shared CuratorFramework and group factory are not 
closed",
+                  "the world itself ran as planned" },
+          issues = { "CAMEL-25315", "CAMEL-26540", "CAMEL-26802", 
"CAMEL-26870", "CAMEL-26887", "CAMEL-27037", "CAMEL-27044" })
+final class ZooKeeperMasterWorld {
+
+    static final String ROUTE = "master";
+    static final String GROUP = "dst";
+    static final String GROUP_PATH = "/camel/zookeepermaster/clusters/master/" 
+ GROUP;
+
+    private final Dst dst;
+    private final FakeZooKeeper zk;
+    private final boolean policyMode;
+    private final long maxStart;
+    private final double startFailRate;
+    private final long grace;
+    // CAMEL-27044: a started route of the policy started again starts its 
consumer without the onStart of the policy
+    private final boolean startStartedRoute = 
Boolean.getBoolean("dst.zookeeper.startStartedRoute");
+    // CAMEL-27037: a group closed while the client is disconnected does not 
delete its znode
+    private final boolean stopWhileSuspended = 
Boolean.getBoolean("dst.zookeeper.stopWhileSuspended");
+    // CAMEL-27037: a create interrupted by the close of the group is applied, 
and its znode stays
+    private final boolean stopDuringCreate = 
Boolean.getBoolean("dst.zookeeper.stopDuringCreate");
+    private final List<String> violations = new ArrayList<>();
+    private final List<WorkConsumer> running = new ArrayList<>();
+    private final RecordingGroupFactory groupFactory;
+    private MasterRoutePolicy policy;
+    private boolean faults = true;
+    private boolean routeActive;
+    private boolean finished;
+    private int opsLeft;
+    private int baselineListeners;
+    private long wrongSince = -1;
+    private int startsInFlight;
+    private long lastStartEnd = -1;
+    private String wrongWhy;
+    private String finalCheck;
+    private String afterStop;
+    private int peerNumber;
+
+    ZooKeeperMasterWorld(Dst dst) {
+        this.dst = dst;
+        this.zk = new FakeZooKeeper(dst);
+        this.groupFactory = new RecordingGroupFactory();
+        this.policyMode = dst.chance(0.5);
+        zk.sessionTimeout = dst.pick(List.of(4000L, 10_000L));
+        zk.retries = dst.pick(List.of(0, 1));
+        // a write of a thread the close interrupted is applied by ZooKeeper, 
which leaves a znode behind as in a stop
+        // during a create (CAMEL-27037)
+        zk.writesAfterInterrupt = stopDuringCreate;
+        zk.slowRate = dst.pick(List.of(0.0, 0.05));
+        // a connection lost before the answer of a write: the client retries 
it; a create retried that way makes a second
+        // znode, which the group removes as it finds it with its own uuid 
(prunePartialState)
+        zk.responseLostRate = dst.pick(List.of(0.0, 0.05));
+        // the route policy starts the consumer while it holds its leadership 
lock, and locks do not exclude on the one
+        // stack of the harness: a start that blocks would let a disconnect or 
a removal run inside that lock, which
+        // cannot happen for real. Its starts take no time
+        this.maxStart = policyMode ? 0 : dst.pick(List.of(100L, 3000L, 8000L));
+        this.startFailRate = dst.pick(List.of(0.0, 0.3, 0.6));
+        // the events to arrive, once no start of the consumer is in flight (a 
start that began while this node was the
+        // master completes, and the consumer stops it then)
+        this.grace = 2000;
+        dst.trace("world: " + config());
+    }
+
+    private String config() {
+        return (policyMode ? "MasterRoutePolicy" : "master consumer") + " 
sessionTimeout=" + zk.sessionTimeout
+               + " retries=" + zk.retries + " slow=" + zk.slowRate + " 
answerLost=" + zk.responseLostRate
+               + " maxStart=" + maxStart + " startFail=" + startFailRate;
+    }
+
+    void run() throws Exception {
+        DefaultCamelContext context = DstCamel.newContext(dst);
+        context.getRegistry().bind("curator", zk.client());
+        context.getRegistry().bind("groupFactory", groupFactory);
+        groupFactory.curator = zk.client();
+        groupFactory.context = context;
+        context.addComponent("work", new WorkComponent());
+        if (policyMode) {
+            policy = new MasterRoutePolicy();
+            policy.setGroupName(GROUP);
+            if (dst.chance(0.5)) {
+                policy.setCurator(zk.client());
+            }
+        }
+        context.addRoutes(routes());
+        context.start();
+        baselineListeners = policyMode ? 0 : zk.listenerCount();
+        routeActive = policyMode;
+        DstExecutor operator = dst.executor("operator", 1);
+        try {
+            if (!policyMode) {
+                operator.schedule("start route", 0, () -> startRoute(context));
+            }
+            planPeers(operator);
+            planOperations(context, operator);
+            DstExecutor monitor = dst.executor("monitor", 1);
+            monitor.scheduleWithFixedDelay(this::monitor, 100, 100, 
TimeUnit.MILLISECONDS);
+            operator.schedule("finisher", 1000, () -> finisher(context, 
operator, -1));
+            dst.runUntil(() -> finished, "the finisher");
+            dst.check(
+                    Invariants.of("the client is used as ZooKeeper and Curator 
allow",
+                            () -> zk.violations.isEmpty() ? null : 
zk.violations + " (" + config() + ")"),
+                    Invariants.of("this node runs its consumer only while it 
is the master",
+                            () -> violations.stream().noneMatch(v -> 
v.startsWith("runs"))
+                                    ? null : violations + " (" + config() + 
")"),
+                    Invariants.of("once the faults cleared, the master runs 
its consumer",
+                            () -> finalCheck == null ? null : finalCheck + " 
(" + config() + ")"),
+                    Invariants.of("a stopped consumer leaves no listener on 
the shared CuratorFramework",
+                            () -> violations.stream().noneMatch(v -> 
v.startsWith("listeners"))
+                                    ? null : violations + " (" + config() + 
")"),
+                    Invariants.of("no znode of this node is left in the group 
after the final stop",
+                            () -> afterStop == null ? null : afterStop + " (" 
+ config() + ")"),
+                    Invariants.of("the shared CuratorFramework and group 
factory are not closed",
+                            () -> !groupFactory.closed && !zk.isClosed()
+                                    ? null : "group factory closed=" + 
groupFactory.closed + " client closed="
+                                             + zk.isClosed() + " (" + config() 
+ ")"),
+                    Invariants.of("the world itself ran as planned",
+                            () -> violations.stream().noneMatch(v -> 
v.startsWith("operator"))
+                                    ? null : violations + " (" + config() + 
")"));
+        } finally {
+            dst.dropPending();
+            context.stop();
+        }
+    }
+
+    private RouteBuilder routes() {
+        return new RouteBuilder() {
+            @Override
+            public void configure() {
+                if (policyMode) {
+                    
from("work:in").routeId(ROUTE).autoStartup(false).routePolicy(policy)
+                            .process(exchange -> {
+                                // nothing is routed: the world looks at which 
node runs the consumer
+                            });
+                } else {
+                    from("zookeeper-master:" + GROUP + 
":work:in").routeId(ROUTE).autoStartup(false)
+                            .process(exchange -> {
+                                // nothing is routed: the world looks at which 
node runs the consumer
+                            });
+                }
+            }
+        };
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the properties, checked every 100 ms
+
+    private boolean localIsMaster() {
+        String master = zk.master(GROUP_PATH);
+        String id = groupFactory.currentId();
+        return master != null && master.equals(id);
+    }
+
+    private void monitor() {
+        if (finished) {
+            return;
+        }
+        if (running.size() > 1 && violations.stream().noneMatch(v -> 
v.startsWith("runs two"))) {
+            violations.add("runs two consumers at once at t=" + dst.now());
+        }
+        String why = null;
+        if (!running.isEmpty()) {
+            if (!routeActive) {
+                why = "its route is stopped";
+            } else if (!zk.isClientConnected()) {
+                why = "it is disconnected from ZooKeeper";
+            } else if (!localIsMaster()) {
+                String master = zk.master(GROUP_PATH);
+                String peer = zk.peerOf(master);
+                why = peer != null ? "peer " + peer + " is the master" : "the 
master is " + master;
+            }
+        }
+        if (why == null) {
+            wrongSince = -1;
+            return;
+        }
+        if (wrongSince < 0) {
+            wrongSince = dst.now();
+            wrongWhy = why;
+        } else if (dst.now() - wrongSince > grace && startsInFlight == 0 && 
dst.now() - lastStartEnd > grace
+                && violations.stream().noneMatch(v -> v.startsWith("runs"))) {
+            violations.add("runs its consumer while " + wrongWhy + " since t=" 
+ wrongSince + " (now " + dst.now()
+                           + ", last start ended at t=" + lastStartEnd + ", 
grace " + grace + ")");
+        }
+    }
+
+    private void checkStopped(String when) {
+        if (!running.isEmpty() && startsInFlight == 0
+                && violations.stream().noneMatch(v -> v.startsWith("runs"))) {
+            violations.add("runs its consumer " + when + " at t=" + dst.now());
+        }
+    }
+
+    private void checkListeners(String when) {
+        if (zk.listenerCount() != baselineListeners
+                && violations.stream().noneMatch(v -> 
v.startsWith("listeners"))) {
+            violations.add("listeners on the shared CuratorFramework " + when 
+ ": " + zk.listenerCount()
+                           + ", expected " + baselineListeners + " at t=" + 
dst.now());
+        }
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the load and the operator
+
+    private void planPeers(DstExecutor operator) {
+        int initial = (int) dst.between(0, 2);
+        for (int i = 0; i < initial; i++) {
+            String c = "peer-" + ++peerNumber;
+            operator.schedule("peer joins", dst.between(0, 2000), () -> 
zk.peerJoins(GROUP_PATH, c));
+        }
+        int changes = (int) dst.between(0, 4);
+        long t = dst.between(1000, 5000);
+        for (int i = 0; i < changes; i++) {
+            opsLeft++;
+            operator.schedule("peer change", t, () -> {
+                if (!faults) {
+                    opsLeft--;
+                    return;
+                }
+                List<String> present = new ArrayList<>(zk.peers());
+                if (present.isEmpty() || dst.chance(0.4)) {
+                    zk.peerJoins(GROUP_PATH, "peer-" + ++peerNumber);
+                } else if (dst.chance(0.5)) {
+                    zk.peerLeaves(dst.pick(present));
+                } else {
+                    zk.peerDies(dst.pick(present));
+                }
+                opsLeft--;
+            });
+            t += dst.between(1000, 15_000);
+        }
+    }
+
+    private void planOperations(DefaultCamelContext context, DstExecutor 
operator) {
+        int ops = (int) dst.between(0, 4);
+        long t = dst.between(500, 4000);
+        for (int i = 0; i < ops; i++) {
+            opsLeft++;
+            int kind = (int) dst.between(0, 2);
+            if (kind == 0) {
+                long length = dst.chance(0.5)
+                        ? dst.between(100, zk.sessionTimeout / 2) : 
dst.between(zk.sessionTimeout, 3 * zk.sessionTimeout);
+                operator.schedule("connection loss", t, () -> {
+                    zk.disconnect(length);
+                    opsLeft--;
+                });
+            } else if (policyMode && kind == 1) {
+                operator.schedule("remove route", t, () -> 
removeRoute(context, operator));
+            } else if (policyMode) {
+                operator.schedule("start route from outside", t, () -> {
+                    // a route that is already started is started again only 
with -Ddst.zookeeper.startStartedRoute=true:
+                    // that starts its consumer again without the onStart of 
the policy (CAMEL-27044)
+                    if (startStartedRoute || 
context.getRouteController().getRouteStatus(ROUTE) != ServiceStatus.Started) {
+                        startRoute(context);
+                    }
+                    opsLeft--;
+                });
+            } else {
+                operator.schedule("stop route", t, () -> stopRoute(context, 
operator));
+            }
+            t += dst.between(1000, 20_000);
+        }
+    }
+
+    private void startRoute(CamelContext context) {
+        try {
+            context.getRouteController().startRoute(ROUTE);
+            routeActive = true;
+        } catch (Exception e) {
+            if (policyMode && causedByStartFault(e)) {
+                // the consumer of a route started from outside failed to 
start: the route stays stopped
+                return;
+            }
+            violations.add("operator: start failed: " + e);
+        }
+    }
+
+    private static boolean causedByStartFault(Throwable e) {
+        for (Throwable t = e; t != null; t = t.getCause()) {
+            if (t.getMessage() != null && t.getMessage().startsWith("dst: the 
consumer cannot connect")) {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    /**
+     * Whether the operator waits before it stops or removes the route:
+     * <ul>
+     * <li>while the thread of the group waits in a call of ZooKeeper other 
than a create (a read, a setData, a delete,
+     * the ensure of the path): a real close interrupts that call and waits 
for the thread to end before it leaves the
+     * group, so the thread records the znode it made first; on the one stack 
of the harness the thread can only end
+     * after the close returned, so the close would leave the group before (a 
limit of the harness);</li>
+     * <li>while the thread of the group waits in a create (the create is 
applied, and the group never records the
+     * znode, which stays in the group as long as the session of the shared 
client lives; CAMEL-27037,
+     * {@code -Ddst.zookeeper.stopDuringCreate=true} adds it);</li>
+     * <li>while the client is disconnected (a group closed then does not 
delete its znode, with the same result;
+     * CAMEL-27037, {@code -Ddst.zookeeper.stopWhileSuspended=true} adds 
it).</li>
+     * </ul>
+     * A stop while the thread starts the consumer is driven.
+     */
+    private boolean mustWaitToStop() {
+        return zk.inCallOtherThanCreate("ZooKeeperGroup-") || 
!stopDuringCreate && zk.inCreate("ZooKeeperGroup-")
+                || !stopWhileSuspended && !zk.isClientConnected();
+    }
+
+    private void stopRoute(DefaultCamelContext context, DstExecutor operator) {
+        if (context.getRouteController().getRouteStatus(ROUTE) != 
ServiceStatus.Started) {
+            // stopped already by an earlier operation, which starts it again
+            opsLeft--;
+            return;
+        }
+        if (mustWaitToStop()) {
+            operator.schedule("stop route (later)", 10, () -> 
stopRoute(context, operator));
+            return;
+        }
+        try {
+            context.getRouteController().stopRoute(ROUTE);
+        } catch (Exception e) {
+            violations.add("operator: stop failed: " + e);
+        }
+        routeActive = false;
+        checkStopped("after the route stopped");
+        checkListeners("after the route stopped");
+        operator.schedule("start route", dst.between(100, 10_000), () -> {
+            startRoute(context);
+            opsLeft--;
+        });
+    }
+
+    private void removeRoute(DefaultCamelContext context, DstExecutor 
operator) {
+        if (context.getRoute(ROUTE) == null) {
+            // removed already by an earlier operation, which adds it again
+            opsLeft--;
+            return;
+        }
+        if (mustWaitToStop()) {
+            operator.schedule("remove route (later)", 10, () -> 
removeRoute(context, operator));
+            return;
+        }
+        try {
+            if (context.getRouteController().getRouteStatus(ROUTE) == 
ServiceStatus.Started) {
+                context.getRouteController().stopRoute(ROUTE);
+            }
+            context.removeRoute(ROUTE);
+        } catch (Exception e) {
+            violations.add("operator: remove failed: " + e);
+        }
+        routeActive = false;
+        checkStopped("after the route was removed");
+        checkListeners("after the route was removed");
+        operator.schedule("add route again", dst.between(100, 10_000), () -> {
+            try {
+                context.addRoutes(routes());
+                routeActive = true;
+            } catch (Exception e) {
+                violations.add("operator: add failed: " + e);
+            }
+            opsLeft--;
+        });
+    }
+
+    private void finisher(DefaultCamelContext context, DstExecutor operator, 
long deadline) {
+        if (deadline == -1 && opsLeft > 0) {
+            operator.schedule("finisher", 500, () -> finisher(context, 
operator, -1));
+            return;
+        }
+        if (deadline == -1) {
+            faults = false;
+            zk.faults = false;
+            dst.trace("world: faults cleared");
+            // the other nodes leave sometimes, so that this node becomes the 
master
+            if (dst.chance(0.5)) {
+                new ArrayList<>(zk.peers()).forEach(zk::peerLeaves);
+            }
+            long d = dst.now() + 5 * 60_000L;
+            operator.schedule("finisher", 500, () -> finisher(context, 
operator, d));
+            return;
+        }
+        boolean expected = routeActive && zk.isClientConnected() && 
localIsMaster();
+        String master = zk.master(GROUP_PATH);
+        // the master is this node's znode or a peer that is alive: else no 
node runs the consumer
+        boolean liveMaster = master != null
+                && (master.equals(groupFactory.currentId())
+                        || zk.peerOf(master) != null && 
zk.peers().contains(zk.peerOf(master)));
+        boolean settled = zk.isNetworkUp() && zk.isClientConnected() && 
zk.pendingDeletes().isEmpty()
+                && (expected ? running.size() == 1 : running.isEmpty())
+                && (liveMaster || !routeActive && zk.peers().isEmpty());
+        if (!settled && dst.now() < deadline) {
+            operator.schedule("finisher", 500, () -> finisher(context, 
operator, deadline));
+            return;
+        }
+        if (mustWaitToStop()) {
+            operator.schedule("finisher (later)", 10, () -> finisher(context, 
operator, deadline));
+            return;
+        }
+        if (!settled) {
+            finalCheck = "at the deadline: route active=" + routeActive + " 
connected=" + zk.isClientConnected()
+                         + " master=" + master + " (peer " + zk.peerOf(master) 
+ ") this node=" + groupFactory.currentId()
+                         + " local znodes=" + zk.localZnodes(GROUP_PATH) + " 
consumers running=" + running.size();
+        }
+        // the final stop: the consumer or the route of the policy leaves the 
group
+        try {
+            if (context.getRouteController().getRouteStatus(ROUTE) == 
ServiceStatus.Started) {
+                context.getRouteController().stopRoute(ROUTE);
+            }
+            if (policyMode) {
+                context.removeRoute(ROUTE);
+            }
+            routeActive = false;
+        } catch (Exception e) {
+            violations.add("operator: final stop failed: " + e);
+        }
+        checkListeners("after the final stop");
+        operator.schedule("after the final stop", 2000, () -> {
+            List<String> left = zk.localZnodes(GROUP_PATH);
+            if (!left.isEmpty()) {
+                String last = zk.master(GROUP_PATH);
+                afterStop = "after the final stop this node still has the 
znodes " + left + " in the group (the master is "
+                            + last + (left.contains(last) ? ", one of them)" : 
")");
+            }
+            if (!running.isEmpty()) {
+                violations.add("runs its consumer after the final stop");
+            }
+            finished = true;
+        });
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the consumer that only the master runs
+
+    private final class WorkComponent extends DefaultComponent {
+        @Override
+        protected Endpoint createEndpoint(String uri, String remaining, 
Map<String, Object> parameters) {
+            return new WorkEndpoint(uri, this);
+        }
+    }
+
+    private final class WorkEndpoint extends DefaultEndpoint {
+        WorkEndpoint(String uri, WorkComponent component) {
+            super(uri, component);
+        }
+
+        @Override
+        public Producer createProducer() {
+            throw new UnsupportedOperationException("work: consumer only");
+        }
+
+        @Override
+        public Consumer createConsumer(Processor processor) {
+            return new WorkConsumer(this, processor);
+        }
+    }
+
+    /** A consumer whose start connects to a broker: it takes a while and 
fails sometimes. */
+    private final class WorkConsumer extends DefaultConsumer {
+        WorkConsumer(Endpoint endpoint, Processor processor) {
+            super(endpoint, processor);
+        }
+
+        @Override
+        protected void doStart() throws Exception {
+            startsInFlight++;
+            try {
+                if (maxStart > 0) {
+                    dst.block(dst.between(0, maxStart), "start of the 
consumer");
+                }
+                if (faults && dst.chance(startFailRate)) {
+                    throw Dst.injected(new IOException("dst: the consumer 
cannot connect to its broker"));
+                }
+                super.doStart();
+                running.add(this);
+                dst.trace("world: consumer started");
+            } finally {
+                startsInFlight--;
+                lastStartEnd = dst.now();
+            }
+        }
+
+        @Override
+        protected void doStop() throws Exception {
+            if (running.remove(this)) {
+                dst.trace("world: consumer stopped");
+            }
+            super.doStop();
+        }
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the group factory
+
+    /**
+     * A group factory as a bean of the registry. It makes the groups as 
DefaultManagedGroupFactory does (a
+     * ZooKeeperGroup on a single thread executor of the 
ExecutorServiceManager), with a thread name of its own per
+     * group, so that the fake can tell the thread of a closed group, which a 
real close interrupts.
+     */
+    private final class RecordingGroupFactory implements ManagedGroupFactory {
+        CuratorFramework curator;
+        CamelContext context;
+        boolean closed;
+        private int groups;
+        private ClosingGroup<?> current;
+
+        String currentId() {
+            ClosingGroup<?> g = current;
+            return g != null && !g.closed ? g.delegate.getId() : null;
+        }
+
+        @Override
+        public CuratorFramework getCurator() {
+            return curator;
+        }
+
+        @Override
+        public void close() {
+            closed = true;
+        }
+
+        @Override
+        public <T extends NodeState> Group<T> createGroup(String path, 
Class<T> clazz) {
+            String name = "ZooKeeperGroup-" + path + "-" + ++groups;
+            ExecutorService executor = 
context.getExecutorServiceManager().newSingleThreadExecutor(this, name);
+            ClosingGroup<T> g = new ClosingGroup<>(new 
ZooKeeperGroup<>(curator, path, clazz, executor), name);
+            current = g;
+            return g;
+        }
+
+        @Override
+        public <T extends NodeState> Group<T> createGroup(String path, 
Class<T> clazz, ThreadFactory threadFactory) {
+            return createGroup(path, clazz);
+        }
+
+        @Override
+        public <T extends NodeState> Group<T> createMultiGroup(String path, 
Class<T> clazz) {
+            violations.add("operator: a multi group is not modelled");
+            throw new UnsupportedOperationException("not modelled");
+        }
+
+        @Override
+        public <T extends NodeState> Group<T> createMultiGroup(String path, 
Class<T> clazz, ThreadFactory threadFactory) {
+            return createMultiGroup(path, clazz);
+        }
+    }
+
+    /**
+     * The real ZooKeeperGroup. Its close interrupts the thread of the group 
in the fake, and ends the wait of the
+     * group's main loop (see the class Javadoc).
+     */
+    private final class ClosingGroup<T extends NodeState> implements Group<T> {
+        final ZooKeeperGroup<T> delegate;
+        final String thread;
+        boolean closed;
+
+        ClosingGroup(ZooKeeperGroup<T> delegate, String thread) {
+            this.delegate = delegate;
+            this.thread = thread;
+        }
+
+        @Override
+        public void close() throws IOException {
+            closed = true;
+            zk.interrupt(thread);
+            delegate.close();
+            try {
+                delegate.clearAndRefresh();
+            } catch (Exception e) {
+                // the refresh only wakes the main loop of the closed group
+                dst.trace("world: refresh of a closed group failed: " + e);
+            }
+        }
+
+        @Override
+        public boolean isConnected() {
+            return delegate.isConnected();
+        }
+
+        @Override
+        public void start() {
+            delegate.start();
+        }
+
+        @Override
+        public void add(GroupListener<T> listener) {
+            delegate.add(listener);
+        }
+
+        @Override
+        public void remove(GroupListener<T> listener) {
+            delegate.remove(listener);
+        }
+
+        @Override
+        public void update(T state) {
+            delegate.update(state);
+        }
+
+        @Override
+        public Map<String, T> members() {
+            return delegate.members();
+        }
+
+        @Override
+        public boolean isMaster() {
+            return delegate.isMaster();
+        }
+
+        @Override
+        public T master() {
+            return delegate.master();
+        }
+
+        @Override
+        public List<T> slaves() {
+            return delegate.slaves();
+        }
+
+        @Override
+        public T getLastState() {
+            return delegate.getLastState();
+        }
+    }
+}
diff --git 
a/components/camel-zookeeper-master/src/test/resources/log4j2.properties 
b/components/camel-zookeeper-master/src/test/resources/log4j2.properties
index 1c5b0d8787c8..4c3f8310d969 100644
--- a/components/camel-zookeeper-master/src/test/resources/log4j2.properties
+++ b/components/camel-zookeeper-master/src/test/resources/log4j2.properties
@@ -15,9 +15,16 @@
 ## limitations under the License.
 ## ---------------------------------------------------------------------------
 
-appender.file.type = File
+# capped: the DST test runs the consumer for a thousand seeds
+appender.file.type = RollingFile
 appender.file.name = file
 appender.file.fileName = target/camel-zookeeper-master-test.log
+appender.file.filePattern = target/camel-zookeeper-master-test.%i.log
+appender.file.policies.type = Policies
+appender.file.policies.size.type = SizeBasedTriggeringPolicy
+appender.file.policies.size.size = 50MB
+appender.file.strategy.type = DefaultRolloverStrategy
+appender.file.strategy.max = 1
 appender.file.layout.type = PatternLayout
 appender.file.layout.pattern = %d [%-15.15t] %-5p %-30.30c{1} - %m%n
 appender.out.type = Console

Reply via email to