allthingssecurity opened a new pull request, #30137: URL: https://github.com/apache/camel/pull/30137
# Description [CAMEL-26974](https://issues.apache.org/jira/browse/CAMEL-26974), part of CAMEL-26610. What the world found on main, up front: three cases fail on main and are left out of the default set while they are looked at separately (each added back by a system property, see Tests): a master route stopped while the client is disconnected, or while its group waits for the answer of a create, leaves a znode of the node in the group for as long as the session of the shared CuratorFramework lives; and a route of the `MasterRoutePolicy` that is started again while it is started already (from JMX, say) runs its consumer on a node that is not the master. Differences from the JIRA text: a start of the consumer that takes long is driven with the master consumer only (the route policy starts its consumer while it holds its leadership lock, which the one stack of the harness cannot run with a start that takes time), and the check that no znode of the node is left in the group runs after the final stop. A deterministic simulation test (camel-test-dst) of the camel-zookeeper-master consumer and route policy, in the `dst` package of its tests, in the same form as the SQS world: `ZooKeeperMasterDstTest` (1000 seeds), `ZooKeeperMasterWorld` and `FakeZooKeeper`. Test only; no main code changes. What the world models: one node of a master group. The real master consumer (`zookeeper-master:` endpoint) or the real `MasterRoutePolicy` (the seed picks one) and the real `ZooKeeperGroup` of the component run on the virtual-time loop; only ZooKeeper is faked, behind the CuratorFramework that the application shares as the `curator` bean (a dynamic proxy of `CuratorFramework` and of its fluent builders). The fake is written from the ZooKeeper Programmer's Guide, the ZooKeeper leader election recipe and the Curator documentation, and cites them: sessions, ephemeral sequential znodes, one-time child and data watches (held while the client is disconnected and delivered after it reconnected to the same session), a session that expires when the client cannot reach the ensemble for longer than the session timeout, the connection states SUSPENDED, LOST and RECONNECTED on their own thread, the wait of an operation for the connection and its retry after a connection loss (the connection lo st before the answer of a write, which the client retries: a create retried that way makes a second znode, which the group removes), and guaranteed deletes done again in the background. The other nodes of the group are sessions of the fake that join, leave and die. The consumer that only the master runs is a test consumer whose start takes a while and sometimes fails, as a consumer that connects to a broker does. The groups are made by a `ManagedGroupFactory` bean, as `DefaultManagedGroupFactory` makes them, with a thread name of its own per group. This stands in for the thread interrupt of `ExecutorService.shutdownNow`, which the harness does not do: a real close of a `ZooKeeperGroup` interrupts its thread, so a ZooKeeper call it waits in throws (a write already sent is applied) and the main loop ends. The fake fails the calls of the thread of a closed group, and the world offers the closed group a refresh (the public `clearAndRefresh`), after which its loop ends. A close therefore waits the 5 seconds of its `awaitTermination` in virtual time (a real close waits only while the thread is busy), so the znode of a closed group is deleted 5 seconds after the close began, and a peer takes over that much later. As the factory is a bean, the default path (`DefaultGroupFactoryStrategy` and `DefaultManagedGroupFactory`) does not run; the bean is also what lets the world check that a consumer or a policy does not close a group factory it did not create (CAMEL-26887). The seed picks the mode, the session timeout, the retry policy of the client, how long a start of the consumer takes, the fault rates, and the operations: connection losses shorter or longer than the session timeout, other nodes that join, leave or die, route stops and starts (master consumer), routes removed and added again and routes started from outside (route policy). Invariants: this node runs its consumer only while it is the master (allowing 2 seconds for the events to arrive, and the end of a start that was in flight); once the faults cleared, the master runs its consumer, and this node does when it is the master; a stopped consumer or a removed route leaves no listener on the shared CuratorFramework, and no znode of this node is left in the group after the final stop; the shared CuratorFramework and group factory are not closed by a consumer or a policy; the client is used as ZooKeeper and Curator allow. Limits, written in the world's Javadoc: one master endpoint or policy per node (each `ZooKeeperGroup` main loop runs for ever, and on the one stack of the harness a second loop nested above the first would never let it resume); the operator does not stop or remove the route while the thread of the group waits in a call of ZooKeeper other than a create (a read, a setData, a delete, the ensure of the path) (a real close waits for the interrupted thread, which records the znode it made first; on the one stack that thread can only end after the close returned; a stop while the thread starts the consumer is driven); the route policy starts its consumer at once, as it starts it while holding its leadership lock, and locks do not exclude on the one stack. Not explored: route suspend and resume, the CuratorFramework the component creates itself, a stop of the policy on its own (from JMX), the camel-master cluster services. Three cases are left out of the default set while they are looked at separately, each added back by a system property: a route stopped while the client is disconnected (`-Ddst.zookeeper.stopWhileSuspended=true`), a route stopped while the group waits for the answer of a create (`-Ddst.zookeeper.stopDuringCreate=true`), and a route of the policy started from outside when it is started already (`-Ddst.zookeeper.startStartedRoute=true`). Other changes: - `camel-test-dst` as a test dependency of camel-zookeeper-master, and its byte code agent on the surefire command line, as for the other worlds. - The test log of camel-zookeeper-master goes to a size-capped rolling file (50 MB, one backup), as a failing world can log a lot. - The `@DstWorld` annotation lists the world in the component JSON (and its catalog copy) and in the table of the camel-test-dst page (and its catalog copy, byte-identical). Tests: - On main: 5000 of 5000 seeds pass (about 9 ms a seed). Seeds 8 and 77 replayed in two JVMs give the same trace hashes. - Self-check against fixed bugs, each on the code from before its fix (the runs stop at 100 failing seeds; run with the same world code before the rebase onto current main, which only brought the opt-in wall clock jumps to camel-test-dst): - before CAMEL-25315 (`MasterConsumer` of e670025a49a4^, with the Jackson 3 imports of main): 100 of the first 829 seeds fail, all with the master consumer: the delegate consumer left running after the route stopped during its start, two delegate consumers at once, a failed start not retried, the delegate running while disconnected. - before CAMEL-26540 (`MasterRoutePolicy` of 706beec6d5ab^, which is also before CAMEL-26802): 100 of the first 206 seeds fail, all with the route policy. These are the same seeds that fail before CAMEL-26802, and 76 of them fail first on the same invariant there; in 15 of the other 24, all with failing starts, the master never runs its consumer once the faults cleared (a failed start is not retried), where the code before CAMEL-26802 alone fails on the listener (11) or on a slave running its consumer (4). - before CAMEL-26802 (136ffd929263 reverted): 100 of the first 206 seeds fail, all with the route policy: in 61 the listener is left on the shared client after the route was removed or stopped, in 26 the master never runs its consumer once the faults cleared (a removed route keeps the leadership or never starts again), and in 13 a node that is not the master runs it. - CAMEL-26870, with its change taken out of main by hand (the `unregisterAsListener()` of `ZookeeperGroupSupport.doStop` removed, CAMEL-26887 kept, as the two changed the same lines): 100 of the first 194 seeds fail, all with the master consumer, all on "a stopped consumer leaves no listener on the shared CuratorFramework" (the route policy removes the listener itself). - before CAMEL-26887 (28f3170dcc6a reverted): the first 100 of 100 seeds fail, in both modes, on "the shared CuratorFramework and group factory are not closed". - The cases left out of the default set, each with its property (these are being looked at separately): - `-Ddst.zookeeper.stopWhileSuspended=true`: 7 of 5000 seeds fail. A group closed while the client is disconnected does not delete its znode; after the reconnect to the same session the znode stays in the group, as the master or ahead of this node's new znode. - `-Ddst.zookeeper.stopDuringCreate=true`: 100 of the first 1189 seeds fail. A create that the close interrupts is applied, but the group never records the znode, which stays in the group as long as the session of the shared client lives. - `-Ddst.zookeeper.startStartedRoute=true`: 8 of 2000 seeds fail, all with the route policy. A route that is started already and is started again (from JMX, say) starts its consumer again without the `onStart` of the policy, so a node that is not the master runs it. - camel-zookeeper-master unit tests: 1040 run (40, plus 1000 seeds), 0 failures, 0 errors, 0 skipped (the ITs need Docker and were not run). What was not run locally: the ITs (they need Docker) and the full root build. I built and tested offline with the versions cached on my machine: commons-lang3 3.21.0 and Guava 33.7.2 (main has 3.22.0 and 33.8.0-jre). Curator 5.9.0 and ZooKeeper 3.9.7 are the versions of main. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26974](https://issues.apache.org/jira/browse/CAMEL-26974). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-test-dst and camel-zookeeper-master, including the formatter and import-sort plugins. The generated files touched are the component JSON with its catalog copy, and the table of the camel-test-dst page with its catalog copy (byte-identical). I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
