allthingssecurity opened a new pull request, #30135: URL: https://github.com/apache/camel/pull/30135
# Description [CAMEL-26973](https://issues.apache.org/jira/browse/CAMEL-26973), part of CAMEL-26610. One difference from the CAMEL-26610 list: CAMEL-25711 and CAMEL-25607 are fixes of `camel-mongodb-gridfs`, a separate module, so the self-check uses the two change stream fixes of camel-mongodb, CAMEL-25946 and CAMEL-25679. Two more points up front: the consumer's own graceful stop is not run (its stop waits for its thread, which the one stack of the harness cannot run), so a route restart interrupts the consumer thread, as a forced shutdown does, and then stops and starts the route; and one case is left out of the default set while it is looked at separately (a cursor opened again before any exchange completed, below). A deterministic simulation test (camel-test-dst) of the camel-mongodb change streams consumer, in the `dst` package of its tests, in the same form as the SQS world: `MongoDbChangeStreamsDstTest` (1000 seeds), `MongoDbChangeStreamsWorld` and `FakeMongo`. Test only; no main code changes. What the world models: the real consumer (`MongoDbChangeStreamsConsumer`, `MongoDbChangeStreamsThread` and its loop, the commit manager of the resume token) and Camel's routing run on the virtual-time loop; only MongoDB is faked, behind the `MongoClient` of the `connectionBean` option (dynamic proxies of the driver interfaces, and the driver's own `ChangeStreamDocument`). The fake is written from the MongoDB manual on change streams and the Java driver, and cites them: the events of the collection in order with their resume tokens; a change stream opened at the current time, after a token (`resumeAfter`) or after an invalidate event (`startAfter`, while `resumeAfter` of an invalidate token fails with code 260); a dropped collection, whose change streams get a drop and an invalidate event and then end, made again by the next insert; a server that cannot be reached for a while, so that a new cursor cannot be opened and a getMore fails, while the driver resumes the change stream on t hat resumable error (the driver's resume of the change stream is outside its retry, so an error of the resume reaches the consumer); and the `fullDocument` option. The resume token is stored in a durable `changeStreamTokenRepository` (it keeps its state when the consumer stops, as a database table would). The seed picks `cursorRegenerationDelay`, `fullDocument`, the server selection timeout and the fault rates, and injects collection drops, unreachable periods shorter and longer than the server selection timeout, failed getMores, failed and slow processing, and route stops and starts. Invariants: the route starts once the collection exists and the server can be reached; the consumer thread runs as long as the route is started; a resume token is stored only after its exchange completed; a stored token never passes an event that was not routed; once the faults cleared, every event of the collection after the first change stream is routed, also after a drop, a restart of the route, a failed getMore or a cursor that could not be opened; the consumer uses the client as documented. Events may be routed more than once, and an event whose exchange failed is not routed again when a later one completed, as the consumer stores the token of the later one (as its code says). One case is left out of the default set while it is looked at separately: a cursor that the consumer thread opens again before any exchange of the route completed (say the first change stream failed and the driver could not resume it) starts at the current time, as the consumer has no token yet, so the events written since the first change stream are skipped. By default the world does not count these events; `-Ddst.mongodb.strictFirstCursor=true` counts them (50 of 2000 seeds fail then). Limits, written in the world's Javadoc: the consumer thread loops for ever and its stop waits for it, which the one stack of the harness cannot run (the stop would wait above the thread it waits for). A route restart is therefore made in two steps: the consumer thread is interrupted at its next call of the change stream (as a forced shutdown of its executor does; the driver throws `MongoInterruptedException` and keeps the interrupt flag, and the thread ends), then the route is stopped and started. The phases of the run are in the test body, so that the interrupt flag of the one thread of the harness is cleared as soon as the consumer thread ended (no other task of the world blocks, so the condition of the test body runs right after the consumer task returned). The operator starts the route only once the collection exists and the server can be reached, and does not drop the collection or make the server unreachable while the route starts, so a start that fails is the consumer's. No t explored: the graceful stop of the consumer (`keepRunning` set to false and the cursor closed under a waiting getMore or an exchange in flight), tailable cursors, `streamFilter`, the `ResumeStrategy` of the consumer, a change stream without a token repository after a restart (it starts at the current time, by design), and camel-mongodb-gridfs. Other changes: - `camel-test-dst` as a test dependency of camel-mongodb, and its byte code agent on the surefire command line, as for the other worlds. - The test log of camel-mongodb goes to a size-capped rolling file (50 MB, one backup), as a failing world can log a lot. - The `@DstWorld` annotation lists the world in the component JSON (and its catalog copy) and in the table of the camel-test-dst page (and its catalog copy, byte-identical). Tests: - On main: 5000 of 5000 seeds pass (about 8 ms a seed). Seeds 8 and 77 replayed in two JVMs give the same trace hashes. - Self-check against fixed bugs, each on the code from before its fix: - before CAMEL-25946 (5f10c225652e reverted): 100 of the first 256 seeds fail (the runs stop at 100 failing seeds). After a drop, `resumeAfter` of the invalidate token fails: 56 never read the collection made again ("every event is routed once the faults cleared"), and in 44 the route cannot start again ("the route starts once the collection exists and the server can be reached"), as the stored token is the one of the invalidate event. - before CAMEL-25679 (its change to `MongoAbstractConsumerThread.run` reverted by hand, as a later logging change touches the same lines): 100 of the first 1943 seeds fail, all on "the consumer thread runs as long as the route is started": a cursor that cannot be opened while the server is unreachable ends the consumer thread. - camel-mongodb unit tests: 1028 run (28, plus 1000 seeds), 0 failures, 0 errors, 0 skipped (the ITs need Docker and were not run). What was not run locally: the ITs (they need Docker) and the full root build. I built and tested offline with the versions cached on my machine: the MongoDB Java driver 5.13.0 (main has 5.14.0, which is not cached, so the world was not compiled against it), commons-lang3 3.21.0 and Guava 33.7.2 (main has 3.22.0 and 33.8.0-jre). # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26973](https://issues.apache.org/jira/browse/CAMEL-26973). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-test-dst and camel-mongodb, including the formatter and import-sort plugins. The generated files touched are the component JSON with its catalog copy, and the table of the camel-test-dst page with its catalog copy (byte-identical). I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) The fake also answers the `ping` command and `startAtOperationTime` that the fix of CAMEL-26992 uses, so the world keeps passing once that fix is in (test-only; on main the consumer does not send them). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
