allthingssecurity opened a new pull request, #30135:
URL: https://github.com/apache/camel/pull/30135

   # Description
   
   [CAMEL-26973](https://issues.apache.org/jira/browse/CAMEL-26973), part of 
CAMEL-26610. One difference from the CAMEL-26610 list: CAMEL-25711 and 
CAMEL-25607 are fixes of `camel-mongodb-gridfs`, a separate module, so the 
self-check uses the two change stream fixes of camel-mongodb, CAMEL-25946 and 
CAMEL-25679. Two more points up front: the consumer's own graceful stop is not 
run (its stop waits for its thread, which the one stack of the harness cannot 
run), so a route restart interrupts the consumer thread, as a forced shutdown 
does, and then stops and starts the route; and one case is left out of the 
default set while it is looked at separately (a cursor opened again before any 
exchange completed, below).
   
   A deterministic simulation test (camel-test-dst) of the camel-mongodb change 
streams consumer, in the `dst` package of its tests, in the same form as the 
SQS world: `MongoDbChangeStreamsDstTest` (1000 seeds), 
`MongoDbChangeStreamsWorld` and `FakeMongo`. Test only; no main code changes.
   
   What the world models: the real consumer (`MongoDbChangeStreamsConsumer`, 
`MongoDbChangeStreamsThread` and its loop, the commit manager of the resume 
token) and Camel's routing run on the virtual-time loop; only MongoDB is faked, 
behind the `MongoClient` of the `connectionBean` option (dynamic proxies of the 
driver interfaces, and the driver's own `ChangeStreamDocument`). The fake is 
written from the MongoDB manual on change streams and the Java driver, and 
cites them: the events of the collection in order with their resume tokens; a 
change stream opened at the current time, after a token (`resumeAfter`) or 
after an invalidate event (`startAfter`, while `resumeAfter` of an invalidate 
token fails with code 260); a dropped collection, whose change streams get a 
drop and an invalidate event and then end, made again by the next insert; a 
server that cannot be reached for a while, so that a new cursor cannot be 
opened and a getMore fails, while the driver resumes the change stream on t
 hat resumable error (the driver's resume of the change stream is outside its 
retry, so an error of the resume reaches the consumer); and the `fullDocument` 
option. The resume token is stored in a durable `changeStreamTokenRepository` 
(it keeps its state when the consumer stops, as a database table would).
   
   The seed picks `cursorRegenerationDelay`, `fullDocument`, the server 
selection timeout and the fault rates, and injects collection drops, 
unreachable periods shorter and longer than the server selection timeout, 
failed getMores, failed and slow processing, and route stops and starts.
   
   Invariants: the route starts once the collection exists and the server can 
be reached; the consumer thread runs as long as the route is started; a resume 
token is stored only after its exchange completed; a stored token never passes 
an event that was not routed; once the faults cleared, every event of the 
collection after the first change stream is routed, also after a drop, a 
restart of the route, a failed getMore or a cursor that could not be opened; 
the consumer uses the client as documented. Events may be routed more than 
once, and an event whose exchange failed is not routed again when a later one 
completed, as the consumer stores the token of the later one (as its code says).
   
   One case is left out of the default set while it is looked at separately: a 
cursor that the consumer thread opens again before any exchange of the route 
completed (say the first change stream failed and the driver could not resume 
it) starts at the current time, as the consumer has no token yet, so the events 
written since the first change stream are skipped. By default the world does 
not count these events; `-Ddst.mongodb.strictFirstCursor=true` counts them (50 
of 2000 seeds fail then).
   
   Limits, written in the world's Javadoc: the consumer thread loops for ever 
and its stop waits for it, which the one stack of the harness cannot run (the 
stop would wait above the thread it waits for). A route restart is therefore 
made in two steps: the consumer thread is interrupted at its next call of the 
change stream (as a forced shutdown of its executor does; the driver throws 
`MongoInterruptedException` and keeps the interrupt flag, and the thread ends), 
then the route is stopped and started. The phases of the run are in the test 
body, so that the interrupt flag of the one thread of the harness is cleared as 
soon as the consumer thread ended (no other task of the world blocks, so the 
condition of the test body runs right after the consumer task returned). The 
operator starts the route only once the collection exists and the server can be 
reached, and does not drop the collection or make the server unreachable while 
the route starts, so a start that fails is the consumer's. No
 t explored: the graceful stop of the consumer (`keepRunning` set to false and 
the cursor closed under a waiting getMore or an exchange in flight), tailable 
cursors, `streamFilter`, the `ResumeStrategy` of the consumer, a change stream 
without a token repository after a restart (it starts at the current time, by 
design), and camel-mongodb-gridfs.
   
   Other changes:
   - `camel-test-dst` as a test dependency of camel-mongodb, and its byte code 
agent on the surefire command line, as for the other worlds.
   - The test log of camel-mongodb goes to a size-capped rolling file (50 MB, 
one backup), as a failing world can log a lot.
   - The `@DstWorld` annotation lists the world in the component JSON (and its 
catalog copy) and in the table of the camel-test-dst page (and its catalog 
copy, byte-identical).
   
   Tests:
   - On main: 5000 of 5000 seeds pass (about 8 ms a seed). Seeds 8 and 77 
replayed in two JVMs give the same trace hashes.
   - Self-check against fixed bugs, each on the code from before its fix:
     - before CAMEL-25946 (5f10c225652e reverted): 100 of the first 256 seeds 
fail (the runs stop at 100 failing seeds). After a drop, `resumeAfter` of the 
invalidate token fails: 56 never read the collection made again ("every event 
is routed once the faults cleared"), and in 44 the route cannot start again 
("the route starts once the collection exists and the server can be reached"), 
as the stored token is the one of the invalidate event.
     - before CAMEL-25679 (its change to `MongoAbstractConsumerThread.run` 
reverted by hand, as a later logging change touches the same lines): 100 of the 
first 1943 seeds fail, all on "the consumer thread runs as long as the route is 
started": a cursor that cannot be opened while the server is unreachable ends 
the consumer thread.
   - camel-mongodb unit tests: 1028 run (28, plus 1000 seeds), 0 failures, 0 
errors, 0 skipped (the ITs need Docker and were not run).
   
   What was not run locally: the ITs (they need Docker) and the full root 
build. I built and tested offline with the versions cached on my machine: the 
MongoDB Java driver 5.13.0 (main has 5.14.0, which is not cached, so the world 
was not compiled against it), commons-lang3 3.21.0 and Guava 33.7.2 (main has 
3.22.0 and 33.8.0-jre).
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-26973](https://issues.apache.org/jira/browse/CAMEL-26973).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built and tested camel-test-dst and camel-mongodb, including the 
formatter and import-sort plugins. The generated files touched are the 
component JSON with its catalog copy, and the table of the camel-test-dst page 
with its catalog copy (byte-identical). I did not run the full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   The fake also answers the `ping` command and `startAtOperationTime` that the 
fix of CAMEL-26992 uses, so the world keeps passing once that fix is in 
(test-only; on main the consumer does not send them).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to