allthingssecurity opened a new pull request, #30136: URL: https://github.com/apache/camel/pull/30136
# Description [CAMEL-26992](https://issues.apache.org/jira/browse/CAMEL-26992) Found by the MongoDB change streams DST world of CAMEL-26973, which leaves this case out of its default set: with `-Ddst.mongodb.strictFirstCursor=true`, 50 of 2000 seeds (first failing seed 12) end with changes the route never got. Reproduced here with a plain unit test, outside the DST harness. Without a resume token (no `changeStreamToken`, nothing in the `changeStreamTokenRepository`, no resume strategy) the change streams consumer opens its change stream without a resume option, so it starts at the current time. That is right for the first cursor of a route start. But `MongoDbChangeStreamsThread` also opens a new cursor itself when the change stream fails, for example when a getMore fails and the resume of the driver fails too because the server cannot be reached for longer than the server selection timeout. If no exchange completed yet there is still no resume token, so that cursor also started at the current time, and every change made since the first cursor was opened was skipped. Once an exchange completed, the consumer already resumes after its token. The MongoDB manual ("Resume a Change Stream") documents `startAtOperationTime`: "You can specify a startAtOperationTime to open the cursor at a particular point in time. If the specified starting point is in the past, it must be in the time range of the oplog." The driver does this for its own resume (change streams specification: it saves the `operationTime` of the initial aggregate and uses it as `startAtOperationTime` while it has no cached token), but the Java driver does not expose that time. `MongoChangeStreamCursor.getResumeToken()` is no help either: it is only set once a getMore succeeded (`MongoChangeStreamCursorImpl`, driver 5.13), and a failing first getMore is exactly the case here. I tried that first; with it the world still failed 42 of the 50 seeds, all on a first getMore that failed. Change, in `MongoDbChangeStreamsThread`: before it opens a change stream at the current time, the consumer sends a `ping`, with the read preference of the collection so it goes where the change stream reads from (`runCommand` alone always selects the primary), and keeps the `operationTime` of the reply plus one increment (for read concern local, the operation time of a command that writes nothing is "the timestamp of the most recent entry in the oplog", `db.runCommand` reference; through mongos it is the cluster time). A change stream includes the changes at its `startAtOperationTime`, and one opened without a start option starts after the newest oplog entry, so when the ping and the change stream go to the same member the kept time is not later than where the first change stream started; a change made between the ping and that opening can be delivered by a cursor opened again, which is a change after the route started. With a secondary read preference they can go to different mem bers, and a cursor opened again then starts at most at a change made before the ping. Until an exchange completed, a cursor that the consumer thread opens again starts with `startAtOperationTime` at that time, so it delivers the changes made since. After an exchange completed nothing changes: the consumer resumes after the token as before. - The time is not stored, so a route started again without a token still starts at the current time. - Changes that were routed but failed before the cursor broke are delivered again by the new cursor, as they already are when it resumes after a token. - If the oplog no longer goes back that far (ChangeStreamHistoryLost 286 on MongoDB 4.4 and the later 4.2 releases, ChangeStreamFatalError 280 or 40576 on earlier 4.2 releases; driver 5 needs 4.2 or later), the consumer logs a WARN and starts at the current time, as it did before. Without this, a route that sees no change for longer than the oplog window and then loses its change stream would retry the old time forever. Any other error of that open is retried after `cursorRegenerationDelay` as before. - A reply without `operationTime` (a standalone server, which has no change streams) or a ping that the server answers with an error (`MongoServerException`, a member shutting down for example) keeps the current behaviour. No server to send the ping to is not caught: with the same read preference, opening the change stream would fail the same way. - The time and the token are only used by the consumer thread (the first cursor is opened in `doStart` before the thread is handed to the executor), so there is no new shared state. Cost: one `ping` before each change stream opened at the current time, which is once per route start without a token (and again after the WARN above). Reopening from the start time reads the oplog from that time, as reopening from an old token already does. With a selective `streamFilter` or a quiet collection, no exchange may complete for a long time, so the start time stays at the route start; once it is older than the oplog window a reopen gets the WARN, which then may name changes that did not exist (nothing matched). Not changed, noted while looking at this: a stored token older than the oplog window makes `resumeAfter` fail with 286 (or 280) on every attempt, so the consumer retries it for as long as the route runs. That is on main already and a different decision (falling back to the current time would skip changes silently), so I would rather open a follow-up JIRA for it. Tests, new `MongoDbChangeStreamsReopenTest` (a mocked client whose change streams answer from a list of changes, with `resumeAfter`, `startAtOperationTime`, ChangeStreamHistoryLost and the `operationTime` of the ping reply): - `cursorOpenedAgainBeforeAnyExchangeCompletedStartsWhereTheFirstStarted`: one change before the route starts, a and b inserted, the first getMore fails (`MongoSocketReadException`), c inserted after the new cursor was opened. The route must get a, b and c (on main it gets `[c]`), and once c completed the next cursor resumes after its token. The collection reads from secondaries (`secondaryPreferred`) and the fake answers only a ping sent with that read preference. - `startNoLongerInTheOplogStartsAtTheCurrentTime` (286), `startNoLongerInTheOplogOfMongoDb42StartsAtTheCurrentTime` (280) and `startNoLongerInTheOplogOfMongoDb420StartsAtTheCurrentTime` (40576): the oplog no longer has the start time, so the consumer logs the WARN, opens at the current time and goes on (on main it opened at the current time without trying). - `serverWithoutOperationTimeStartsAtTheCurrentTime` and `pingAnsweredWithAnErrorStartsAtTheCurrentTime` (ShutdownInProgress): controls, the behaviour before the change; they pass on main too. Without the change the first four fail; with it all six pass, and the camel-mongodb unit tests pass (34 run, 0 failures). With the change merged locally into the world of CAMEL-26973, `-Ddst.mongodb.strictFirstCursor=true` passes 2000 of 2000 seeds (50 fail without it) and the default set 5000 of 5000. The world's fake needed two more answers for that, the `ping` reply and `startAtOperationTime`, which go into the world branch. The ITs need a MongoDB container and were not run locally (CI runs them). The local offline build used the cached MongoDB driver 5.13.0. No documentation or upgrade guide change: a consumer that loses its change stream before the first exchange completed now goes on where it started, as it already does after the first exchange. Backport: camel-4.18.x opens a new cursor the same way and has the same gap, but CAMEL-25946 and CAMEL-25679 reshaped this code on main only, so the change does not cherry-pick; a backport would be a small rewrite of `initializeCursor` there. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26992](https://issues.apache.org/jira/browse/CAMEL-26992). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-mongodb and the core modules with the formatter and import-sort plugins and ran the camel-mongodb unit tests. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
