allthingssecurity opened a new pull request, #30136:
URL: https://github.com/apache/camel/pull/30136

   # Description
   
   [CAMEL-26992](https://issues.apache.org/jira/browse/CAMEL-26992)
   
   Found by the MongoDB change streams DST world of CAMEL-26973, which leaves 
this case out of its default set: with `-Ddst.mongodb.strictFirstCursor=true`, 
50 of 2000 seeds (first failing seed 12) end with changes the route never got. 
Reproduced here with a plain unit test, outside the DST harness.
   
   Without a resume token (no `changeStreamToken`, nothing in the 
`changeStreamTokenRepository`, no resume strategy) the change streams consumer 
opens its change stream without a resume option, so it starts at the current 
time. That is right for the first cursor of a route start. But 
`MongoDbChangeStreamsThread` also opens a new cursor itself when the change 
stream fails, for example when a getMore fails and the resume of the driver 
fails too because the server cannot be reached for longer than the server 
selection timeout. If no exchange completed yet there is still no resume token, 
so that cursor also started at the current time, and every change made since 
the first cursor was opened was skipped. Once an exchange completed, the 
consumer already resumes after its token.
   
   The MongoDB manual ("Resume a Change Stream") documents 
`startAtOperationTime`: "You can specify a startAtOperationTime to open the 
cursor at a particular point in time. If the specified starting point is in the 
past, it must be in the time range of the oplog." The driver does this for its 
own resume (change streams specification: it saves the `operationTime` of the 
initial aggregate and uses it as `startAtOperationTime` while it has no cached 
token), but the Java driver does not expose that time. 
`MongoChangeStreamCursor.getResumeToken()` is no help either: it is only set 
once a getMore succeeded (`MongoChangeStreamCursorImpl`, driver 5.13), and a 
failing first getMore is exactly the case here. I tried that first; with it the 
world still failed 42 of the 50 seeds, all on a first getMore that failed.
   
   Change, in `MongoDbChangeStreamsThread`: before it opens a change stream at 
the current time, the consumer sends a `ping`, with the read preference of the 
collection so it goes where the change stream reads from (`runCommand` alone 
always selects the primary), and keeps the `operationTime` of the reply plus 
one increment (for read concern local, the operation time of a command that 
writes nothing is "the timestamp of the most recent entry in the oplog", 
`db.runCommand` reference; through mongos it is the cluster time). A change 
stream includes the changes at its `startAtOperationTime`, and one opened 
without a start option starts after the newest oplog entry, so when the ping 
and the change stream go to the same member the kept time is not later than 
where the first change stream started; a change made between the ping and that 
opening can be delivered by a cursor opened again, which is a change after the 
route started. With a secondary read preference they can go to different mem
 bers, and a cursor opened again then starts at most at a change made before 
the ping. Until an exchange completed, a cursor that the consumer thread opens 
again starts with `startAtOperationTime` at that time, so it delivers the 
changes made since. After an exchange completed nothing changes: the consumer 
resumes after the token as before.
   
   - The time is not stored, so a route started again without a token still 
starts at the current time.
   - Changes that were routed but failed before the cursor broke are delivered 
again by the new cursor, as they already are when it resumes after a token.
   - If the oplog no longer goes back that far (ChangeStreamHistoryLost 286 on 
MongoDB 4.4 and the later 4.2 releases, ChangeStreamFatalError 280 or 40576 on 
earlier 4.2 releases; driver 5 needs 4.2 or later), the consumer logs a WARN 
and starts at the current time, as it did before. Without this, a route that 
sees no change for longer than the oplog window and then loses its change 
stream would retry the old time forever. Any other error of that open is 
retried after `cursorRegenerationDelay` as before.
   - A reply without `operationTime` (a standalone server, which has no change 
streams) or a ping that the server answers with an error 
(`MongoServerException`, a member shutting down for example) keeps the current 
behaviour. No server to send the ping to is not caught: with the same read 
preference, opening the change stream would fail the same way.
   - The time and the token are only used by the consumer thread (the first 
cursor is opened in `doStart` before the thread is handed to the executor), so 
there is no new shared state.
   
   Cost: one `ping` before each change stream opened at the current time, which 
is once per route start without a token (and again after the WARN above). 
Reopening from the start time reads the oplog from that time, as reopening from 
an old token already does. With a selective `streamFilter` or a quiet 
collection, no exchange may complete for a long time, so the start time stays 
at the route start; once it is older than the oplog window a reopen gets the 
WARN, which then may name changes that did not exist (nothing matched).
   
   Not changed, noted while looking at this: a stored token older than the 
oplog window makes `resumeAfter` fail with 286 (or 280) on every attempt, so 
the consumer retries it for as long as the route runs. That is on main already 
and a different decision (falling back to the current time would skip changes 
silently), so I would rather open a follow-up JIRA for it.
   
   Tests, new `MongoDbChangeStreamsReopenTest` (a mocked client whose change 
streams answer from a list of changes, with `resumeAfter`, 
`startAtOperationTime`, ChangeStreamHistoryLost and the `operationTime` of the 
ping reply):
   
   - `cursorOpenedAgainBeforeAnyExchangeCompletedStartsWhereTheFirstStarted`: 
one change before the route starts, a and b inserted, the first getMore fails 
(`MongoSocketReadException`), c inserted after the new cursor was opened. The 
route must get a, b and c (on main it gets `[c]`), and once c completed the 
next cursor resumes after its token.
     The collection reads from secondaries (`secondaryPreferred`) and the fake 
answers only a ping sent with that read preference.
   - `startNoLongerInTheOplogStartsAtTheCurrentTime` (286), 
`startNoLongerInTheOplogOfMongoDb42StartsAtTheCurrentTime` (280) and 
`startNoLongerInTheOplogOfMongoDb420StartsAtTheCurrentTime` (40576): the oplog 
no longer has the start time, so the consumer logs the WARN, opens at the 
current time and goes on (on main it opened at the current time without trying).
   - `serverWithoutOperationTimeStartsAtTheCurrentTime` and 
`pingAnsweredWithAnErrorStartsAtTheCurrentTime` (ShutdownInProgress): controls, 
the behaviour before the change; they pass on main too.
   
   Without the change the first four fail; with it all six pass, and the 
camel-mongodb unit tests pass (34 run, 0 failures). With the change merged 
locally into the world of CAMEL-26973, `-Ddst.mongodb.strictFirstCursor=true` 
passes 2000 of 2000 seeds (50 fail without it) and the default set 5000 of 
5000. The world's fake needed two more answers for that, the `ping` reply and 
`startAtOperationTime`, which go into the world branch. The ITs need a MongoDB 
container and were not run locally (CI runs them). The local offline build used 
the cached MongoDB driver 5.13.0.
   
   No documentation or upgrade guide change: a consumer that loses its change 
stream before the first exchange completed now goes on where it started, as it 
already does after the first exchange.
   
   Backport: camel-4.18.x opens a new cursor the same way and has the same gap, 
but CAMEL-25946 and CAMEL-25679 reshaped this code on main only, so the change 
does not cherry-pick; a backport would be a small rewrite of `initializeCursor` 
there.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-26992](https://issues.apache.org/jira/browse/CAMEL-26992).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built camel-mongodb and the core modules with the formatter and 
import-sort plugins and ran the camel-mongodb unit tests. I did not run the 
full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to