[
https://issues.apache.org/jira/browse/CAMEL-26992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-26992.
---------------------------------
Resolution: Fixed
Fixed by https://github.com/apache/camel/pull/30136 (merged to main as
1eb59175e600).
_Claude Code on behalf of davsclaus_
> camel-mongodb - a change stream that the consumer opens again before any
> exchange completed starts at the current time and skips the changes made in
> between
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26992
> URL: https://issues.apache.org/jira/browse/CAMEL-26992
> Project: Camel
> Issue Type: Bug
> Components: camel-mongodb
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> Found by the MongoDB change streams DST world of CAMEL-26973 (branch
> {{camel-mongodb-dst-world}}), which leaves this case out of its default set:
> with {{-Ddst.mongodb.strictFirstCursor=true}}, 50 of 2000 seeds (first
> failing seed 12) end with changes of the collection that the route never got.
> Reproduced outside the DST harness with a plain unit test (below).
> h3. What happens
> When the change streams consumer has no resume token (no
> {{changeStreamToken}}, nothing in the {{changeStreamTokenRepository}}, no
> resume strategy), it opens its change stream without a resume option, so the
> stream starts at the current time. That is fine for the first cursor of a
> route start. But {{MongoDbChangeStreamsThread}} also opens a new cursor
> itself when the change stream fails, for example when a getMore fails and the
> resume of the driver fails too because the server cannot be reached for
> longer than the server selection timeout (the retry loop of CAMEL-25679). If
> no exchange completed yet, the consumer still has no resume token, so that
> new cursor also starts at the current time, and every change made since the
> first cursor was opened is skipped. Once an exchange completed, the consumer
> resumes after its token as expected.
> h3. Expected
> The consumer thread is still running, so a cursor it opens again should go on
> where the previous one stopped, as it does once there is a token. The MongoDB
> manual (Change Streams, "Resume a Change Stream") documents
> {{startAtOperationTime}}: "You can specify a startAtOperationTime to open the
> cursor at a particular point in time. If the specified starting point is in
> the past, it must be in the time range of the oplog." The driver does this
> itself for its own resume (change streams specification: the operationTime of
> the initial aggregate is saved and used as {{startAtOperationTime}} when
> there is no cached token), but the Java driver does not expose that time, and
> {{MongoChangeStreamCursor.getResumeToken()}} is only set once a getMore
> succeeded, which is exactly what fails here.
> h3. Reproduction
> New {{MongoDbChangeStreamsReopenTest}} (a mocked client whose change streams
> answer from a list of changes): one change before the route starts, then the
> route starts with no token; a and b are inserted, the first getMore fails
> ({{MongoSocketReadException}}, the driver could not resume), and c is
> inserted after the consumer opened a new cursor. On main the route gets
> {{[c]}}: a and b are lost.
> h3. Fix
> Before it opens a change stream at the current time, the consumer sends a
> {{ping}} (with the read preference of the collection, so it goes where the
> change stream reads from) and keeps the {{operationTime}} of the reply (for
> read concern local, "the timestamp of the most recent entry in the oplog",
> {{db.runCommand}} reference), plus one increment. Until an exchange
> completed, a cursor that the consumer thread opens again starts with
> {{startAtOperationTime}} at that time, so it delivers the changes made since.
> The time is not stored: a route started again without a token still starts at
> the current time, as documented. If the oplog no longer goes back that far
> (ChangeStreamHistoryLost 286, or ChangeStreamFatalError 280 or 40576 on
> earlier 4.2 releases) the consumer logs a WARN and starts at the current time
> as before. A server whose reply has no {{operationTime}} (a standalone
> server, which has no change streams) or that answers the ping with an error
> keeps the current behaviour. Branch
> {{camel-mongodb-change-streams-reopen-before-first-exchange-v2}}.
> The DST world needs its fake to answer the {{ping}} and
> {{startAtOperationTime}} once both are in; with that and the fix,
> {{-Ddst.mongodb.strictFirstCursor=true}} passes 2000 of 2000 seeds and the
> default set 5000 of 5000.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)