[ 
https://issues.apache.org/jira/browse/CAMEL-26992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen updated CAMEL-26992:
--------------------------------
    Fix Version/s: 4.24.0

> camel-mongodb - a change stream that the consumer opens again before any 
> exchange completed starts at the current time and skips the changes made in 
> between
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-26992
>                 URL: https://issues.apache.org/jira/browse/CAMEL-26992
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-mongodb
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> Found by the MongoDB change streams DST world of CAMEL-26973 (branch 
> {{camel-mongodb-dst-world}}), which leaves this case out of its default set: 
> with {{-Ddst.mongodb.strictFirstCursor=true}}, 50 of 2000 seeds (first 
> failing seed 12) end with changes of the collection that the route never got. 
> Reproduced outside the DST harness with a plain unit test (below).
> h3. What happens
> When the change streams consumer has no resume token (no 
> {{changeStreamToken}}, nothing in the {{changeStreamTokenRepository}}, no 
> resume strategy), it opens its change stream without a resume option, so the 
> stream starts at the current time. That is fine for the first cursor of a 
> route start. But {{MongoDbChangeStreamsThread}} also opens a new cursor 
> itself when the change stream fails, for example when a getMore fails and the 
> resume of the driver fails too because the server cannot be reached for 
> longer than the server selection timeout (the retry loop of CAMEL-25679). If 
> no exchange completed yet, the consumer still has no resume token, so that 
> new cursor also starts at the current time, and every change made since the 
> first cursor was opened is skipped. Once an exchange completed, the consumer 
> resumes after its token as expected.
> h3. Expected
> The consumer thread is still running, so a cursor it opens again should go on 
> where the previous one stopped, as it does once there is a token. The MongoDB 
> manual (Change Streams, "Resume a Change Stream") documents 
> {{startAtOperationTime}}: "You can specify a startAtOperationTime to open the 
> cursor at a particular point in time. If the specified starting point is in 
> the past, it must be in the time range of the oplog." The driver does this 
> itself for its own resume (change streams specification: the operationTime of 
> the initial aggregate is saved and used as {{startAtOperationTime}} when 
> there is no cached token), but the Java driver does not expose that time, and 
> {{MongoChangeStreamCursor.getResumeToken()}} is only set once a getMore 
> succeeded, which is exactly what fails here.
> h3. Reproduction
> New {{MongoDbChangeStreamsReopenTest}} (a mocked client whose change streams 
> answer from a list of changes): one change before the route starts, then the 
> route starts with no token; a and b are inserted, the first getMore fails 
> ({{MongoSocketReadException}}, the driver could not resume), and c is 
> inserted after the consumer opened a new cursor. On main the route gets 
> {{[c]}}: a and b are lost.
> h3. Fix
> Before it opens a change stream at the current time, the consumer sends a 
> {{ping}} (with the read preference of the collection, so it goes where the 
> change stream reads from) and keeps the {{operationTime}} of the reply (for 
> read concern local, "the timestamp of the most recent entry in the oplog", 
> {{db.runCommand}} reference), plus one increment. Until an exchange 
> completed, a cursor that the consumer thread opens again starts with 
> {{startAtOperationTime}} at that time, so it delivers the changes made since. 
> The time is not stored: a route started again without a token still starts at 
> the current time, as documented. If the oplog no longer goes back that far 
> (ChangeStreamHistoryLost 286, or ChangeStreamFatalError 280 or 40576 on 
> earlier 4.2 releases) the consumer logs a WARN and starts at the current time 
> as before. A server whose reply has no {{operationTime}} (a standalone 
> server, which has no change streams) or that answers the ping with an error 
> keeps the current behaviour. Branch 
> {{camel-mongodb-change-streams-reopen-before-first-exchange-v2}}.
> The DST world needs its fake to answer the {{ping}} and 
> {{startAtOperationTime}} once both are in; with that and the fix, 
> {{-Ddst.mongodb.strictFirstCursor=true}} passes 2000 of 2000 seeds and the 
> default set 5000 of 5000.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to