shashank created CAMEL-26992:
--------------------------------
Summary: camel-mongodb - a change stream that the consumer opens
again before any exchange completed starts at the current time and skips the
changes made in between
Key: CAMEL-26992
URL: https://issues.apache.org/jira/browse/CAMEL-26992
Project: Camel
Issue Type: Bug
Components: camel-mongodb
Reporter: shashank
Assignee: shashank
Found by the MongoDB change streams DST world of CAMEL-26973 (branch
{{camel-mongodb-dst-world}}), which leaves this case out of its default set:
with {{-Ddst.mongodb.strictFirstCursor=true}}, 50 of 2000 seeds (first failing
seed 12) end with changes of the collection that the route never got.
Reproduced outside the DST harness with a plain unit test (below).
h3. What happens
When the change streams consumer has no resume token (no {{changeStreamToken}},
nothing in the {{changeStreamTokenRepository}}, no resume strategy), it opens
its change stream without a resume option, so the stream starts at the current
time. That is fine for the first cursor of a route start. But
{{MongoDbChangeStreamsThread}} also opens a new cursor itself when the change
stream fails, for example when a getMore fails and the resume of the driver
fails too because the server cannot be reached for longer than the server
selection timeout (the retry loop of CAMEL-25679). If no exchange completed
yet, the consumer still has no resume token, so that new cursor also starts at
the current time, and every change made since the first cursor was opened is
skipped. Once an exchange completed, the consumer resumes after its token as
expected.
h3. Expected
The consumer thread is still running, so a cursor it opens again should go on
where the previous one stopped, as it does once there is a token. The MongoDB
manual (Change Streams, "Resume a Change Stream") documents
{{startAtOperationTime}}: "You can specify a startAtOperationTime to open the
cursor at a particular point in time. If the specified starting point is in the
past, it must be in the time range of the oplog." The driver does this itself
for its own resume (change streams specification: the operationTime of the
initial aggregate is saved and used as {{startAtOperationTime}} when there is
no cached token), but the Java driver does not expose that time, and
{{MongoChangeStreamCursor.getResumeToken()}} is only set once a getMore
succeeded, which is exactly what fails here.
h3. Reproduction
New {{MongoDbChangeStreamsReopenTest}} (a mocked client whose change streams
answer from a list of changes): one change before the route starts, then the
route starts with no token; a and b are inserted, the first getMore fails
({{MongoSocketReadException}}, the driver could not resume), and c is inserted
after the consumer opened a new cursor. On main the route gets {{[c]}}: a and b
are lost.
h3. Fix
Before it opens a change stream at the current time, the consumer sends a
{{ping}} (with the read preference of the collection, so it goes where the
change stream reads from) and keeps the {{operationTime}} of the reply (for
read concern local, "the timestamp of the most recent entry in the oplog",
{{db.runCommand}} reference), plus one increment. Until an exchange completed,
a cursor that the consumer thread opens again starts with
{{startAtOperationTime}} at that time, so it delivers the changes made since.
The time is not stored: a route started again without a token still starts at
the current time, as documented. If the oplog no longer goes back that far
(ChangeStreamHistoryLost 286, or ChangeStreamFatalError 280 or 40576 on earlier
4.2 releases) the consumer logs a WARN and starts at the current time as
before. A server whose reply has no {{operationTime}} (a standalone server,
which has no change streams) or that answers the ping with an error keeps the
current behaviour. Branch
{{camel-mongodb-change-streams-reopen-before-first-exchange-v2}}.
The DST world needs its fake to answer the {{ping}} and
{{startAtOperationTime}} once both are in; with that and the fix,
{{-Ddst.mongodb.strictFirstCursor=true}} passes 2000 of 2000 seeds and the
default set 5000 of 5000.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)