davsclaus opened a new pull request, #3204:
URL: https://github.com/apache/camel-kamelets/pull/3204

   Part of #3170: `mongodb-changes-stream-source` sends the updated document 
for an update, instead of the text `null`.
   
   ### What was missing
   
   MongoDB records an update as the fields that changed. The Kamelet sends only 
the document of a change, and its change stream did not ask MongoDB for the 
document of an update. So for an update the body was `null`, and only the `_id` 
header said which document changed: what was updated was lost.
   
   ### The option
   
   `fullDocument` is an enum, mapped to the `fullDocument` endpoint option of 
camel-mongodb as `fullDocument={{?fullDocument}}`. camel-mongodb passes it to 
the change stream (`ChangeStreamIterable.fullDocument`). It only changes the 
body of an update; inserts, replaces and deletes are sent the same with any 
value.
   
   | Value | Body for an update |
   |---|---|
   | `updateLookup` (**new default**) | The whole document, read again from the 
collection when the change is sent. It can already hold later changes, and it 
is `null` when the document was deleted meanwhile. |
   | `default` | `null`, as before. |
   | `whenAvailable` | The document as it was right after this update, on a 
collection with change stream post-images turned on 
(`changeStreamPreAndPostImages`); `null` on another collection. |
   | `required` | As `whenAvailable`, but on a collection without post-images 
the stream fails on the update, retries it every second and sends no later 
change. |
   
   camel-mongodb has no `fullDocumentBeforeChange` option, so the pre-image 
cannot be asked for.
   
   **Why `updateLookup` is the default.** The Kamelet's use cases (copy changes 
to another system, keep a cache or a search index up to date) need the 
document, and `null` is of no use: a consumer that wants the data has to read 
MongoDB again itself, which is what `updateLookup` does, only later. The costs, 
which the descriptions state:
   
   - one more read on the server for each update (by `_id`);
   - the document can be newer than the change, or `null` if deleted meanwhile;
   - an update whose document and changed fields are together larger than about 
16 MB cannot be sent: MongoDB fails with `BSONObjectTooLarge`, and the stream 
retries the update every second and sends no later change. With `default`, the 
same update passes. Such a collection must set `fullDocument: default`.
   
   `whenAvailable` and `required` need a setting on the collection, so they 
cannot be the default.
   
   **Behaviour change.** With the option unset:
   
   - an update now sends the document instead of `null`;
   - a `streamFilter` on a field under `fullDocument`, such as the example `{ 
'$match': { 'fullDocument.item': 'apple' } }`, now also matches updates. 
Before, it never matched an update, as an update had no `fullDocument`.
   
   `fullDocument: default` gives the old behaviour. Upgrade note for #3190: 
"`mongodb-changes-stream-source` now sends the whole document for an update, 
read again from the collection, instead of `null`. Set `fullDocument` to 
`default` for the old behaviour."
   
   **Docs.** The Output paragraph, "Good to know" and the `streamFilter` 
description say what an update sends. The partial page has a table of the 
values, the 16 MB limit, and a `whenAvailable` example with the `collMod` 
command that turns post-images on.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `mongo:8`, a one-member 
replica set (`--replSet rs0`, keyfile) with auth on, as `root` (via `hosts`) 
and as a user with only the `read` role on `shop` (via `connectionString` with 
`authSource=shop`). Each case: insert, update (`$set qty`), replace, delete, 
with the endpoint URI from `camel get endpoint --json`.
   
   | Case | Endpoint URI (local-mongodb) | Insert | Update | Replace | Delete |
   |---|---|---|---|---|---|
   | `main`, option unset | `...consumerType=changeStreams&database=shop` | 
document | `null` | document | `{"_id":1}` |
   | Branch, `fullDocument: default` | same `&fullDocument=default` | document 
| `null` | document | `{"_id":1}` |
   | Branch, option unset | same `&fullDocument=updateLookup` | document | 
`{"_id":1,"item":"apple","qty":2}` | document | `{"_id":1}` |
   | Branch, `updateLookup` explicitly | same `&fullDocument=updateLookup` | 
document | `{"_id":1,"item":"apple","qty":2}` | document | `{"_id":1}` |
   | Branch, option unset, user with only `read` on `shop` | 
`connectionUriString=RAW(...)`, `&fullDocument=updateLookup` | document | 
document | document | `{"_id":1}` |
   | `whenAvailable`, no post-images | `&fullDocument=whenAvailable` | document 
| `null` | document | `{"_id":1}` |
   | `required`, no post-images | `&fullDocument=required` | document | not 
sent: `NoMatchingDocument ... require a post-image`, retried every second (22 
times in the run); replace and delete never sent | | |
   
   Three quick updates (`qty` 2, 3, 4), then a delete one second later:
   
   | Case | The three updates |
   |---|---|
   | `main`, option unset | `null`, `null`, `null` |
   | Branch, option unset (`updateLookup`) | `qty` 4, 4, 4: the document at 
lookup time |
   | `updateLookup`, post-images on | `qty` 4, 4, 4 |
   | `whenAvailable`, post-images on | `qty` 2, 3, 4: the document right after 
each update |
   | `required`, post-images on | `qty` 2, 3, 4 |
   
   Other cases:
   
   | Case | Result |
   |---|---|
   | `updateLookup`, three quick updates and a delete with no pause | All three 
updates are `null`: the document was gone at lookup time |
   | `streamFilter` on `fullDocument.item: apple`, `fullDocument: default` (as 
on `main`) | Only the insert is sent |
   | The same with `updateLookup` | The insert and the update are sent |
   | 14 MB document, update adding a 1.9 MB field, `default` | Update sent as 
`null`, later changes sent |
   | The same with `updateLookup` | `BSONObjectTooLarge` (`Size 18571726 
exceeds maximum 16793600`), retried every second (23 times in the run), the 
next insert never sent |
   | `fullDocument: foo` | Fails at startup: `Enum class ... FullDocument does 
not have any constant with value: foo` |
   | Post-images on, two updates | Two more documents in 
`config.system.preimages` |
   
   `./mvnw clean verify -DskipTests` and `./mvnw clean verify` passed. The 
Citrus itests were not run locally; there is none for this Kamelet.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to