davsclaus opened a new pull request, #3225:
URL: https://github.com/apache/camel-kamelets/pull/3225

   Part of #3170: `mongodb-sink` can save and update documents, with an upsert, 
and `mongodb-source` can name its tail tracking and store it in another 
database or collection. `mongodb-changes-stream-source` is not touched here: 
#3204 changes it.
   
   ### The problem
   
   - `mongodb-sink` only inserted. A record whose `_id` was already stored 
failed, and a field of a stored document, such as the stock of a product, could 
not be changed.
   - `mongodb-source` with `persistentTailTracking` stored its place in 
`camelTailTracking` of the database it reads, as one document with 
`persistentId: null`. Two sources that both stored their place there shared 
that one document. In a test with two sources on `main`, after a restart one 
read all its documents again; when the stored place came from the other source, 
a new document was never read. A user that may only read the database could not 
use `persistentTailTracking` at all: the Kamelet failed to start with `not 
authorized on shop to execute command { insert: "camelTailTracking" ...`.
   
   ### The options
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | sink | `operation` | `operation={{operation}}` (was `operation=insert`), 
enum `insert`, `save`, `update`, `default: insert` | `insert` as before. `save` 
replaces the document with the `_id` of the body as a whole, or inserts it. 
`update` changes the first document that matches a filter. |
   | sink | `upsert` | a step that sets the `CamelMongoDbUpsert` header to the 
value, only when `upsert` is set and `operation` is `update` | With `true`, a 
filter that matches no document inserts one, made of the filter and the 
changes, and the next message updates it. |
   | source | `persistentId` | `persistentId={{?persistentId}}` | The name the 
place is stored under. |
   | source | `tailTrackDb` | `tailTrackDb={{?tailTrackDb}}` | The database to 
store the place in, instead of `database`. |
   | source | `tailTrackCollection` | 
`tailTrackCollection={{?tailTrackCollection}}` | The collection to store the 
place in, instead of `camelTailTracking`. |
   
   - **Defaults:** `operation` has `default: insert`, so the endpoint URI with 
it unset is byte-identical to `main`'s. `upsert` and the three source options 
have no `default`; unset, the optional placeholders drop them.
   - **The body of `update`:** camel-mongodb takes the filter and the changes 
as a JSON array in text, `[{"sku":"A1"},{"$set":{"qty":3}}]`, or as a List of 
documents. It parses only text: a `byte[]` body, which is what many sources 
send, failed with `body is not a List of Document objects with size = 2`. For 
`update` only, the template turns a body that is not a List into text first 
(`convertBodyTo: java.lang.String`). `insert` and `save` are not changed.
   - **`upsert` as a header:** camel-mongodb has no endpoint option for upsert, 
only the `CamelMongoDbUpsert` header. It is set in a `choice` step, only when 
`upsert` is set and `operation` is `update`.
   - **No `upsert` operation:** an upsert of a whole document is `save`; an 
upsert of some fields is `update` with `upsert: true`. `update` changes one 
document; `updateMany` (the `CamelMongoDbMultiUpdate` header) is not offered.
   - **`pattern:`** on the three source options, as they go into the URI: 
`^[A-Za-z0-9_.:-]+$`, `^[A-Za-z0-9_-]+$` and `^[A-Za-z0-9_.-]+$`. 
`tailTrackField` (the name of the field that holds the place) is left out; it 
has no use beyond the other three.
   
   **No `readPreference` option.** The `readPreference` endpoint option of 
camel-mongodb is never applied: `MongoDbEndpoint.getReadPreferenceBean()` has 
no caller. With `readPreference=SECONDARY` on the endpoint, the tailable cursor 
of `mongodb-source` was still on the primary. The client the Kamelet builds 
from `hosts` (`SslAwareMongoClient`) has no read preference either. What works 
today is `readPreference` in `connectionString`: with 
`?replicaSet=rs0&readPreference=secondaryPreferred`, the cursor was on a 
secondary. The source description and its `connectionString` description say 
so, and the partial page has an example. A `readPreference` option can be added 
once camel-mongodb applies the endpoint option. On the sink, a read preference 
has no use, as it only writes.
   
   **Docs.** The sink description has Input and Output by operation, and the 
`CamelMongoDbRecordsMatched` and `CamelMongoDbRecordsAffected` headers are 
declared under `dataTypes`. The source description says that the place is 
stored only when the integration stops, so after a crash the documents since 
the last stop are read again. The sink partial page has a table of the 
operations and an update-with-upsert example; the source partial page has a 
tail tracking example with the three options and a `readPreference` example.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `mongo:8`: a three-member 
replica set (`rs0`, no auth), and a standalone server with auth for the 
read-only user. Endpoint URIs are from the `Initialising MongoDb endpoint` log 
line.
   
   | Case | Result |
   |---|---|
   | Sink, options unset, `main` and branch | Same URI 
`...?collection=orders&createCollection=false&database=shop&operation=insert`; 
both insert, body is the stored document, `CamelMongoOid` set |
   | Source, options unset, `main` and branch | Same URI 
`...?collection=events&database=shop&persistentTailTracking=false&tailTrackIncreasingField=seq`
 |
   | `update`, `upsert: true`, `[{"sku":"A1"},{"$set":{"qty":2}}]`, then `qty` 
3, then 3 again | 1st: matched 0, inserted `{sku:"A1",qty:2}`; 2nd: matched 1, 
changed 1, `qty` 3; 3rd: matched 1, changed 0 |
   | `update`, `upsert: true`, `{"$inc":{"qty":1}}` twice on a new `sku` | 
Inserted with `qty` 1, then `qty` 2 |
   | `update`, `upsert` unset, and `upsert: false`, filter matches nothing | 
Matched 0, nothing inserted |
   | `update`, filter matches two documents | Only the first is changed |
   | `update`, filter by `{"_id":{"$oid":"..."}}` | Matched 1, changed 1 |
   | `update`, body as `byte[]` | Works (on the branch before the text step: 
`body is not a List of Document objects with size = 2`) |
   | `update`, body a single object, `[{"sku":"A1"}]`, `[{...},{"qty":8}]`, or 
not JSON | Fail: `not a List ... size = 2`, `All update operators must start 
with '$'`, or a parse error |
   | `save` with `_id` `o-1` three times, the last body `{"_id":"o-1","qty":6}` 
| Inserted, replaced, replaced: the stored document is `{_id:"o-1",qty:6}` |
   | `save` without `_id` | Inserted with a new ObjectId, `CamelMongoOid` set |
   | `save` with `{"_id":{"$oid":"..."}}`, and with a `byte[]` body | Replaced 
the stored document |
   | `persistentTailTracking`, `persistentId: events-to-kafka`, `tailTrackDb: 
camelstate`, `tailTrackCollection: trackers`; run, stop, insert 2, run | 1st 
run read seq 1-8; `camelstate.trackers` holds `{persistentId:"events-to-kafka", 
lastTrackingValue: 8}`; 2nd run read only seq 9 and 10; 
`shop.camelTailTracking` unchanged |
   | The same, killed with `kill -9` after reading seq 11 | The stored place 
stayed 10; the next run read seq 11 again |
   | `main`, two sources on two capped collections, `persistentTailTracking`, 
no `persistentId` | One tracking document `{persistentId: null}`; after a 
restart, source B read 100-102 again; with the shared place set by hand to B's 
last value (105), source A never read its new seq 13 |
   | Branch, the same two sources with `persistentId` `orders-a` and `orders-b` 
| Two tracking documents (7 and 103); after a restart each read only its new 
document |
   | User with only `read` on `shop` (and `readWrite` on `camelstate`), 
`persistentTailTracking`, no `tailTrackDb` | Fails at start: `not authorized on 
shop to execute command { insert: "camelTailTracking" ...` |
   | The same with `tailTrackDb: camelstate` | Reads, and the place is stored 
in `camelstate.camelTailTracking` |
   | `readPreference=SECONDARY` added to the endpoint (test template only) | 
URI has `readPreference=SECONDARY`, but the tailable cursor (`$currentOp` with 
`idleCursors`) is on the primary |
   | `connectionString` with 
`replicaSet=rs0&readPreference=secondaryPreferred&appName=w5rp` | The cursor of 
appName `w5rp` is on a secondary (`localhost:37103`), none on the primary |
   
   `./mvnw clean verify -DskipTests` and `./mvnw clean verify` passed. 
KameletsCatalogTest now counts three headers for `mongodb-sink`. The MongoDB 
Citrus itests were not run locally; CI runs them.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to