This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 578898515 chore: mongodb-changes-stream-source sends the updated
document (#3170) (#3204)
578898515 is described below
commit 578898515e1a72c4d27f124161ba61e84b66b07a
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 21:37:32 2026 +0100
chore: mongodb-changes-stream-source sends the updated document (#3170)
(#3204)
For an update, mongodb-changes-stream-source sent the text null: the
change stream asked MongoDB only for the change, and the Kamelet sends
only the document of the change, so what was updated was lost.
The Kamelet takes a fullDocument option, mapped to the fullDocument
endpoint option of camel-mongodb as fullDocument={{?fullDocument}}:
- updateLookup (the new default): the whole document, read again from
the collection when the change is sent. It can already hold later
changes, and it is null when the document was deleted meanwhile.
- default: null, as before.
- whenAvailable: the document right after the update, on a collection
with change stream post-images turned on; null on another one.
- required: as whenAvailable, but without post-images the stream retries
the update every second and sends no later change.
The default changes what an update sends, from null to the document, and
a streamFilter on a field under fullDocument now also matches updates.
Setting fullDocument to default restores the old behaviour. The cost is
one more read on the server for each update, and an update whose
document and changed fields exceed 16 MB blocks the stream; both are in
the descriptions.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../mongodb-changes-stream-source-description.adoc | 62 +++++++++++++++++++++-
.../mongodb-changes-stream-source.kamelet.yaml | 14 +++--
.../main/resources/META-INF/kamelets-index.json | 2 +-
.../mongodb-changes-stream-source.kamelet.yaml | 14 +++--
4 files changed, 82 insertions(+), 10 deletions(-)
diff --git
a/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
b/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
index e0299671a..be27b7751 100644
--- a/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
+++ b/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
@@ -11,10 +11,69 @@ This Kamelet connects to MongoDB with:
=== Output format
-The Kamelet watches a collection with a MongoDB change stream, and sends one
message for each change made after it starts, as JSON text: the new document
for an insert or a replace, `null` for an update, and an object with the `_id`
for a delete. The `CamelMongoDbStreamOperationType` header says which kind of
change it is.
+The Kamelet watches a collection with a MongoDB change stream, and sends one
message for each change made after it starts, as JSON text: the new document
for an insert or a replace, the whole document for an update (see below), and
an object with the `_id` for a delete. The `CamelMongoDbStreamOperationType`
header says which kind of change it is.
Change streams need a replica set or a sharded cluster.
+=== The document of an update
+
+MongoDB records an update as the fields that changed, not as the document. The
`fullDocument` option says what the
+Kamelet sends as the body for an update:
+
+[width="100%",cols="1,3",options="header"]
+|===
+| `fullDocument` | Body for an update
+| `updateLookup` (default)
+| The whole document, read again from the collection when the change is sent.
It can already hold later changes:
+ after three quick updates of `qty` to 2, 3 and 4, all three messages can
show `qty` 4. It is `null` when the
+ document was deleted meanwhile. This costs MongoDB one more read for each
update.
+| `default`
+| `null`. Only the `_id` header says which document changed.
+| `whenAvailable`
+| The document as it was right after this update, on a collection with change
stream post-images turned on (see
+ below); `null` on another collection.
+| `required`
+| As `whenAvailable`, but on a collection without post-images, the Kamelet
retries the update every second and sends
+ no later change.
+|===
+
+Inserts, replaces and deletes are sent the same with any value. A
`streamFilter` on a field under `fullDocument`
+matches an update only when the body of the update is a document, so never
with `default`.
+
+With `updateLookup`, an update whose document and changed fields are together
larger than about 16 MB cannot be sent
+(MongoDB fails with `BSONObjectTooLarge`): the Kamelet retries it every second
and sends no later change. Use
+`default` for a collection with such documents.
+
+To send each update with the document as it was right after that update, turn
on post-images for the collection, as
+a user allowed to run `collMod`:
+
+[source,javascript]
+----
+db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled:
true } })
+----
+
+and set `fullDocument` to `whenAvailable`:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:mongodb-changes-stream-source"
+ parameters:
+ hosts: "mongo1.example.com:27017,mongo2.example.com:27017"
+ database: "shop"
+ collection: "orders"
+ username: "mongouser"
+ password: "mongopass"
+ fullDocument: "whenAvailable"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+With post-images turned on, MongoDB saves a copy of the document for each
update in its `config.system.preimages`
+collection, which takes disk space until MongoDB removes it.
+
=== Configuration
The Kamelet requires database connection parameters:
@@ -22,6 +81,7 @@ The Kamelet requires database connection parameters:
- `hosts` (the members of the replica set, as `host:port`) or
`connectionString`
- `database` and `collection`: the collection to watch
- `streamFilter` (optional): a `$match` stage that chooses the changes to send
+- `fullDocument` (optional): what the body is for an update; by default the
whole document
=== Usage example
diff --git a/kamelets/mongodb-changes-stream-source.kamelet.yaml
b/kamelets/mongodb-changes-stream-source.kamelet.yaml
index 298492f3e..45346e4ab 100644
--- a/kamelets/mongodb-changes-stream-source.kamelet.yaml
+++ b/kamelets/mongodb-changes-stream-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
Use it to react to changes in MongoDB, for example to copy new orders to
another system or to keep a cache or a search index up to date. To read the
documents of a capped collection from the start, use `mongodb-source`. To write
documents, use `mongodb-sink`.
- Output: the body is JSON text. For an insert or a replace, it is the
whole new document; for an update, the text `null`; for a delete, an object
with only the `_id` of the deleted document. The
`CamelMongoDbStreamOperationType` header says what happened (`insert`,
`update`, `replace` or `delete`), and the `_id` header holds the `_id` of the
document.
+ Output: the body is JSON text. For an insert or a replace, it is the
whole new document. For an update, it is the whole document as it is when the
change is sent, read again from the collection, so it can already hold later
changes, and it is the text `null` when the document was deleted meanwhile;
`fullDocument` changes this. For a delete, it is an object with only the `_id`
of the deleted document. The `CamelMongoDbStreamOperationType` header says what
happened (`insert`, `update [...]
- Good to know: only changes made after the Kamelet starts are sent.
MongoDB offers change streams only on a replica set or a sharded cluster, not
on a single standalone server. Set either `hosts` or `connectionString`. With
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB
without TLS, such as a local one. With `connectionString`, such as the
`mongodb+srv://` string of MongoDB Atlas, it connects only as that string says,
and `hosts`, `username`, `password`, [...]
+ Good to know: only changes made after the Kamelet starts are sent.
Reading the document again costs MongoDB one more read for each update; set
`fullDocument` to `default` to send `null` for an update instead. MongoDB
offers change streams only on a replica set or a sharded cluster, not on a
single standalone server. Set either `hosts` or `connectionString`. With
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB
without TLS, such as a local one. With `connec [...]
required:
- collection
- database
@@ -91,9 +91,15 @@ spec:
example: shop
streamFilter:
title: Stream Filter
- description: A `$match` stage of a MongoDB aggregation, as JSON text,
that chooses which changes to send. Match on `operationType` (such as `insert`)
or on fields of the new document under `fullDocument`. When it is not set, all
changes are sent.
+ description: A `$match` stage of a MongoDB aggregation, as JSON text,
that chooses which changes to send. Match on `operationType` (such as `insert`)
or on fields of the document under `fullDocument`. For an update,
`fullDocument` holds the document the Kamelet sends, so with the `fullDocument`
option set to `default` an update never matches a field under it. When it is
not set, all changes are sent.
type: string
example: "{ '$match':{'$or':[{'fullDocument.stringValue':
'specificValue'}]} }"
+ fullDocument:
+ title: Document of an Update
+ description: "What the body is for an update; inserts, replaces and
deletes are sent the same with any value. `updateLookup` (the default): the
whole document, read again from the collection when the change is sent. It can
already hold later changes, and it is the text `null` when the document was
deleted meanwhile. Use it to keep a copy, a cache or a search index up to date.
An update whose document and changed fields are together larger than about 16
MB cannot be sent: the Kame [...]
+ type: string
+ enum: ["updateLookup", "default", "whenAvailable", "required"]
+ default: updateLookup
dataTypes:
out:
headers:
@@ -128,7 +134,7 @@ spec:
properties:
mongoConnection: "#{{mongo-client}}"
from:
- uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&connectionUriString=RAW({{?connectionString}})"
+ uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&fullDocument={{?fullDocument}}&connectionUriString=RAW({{?connectionString}})"
steps:
- marshal:
json: {}
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 2624dc5ce..9dd5aed90 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1545,7 +1545,7 @@
"supportLevel" : "Stable",
"title" : "MongoDB Changes Stream Source",
"summary" : "Sends a message for each change to a collection of a MongoDB
database, such as a document inserted, updated or deleted, as it happens.",
- "description" : "Sends a message for each change to a collection of a
MongoDB database, such as a document inserted, updated or deleted, as it
happens.\n\nUse it to react to changes in MongoDB, for example to copy new
orders to another system or to keep a cache or a search index up to date. To
read the documents of a capped collection from the start, use `mongodb-source`.
To write documents, use `mongodb-sink`.\n\nOutput: the body is JSON text. For
an insert or a replace, it is the w [...]
+ "description" : "Sends a message for each change to a collection of a
MongoDB database, such as a document inserted, updated or deleted, as it
happens.\n\nUse it to react to changes in MongoDB, for example to copy new
orders to another system or to keep a cache or a search index up to date. To
read the documents of a capped collection from the start, use `mongodb-source`.
To write documents, use `mongodb-sink`.\n\nOutput: the body is JSON text. For
an insert or a replace, it is the w [...]
"group" : "MongoDB",
"namespace" : "Nosql",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
index 298492f3e..45346e4ab 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
Use it to react to changes in MongoDB, for example to copy new orders to
another system or to keep a cache or a search index up to date. To read the
documents of a capped collection from the start, use `mongodb-source`. To write
documents, use `mongodb-sink`.
- Output: the body is JSON text. For an insert or a replace, it is the
whole new document; for an update, the text `null`; for a delete, an object
with only the `_id` of the deleted document. The
`CamelMongoDbStreamOperationType` header says what happened (`insert`,
`update`, `replace` or `delete`), and the `_id` header holds the `_id` of the
document.
+ Output: the body is JSON text. For an insert or a replace, it is the
whole new document. For an update, it is the whole document as it is when the
change is sent, read again from the collection, so it can already hold later
changes, and it is the text `null` when the document was deleted meanwhile;
`fullDocument` changes this. For a delete, it is an object with only the `_id`
of the deleted document. The `CamelMongoDbStreamOperationType` header says what
happened (`insert`, `update [...]
- Good to know: only changes made after the Kamelet starts are sent.
MongoDB offers change streams only on a replica set or a sharded cluster, not
on a single standalone server. Set either `hosts` or `connectionString`. With
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB
without TLS, such as a local one. With `connectionString`, such as the
`mongodb+srv://` string of MongoDB Atlas, it connects only as that string says,
and `hosts`, `username`, `password`, [...]
+ Good to know: only changes made after the Kamelet starts are sent.
Reading the document again costs MongoDB one more read for each update; set
`fullDocument` to `default` to send `null` for an update instead. MongoDB
offers change streams only on a replica set or a sharded cluster, not on a
single standalone server. Set either `hosts` or `connectionString`. With
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB
without TLS, such as a local one. With `connec [...]
required:
- collection
- database
@@ -91,9 +91,15 @@ spec:
example: shop
streamFilter:
title: Stream Filter
- description: A `$match` stage of a MongoDB aggregation, as JSON text,
that chooses which changes to send. Match on `operationType` (such as `insert`)
or on fields of the new document under `fullDocument`. When it is not set, all
changes are sent.
+ description: A `$match` stage of a MongoDB aggregation, as JSON text,
that chooses which changes to send. Match on `operationType` (such as `insert`)
or on fields of the document under `fullDocument`. For an update,
`fullDocument` holds the document the Kamelet sends, so with the `fullDocument`
option set to `default` an update never matches a field under it. When it is
not set, all changes are sent.
type: string
example: "{ '$match':{'$or':[{'fullDocument.stringValue':
'specificValue'}]} }"
+ fullDocument:
+ title: Document of an Update
+ description: "What the body is for an update; inserts, replaces and
deletes are sent the same with any value. `updateLookup` (the default): the
whole document, read again from the collection when the change is sent. It can
already hold later changes, and it is the text `null` when the document was
deleted meanwhile. Use it to keep a copy, a cache or a search index up to date.
An update whose document and changed fields are together larger than about 16
MB cannot be sent: the Kame [...]
+ type: string
+ enum: ["updateLookup", "default", "whenAvailable", "required"]
+ default: updateLookup
dataTypes:
out:
headers:
@@ -128,7 +134,7 @@ spec:
properties:
mongoConnection: "#{{mongo-client}}"
from:
- uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&connectionUriString=RAW({{?connectionString}})"
+ uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&fullDocument={{?fullDocument}}&connectionUriString=RAW({{?connectionString}})"
steps:
- marshal:
json: {}