This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new a77cb308a chore: mongodb-sink can save and update documents, and
mongodb-source can name and place its tail tracking (#3170) (#3225)
a77cb308a is described below
commit a77cb308a9c2f79ba15bfcc4e7bc99c1f7d0a16f
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:34:49 2026 +0100
chore: mongodb-sink can save and update documents, and mongodb-source can
name and place its tail tracking (#3170) (#3225)
mongodb-sink only inserted. It now takes two options:
- operation (insert, the default, save or update), mapped to
operation={{operation}}, so the endpoint URI is the same as before when
it is not set. save replaces the document with the same _id as a whole,
or inserts it. update takes a JSON array in text of a filter and the
changes, such as [{"sku":"A1"},{"$set":{"qty":3}}], and changes the first
document that matches. For update only, a body that is not a List, such
as bytes from Kafka, is turned into text first, as camel-mongodb parses
only text into the filter and the changes.
- upsert, only for update: with true, a filter that matches no document
inserts one, and the next message updates it. camel-mongodb takes it only
as the CamelMongoDbUpsert header, so the template sets that header, and
only when upsert is set and the operation is update.
mongodb-source already had persistentTailTracking; it now takes
persistentId, tailTrackDb and tailTrackCollection, as optional
placeholders. Without persistentId, all the sources that store their place
in the same collection share one place, so a restarted source read
documents again or skipped them. tailTrackDb lets a user that may only read
the database of the events store the place elsewhere.
No readPreference option: the readPreference endpoint option of
camel-mongodb is never applied, and the tailable cursor stayed on the
primary with readPreference=SECONDARY. The descriptions show
readPreference=secondaryPreferred in connectionString instead, which reads
from a secondary.
KameletsCatalogTest counts three headers for mongodb-sink now: update sets
CamelMongoDbRecordsMatched and CamelMongoDbRecordsAffected.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../ROOT/partials/mongodb-sink-description.adoc | 53 ++++++++++++++++++++-
.../ROOT/partials/mongodb-source-description.adoc | 23 ++++++++-
kamelets/mongodb-sink.kamelet.yaml | 55 ++++++++++++++++++----
kamelets/mongodb-source.kamelet.yaml | 26 ++++++++--
.../kamelets/catalog/KameletsCatalogTest.java | 2 +-
.../main/resources/META-INF/kamelets-index.json | 6 +--
.../resources/kamelets/mongodb-sink.kamelet.yaml | 55 ++++++++++++++++++----
.../resources/kamelets/mongodb-source.kamelet.yaml | 26 ++++++++--
8 files changed, 214 insertions(+), 32 deletions(-)
diff --git a/docs/modules/ROOT/partials/mongodb-sink-description.adoc
b/docs/modules/ROOT/partials/mongodb-sink-description.adoc
index 707b1755c..162f277d4 100644
--- a/docs/modules/ROOT/partials/mongodb-sink-description.adoc
+++ b/docs/modules/ROOT/partials/mongodb-sink-description.adoc
@@ -14,7 +14,58 @@ Optional username and password authentication is supported
for secured MongoDB i
=== Write Operations
-The Kamelet performs insert operations into the specified collection. A
document whose `_id` is already in the collection fails; the `db-upsert` and
`ce-dbupsert` headers do not change that.
+The `operation` option says what the Kamelet does with each message:
+
+[width="100%",cols="1,2,2",options="header"]
+|===
+| `operation` | Body | What happens
+| `insert` (default)
+| One document as a JSON object, such as `{"_id":"o-1","item":"book","qty":2}`
+| The document is added. A document whose `_id` is already in the collection
fails.
+| `save`
+| One document as a JSON object, such as `{"_id":"o-1","item":"book","qty":5}`
+| The stored document with the same `_id` is replaced as a whole: fields that
are not in the body are removed. When
+ there is none, or the body has no `_id`, the document is added.
+| `update`
+| A JSON array of the filter and the changes, such as
`[{"sku":"A1"},{"$set":{"qty":3}}]`
+| The first document that matches the filter is changed. The changes must use
update operators, such as `$set`,
+ `$inc` or `$unset`. When no document matches, nothing is written, unless
`upsert` is true.
+|===
+
+An ObjectId is written in the body as `{"$oid":"6afd00000000000000000001"}`.
With `insert` and `save`, the
+`CamelMongoOid` header holds the `_id` of the document. With `update`, the
`CamelMongoDbRecordsMatched` and
+`CamelMongoDbRecordsAffected` headers say how many documents matched and how
many were changed. The `db-upsert` and
+`ce-dbupsert` headers are not used.
+
+=== Example with Update and Upsert
+
+With `upsert` true, the first message for a product inserts
`{"sku":"A1","qty":2}`, and each later one updates it:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: "stock"
+ bootstrapServers: "kafka.example.com:9092"
+ saslAuthType: "NONE"
+ steps:
+ # each record is a JSON array such as [{"sku":"A1"},{"$set":{"qty":2}}]
+ - to:
+ uri: "kamelet:mongodb-sink"
+ parameters:
+ hosts: "mongo1.example.com:27017"
+ username: "mongouser"
+ password: "mongopass"
+ database: "shop"
+ collection: "stock"
+ operation: "update"
+ upsert: true
+----
+
+To keep a copy of records by their ID instead, set `operation` to `save` and
send each record as a document with its
+`_id`.
=== Write Concerns
diff --git a/docs/modules/ROOT/partials/mongodb-source-description.adoc
b/docs/modules/ROOT/partials/mongodb-source-description.adoc
index b08a7487c..0b31348b3 100644
--- a/docs/modules/ROOT/partials/mongodb-source-description.adoc
+++ b/docs/modules/ROOT/partials/mongodb-source-description.adoc
@@ -18,6 +18,8 @@ The MongoDB Source Kamelet supports the following
configurations:
- **SSL**: Whether to connect with TLS (default: true); set it to false for a
MongoDB without TLS
- **Tail Track Increasing Field**: A field whose value grows with each new
document
- **Persistent Tail Tracking**: Remember the last document read across
restarts (needs the increasing field)
+- **Tail Tracking Name** (`persistentId`): The name the place is stored under;
give each source its own
+- **Tail Tracking Database** and **Collection** (`tailTrackDb`,
`tailTrackCollection`): Where the place is stored
=== Output Format
@@ -43,7 +45,13 @@ The Kamelet reads the documents of a capped collection with
a tailable cursor: f
=== Example with Persistent Tail Tracking
-The Kamelet stores the `seq` value of the last document read in the
`camelTailTracking` collection of the same database, and after a restart goes
on from there.
+The Kamelet stores the `seq` value of the last document read, and after a
restart goes on from there. It stores it
+when the integration stops, so after a crash it reads again the documents
since the last stop.
+
+The place is stored in a document named after `persistentId`, in the
`camelTailTracking` collection of the same
+database. Give each source its own `persistentId`: sources without one share a
single place, and a restarted source
+then reads documents again or skips them. Set `tailTrackDb` and
`tailTrackCollection` to store the place elsewhere,
+for example when the user may only read the database of the events:
[source,yaml,subs='+attributes,macros']
----
@@ -58,11 +66,17 @@ The Kamelet stores the `seq` value of the last document
read in the `camelTailTr
password: "mongopass"
tailTrackIncreasingField: "seq"
persistentTailTracking: true
+ persistentId: "events-to-kafka"
+ tailTrackDb: "camelstate"
+ tailTrackCollection: "trackers"
steps:
- to:
uri: "kamelet:log-sink"
----
+The place is then the document `{ persistentId: "events-to-kafka",
lastTrackingValue: 42 }` in
+`camelstate.trackers`.
+
=== Capped Collections
The collection must be capped, or the Kamelet fails to start. A capped
collection is created with a fixed size, for example in `mongosh`:
@@ -105,3 +119,10 @@ For a user defined in the `shop` database, on a MongoDB
without TLS:
----
connectionString:
"mongodb://shopuser:[email protected]:27017,mongo2.example.com:27017/?authSource=shop&replicaSet=rs0"
----
+
+To read from a secondary member of a replica set, and so take load off the
primary, add `readPreference`:
+
+[source,yaml,subs='+attributes,macros']
+----
+connectionString:
"mongodb://mongouser:[email protected]:27017,mongo2.example.com:27017,mongo3.example.com:27017/?replicaSet=rs0&readPreference=secondaryPreferred"
+----
diff --git a/kamelets/mongodb-sink.kamelet.yaml
b/kamelets/mongodb-sink.kamelet.yaml
index 1e8ca94d5..80806a511 100644
--- a/kamelets/mongodb-sink.kamelet.yaml
+++ b/kamelets/mongodb-sink.kamelet.yaml
@@ -31,15 +31,15 @@ spec:
definition:
title: "MongoDB Sink"
description: |-
- Inserts each message as a new document into a collection of a MongoDB
database.
+ Writes each message to a collection of a MongoDB database: inserts it as
a new document, or, with `operation`, replaces or updates a stored document.
- Use it to store events, orders or other records as documents in MongoDB,
for example each message that arrives on a Kafka topic. To read documents, use
`mongodb-source` or `mongodb-changes-stream-source`.
+ Use it to store events, orders or other records as documents in MongoDB,
for example each message that arrives on a Kafka topic, to keep a copy of
records by their ID (`save`), or to change fields of a document, such as the
stock of a product (`update`). To read documents, use `mongodb-source` or
`mongodb-changes-stream-source`.
- Input: the body is one document, as a JSON object in text, such as
`{"item":"book","qty":2}`. When it has no `_id` field, MongoDB gives the
document a new ObjectId. A JSON array, or text that is not JSON, fails.
+ Input: with `insert` (the default) and `save`, the body is one document,
as a JSON object in text, such as `{"item":"book","qty":2}`. When it has no
`_id` field, MongoDB gives the document a new ObjectId. With `update`, the body
is a JSON array in text of two objects: the filter that selects the document
and the changes, such as `[{"sku":"A1"},{"$set":{"qty":3}}]`. An ObjectId is
written as `{"$oid":"..."}`. Text that is not JSON fails.
- Output: the body afterwards is the document that was stored, as a Map
with its `_id`, and the `CamelMongoOid` header holds that `_id`.
+ Output: with `insert`, the body afterwards is the document that was
stored, as a Map with its `_id`. With `save` and `update`, it is the result
that MongoDB returns. With `insert` and `save`, the `CamelMongoOid` header
holds the `_id` of the document. With `update`, the
`CamelMongoDbRecordsMatched` and `CamelMongoDbRecordsAffected` headers say how
many documents matched the filter and how many were changed.
- Good to know: a document whose `_id` is already in the collection fails:
the Kamelet only inserts, and the `db-upsert` and `ce-dbupsert` headers do not
change that. With `createCollection` false, the default, the Kamelet fails to
start when the collection does not exist yet. Set either `hosts` or
`connectionString`. With `hosts`, it connects with TLS by default; set `ssl` to
false for a MongoDB without TLS, such as a local one. With `connectionString`,
such as the `mongodb+srv://` [...]
+ Good to know: with `insert`, a document whose `_id` is already in the
collection fails; `save` replaces it instead. `save` replaces the whole
document, so fields that are not in the body are removed. `update` changes only
the first document that matches, and nothing when none matches, unless `upsert`
is true. The `db-upsert` and `ce-dbupsert` headers are not used. With
`createCollection` false, the default, the Kamelet fails to start when the
collection does not exist yet. Set eith [...]
required:
- collection
- database
@@ -60,7 +60,7 @@ spec:
example:
"mongodb+srv://myuser:[email protected]/?retryWrites=true&w=majority"
collection:
title: MongoDB Collection
- description: The name of the collection to insert the documents into.
+ description: The name of the collection to write the documents to.
type: string
example: orders
password:
@@ -93,7 +93,7 @@ spec:
example: shop
writeConcern:
title: Write Concern
- description: How many MongoDB servers must confirm an insert before it
counts as done, such as `MAJORITY` for most members of a replica set. When it
is not set, MongoDB acknowledges each insert (`ACKNOWLEDGED`).
+ description: How many MongoDB servers must confirm a write before it
counts as done, such as `MAJORITY` for most members of a replica set. When it
is not set, MongoDB acknowledges each write (`ACKNOWLEDGED`).
type: string
enum: ["ACKNOWLEDGED", "W1", "W2", "W3", "UNACKNOWLEDGED",
"JOURNALED", "MAJORITY"]
createCollection:
@@ -101,6 +101,16 @@ spec:
description: If true, the collection is created when the Kamelet
starts, if it does not exist. If false, the Kamelet fails to start when the
collection does not exist.
type: boolean
default: false
+ operation:
+ title: Operation
+ description: "What to do with each message. `insert` adds the body as
a new document, and fails when its `_id` is already in the collection. `save`
replaces the document that has the `_id` of the body as a whole, or adds the
body when there is none or the body has no `_id`. `update` changes the first
document that matches a filter: the body is the filter and the changes, such as
`[{\"sku\":\"A1\"},{\"$set\":{\"qty\":3}}]`, and the changes must use update
operators such as `$set`, [...]
+ type: string
+ enum: ["insert", "save", "update"]
+ default: insert
+ upsert:
+ title: Insert When Missing
+ description: Only used when `operation` is `update`. If true, a filter
that matches no document inserts a new one, made of the fields of the filter
and the changes, such as `{"sku":"A1","qty":3}`, and the next message with the
same filter updates it. If false or not set, a filter that matches no document
changes nothing.
+ type: boolean
types:
in:
mediaType: application/json
@@ -109,8 +119,16 @@ spec:
headers:
CamelMongoOid:
title: Document ID
- description: The `_id` of the document that was inserted, an
ObjectId when MongoDB made it.
+ description: The `_id` of the document that was inserted or saved,
an ObjectId when MongoDB made it. Only set by `insert` and `save`.
type: string
+ CamelMongoDbRecordsMatched:
+ title: Documents Matched
+ description: How many documents matched the filter of an `update`, 0
or 1. Only set by `update`.
+ type: integer
+ CamelMongoDbRecordsAffected:
+ title: Documents Changed
+ description: How many documents an `update` changed, 0 or 1. It is 0
when the document already had the values, and when `upsert` inserted a new one.
Only set by `update`.
+ type: integer
dependencies:
- "camel:kamelet"
- "camel:mongodb"
@@ -132,5 +150,24 @@ spec:
from:
uri: kamelet:source
steps:
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{operation}}' == 'update' && ${body} !is
'java.util.List'"
+ steps:
+ - convertBodyTo:
+ type: java.lang.String
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{operation}}' == 'update' && '{{?upsert}}'
!= ''"
+ steps:
+ - setHeader:
+ name: CamelMongoDbUpsert
+ expression:
+ constant:
+ expression: "{{?upsert}}"
- to:
- uri:
"{{local-mongodb}}:test?createCollection={{?createCollection}}&writeConcern={{?writeConcern}}&collection={{collection}}&database={{database}}&operation=insert&connectionUriString=RAW({{?connectionString}})"
+ uri:
"{{local-mongodb}}:test?createCollection={{?createCollection}}&writeConcern={{?writeConcern}}&collection={{collection}}&database={{database}}&operation={{operation}}&connectionUriString=RAW({{?connectionString}})"
diff --git a/kamelets/mongodb-source.kamelet.yaml
b/kamelets/mongodb-source.kamelet.yaml
index e551dc2be..70f1f58da 100644
--- a/kamelets/mongodb-source.kamelet.yaml
+++ b/kamelets/mongodb-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is one document as JSON text. An `_id` that is an
ObjectId comes out as an object with only its time (`timestamp` and `date`),
not as the ID text, and the format of dates depends on the Camel version. The
`CamelMongoDbDatabase` and `CamelMongoDbCollection` headers name where the
document comes from.
- Good to know: the collection must be capped (created with a fixed size),
or the Kamelet fails to start. Each time the integration starts, it reads the
whole collection again from the start, unless `persistentTailTracking` is true.
Set either `hosts` or `connectionString`. With `hosts`, it connects with TLS by
default; set `ssl` to false for a MongoDB without TLS, such as a local one.
With `connectionString`, such as the `mongodb+srv://` string of MongoDB Atlas,
it connects only as [...]
+ Good to know: the collection must be capped (created with a fixed size),
or the Kamelet fails to start. Each time the integration starts, it reads the
whole collection again from the start, unless `persistentTailTracking` is true.
It stores its place only when the integration stops, so after a crash it reads
again the documents since the last stop. Set either `hosts` or
`connectionString`. With `hosts`, it connects with TLS by default; set `ssl` to
false for a MongoDB without TLS, [...]
required:
- collection
- database
@@ -50,7 +50,7 @@ spec:
example: mongo1.example.com:27017,mongo2.example.com:27017
connectionString:
title: Connection String
- description: "The MongoDB connection string, which starts with
`mongodb://` or `mongodb+srv://`, such as the one MongoDB Atlas shows under
Connect. Use it instead of `hosts` for MongoDB Atlas, for a user defined in a
database other than `admin` (add `authSource`, such as `?authSource=shop`), or
for settings such as `replicaSet` or `tls`. When it is set, `hosts`,
`username`, `password`, `ssl` and `sslValidationEnabled` are not used. TLS is
used as the string says: always with `mon [...]
+ description: "The MongoDB connection string, which starts with
`mongodb://` or `mongodb+srv://`, such as the one MongoDB Atlas shows under
Connect. Use it instead of `hosts` for MongoDB Atlas, for a user defined in a
database other than `admin` (add `authSource`, such as `?authSource=shop`), or
for settings such as `replicaSet`, `tls` or `readPreference`, such as
`readPreference=secondaryPreferred` to read from a secondary member of a
replica set. When it is set, `hosts`, `userna [...]
type: string
format: password
x-descriptors:
@@ -91,7 +91,7 @@ spec:
example: shop
persistentTailTracking:
title: Remember Last Document
- description: If true, the Kamelet stores the
`tailTrackIncreasingField` value of the last document it read in the
`camelTailTracking` collection of the same database, and after a restart goes
on from there instead of reading the whole collection again. Needs
`tailTrackIncreasingField`.
+ description: If true, the Kamelet stores the
`tailTrackIncreasingField` value of the last document it read, and after a
restart goes on from there instead of reading the whole collection again. It
stores it when the integration stops, so after a crash the documents since the
last stop are read again. It is stored in the `camelTailTracking` collection of
the same database (see `tailTrackDb` and `tailTrackCollection`), under the name
in `persistentId`. Needs `tailTrackIncreasingField`.
type: boolean
default: false
tailTrackIncreasingField:
@@ -99,6 +99,24 @@ spec:
description: The name of a field whose value grows with each new
document, such as a sequence number or a time, used to find the place to go on
reading from. Required when `persistentTailTracking` is true.
type: string
example: seq
+ persistentId:
+ title: Tail Tracking Name
+ description: The name under which the place of this Kamelet is stored
when `persistentTailTracking` is true. Give each source that stores its place
in the same collection its own name. When it is not set, all of them share one
place, so a restarted source reads documents again or skips them, as it goes on
from the place another source stored.
+ type: string
+ pattern: "^[A-Za-z0-9_.:-]+$"
+ example: events-to-kafka
+ tailTrackDb:
+ title: Tail Tracking Database
+ description: The database to store the place in when
`persistentTailTracking` is true. When it is not set, the place is stored in
`database`. Use it when the MongoDB user may only read `database`; the user
must be allowed to write to this one.
+ type: string
+ pattern: "^[A-Za-z0-9_-]+$"
+ example: camelstate
+ tailTrackCollection:
+ title: Tail Tracking Collection
+ description: The collection to store the place in when
`persistentTailTracking` is true. When it is not set, `camelTailTracking`.
MongoDB creates it, if it does not exist, when the Kamelet starts.
+ type: string
+ pattern: "^[A-Za-z0-9_.-]+$"
+ example: trackers
dataTypes:
out:
headers:
@@ -133,7 +151,7 @@ spec:
properties:
mongoConnection: "#{{mongo-client}}"
from:
- uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&persistentTailTracking={{persistentTailTracking}}&tailTrackIncreasingField={{?tailTrackIncreasingField}}&connectionUriString=RAW({{?connectionString}})"
+ uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&persistentTailTracking={{persistentTailTracking}}&tailTrackIncreasingField={{?tailTrackIncreasingField}}&persistentId={{?persistentId}}&tailTrackDb={{?tailTrackDb}}&tailTrackCollection={{?tailTrackCollection}}&connectionUriString=RAW({{?connectionString}})"
steps:
- marshal:
json: {}
diff --git
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
index 9df066819..6cbd9c943 100644
---
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
+++
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
@@ -260,7 +260,7 @@ public class KameletsCatalogTest {
verifyHeaders("minio-source", 5);
verifyHeaders("minio-sink", 2);
verifyHeaders("mongodb-changes-stream-source", 3);
- verifyHeaders("mongodb-sink", 1);
+ verifyHeaders("mongodb-sink", 3);
verifyHeaders("mongodb-source", 3);
verifyHeaders("mqtt-sink", 3);
verifyHeaders("mqtt-source", 2);
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 6b6b95909..409ab4f22 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1554,8 +1554,8 @@
"type" : "sink",
"supportLevel" : "Stable",
"title" : "MongoDB Sink",
- "summary" : "Inserts each message as a new document into a collection of a
MongoDB database.",
- "description" : "Inserts each message as a new document into a collection
of a MongoDB database.\n\nUse it to store events, orders or other records as
documents in MongoDB, for example each message that arrives on a Kafka topic.
To read documents, use `mongodb-source` or
`mongodb-changes-stream-source`.\n\nInput: the body is one document, as a JSON
object in text, such as `{\"item\":\"book\",\"qty\":2}`. When it has no `_id`
field, MongoDB gives the document a new ObjectId. A JSON ar [...]
+ "summary" : "Writes each message to a collection of a MongoDB database:
inserts it as a new document, or, with `operation`, replaces or updates a
stored document.",
+ "description" : "Writes each message to a collection of a MongoDB
database: inserts it as a new document, or, with `operation`, replaces or
updates a stored document.\n\nUse it to store events, orders or other records
as documents in MongoDB, for example each message that arrives on a Kafka
topic, to keep a copy of records by their ID (`save`), or to change fields of a
document, such as the stock of a product (`update`). To read documents, use
`mongodb-source` or `mongodb-changes-str [...]
"group" : "MongoDB",
"namespace" : "Nosql",
"provider" : "Apache Software Foundation"
@@ -1565,7 +1565,7 @@
"supportLevel" : "Stable",
"title" : "MongoDB Source",
"summary" : "Reads the documents of a capped collection in a MongoDB
database, first those already there and then each new one as it is added, one
message per document, as JSON text.",
- "description" : "Reads the documents of a capped collection in a MongoDB
database, first those already there and then each new one as it is added, one
message per document, as JSON text.\n\nUse it to pass on records that an
application appends to a capped collection, such as log entries or events, as
they arrive. For a normal collection, or to also see updates and deletes, use
`mongodb-changes-stream-source`. To write documents, use
`mongodb-sink`.\n\nOutput: the body is one document [...]
+ "description" : "Reads the documents of a capped collection in a MongoDB
database, first those already there and then each new one as it is added, one
message per document, as JSON text.\n\nUse it to pass on records that an
application appends to a capped collection, such as log entries or events, as
they arrive. For a normal collection, or to also see updates and deletes, use
`mongodb-changes-stream-source`. To write documents, use
`mongodb-sink`.\n\nOutput: the body is one document [...]
"group" : "MongoDB",
"namespace" : "Nosql",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-sink.kamelet.yaml
index 1e8ca94d5..80806a511 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-sink.kamelet.yaml
@@ -31,15 +31,15 @@ spec:
definition:
title: "MongoDB Sink"
description: |-
- Inserts each message as a new document into a collection of a MongoDB
database.
+ Writes each message to a collection of a MongoDB database: inserts it as
a new document, or, with `operation`, replaces or updates a stored document.
- Use it to store events, orders or other records as documents in MongoDB,
for example each message that arrives on a Kafka topic. To read documents, use
`mongodb-source` or `mongodb-changes-stream-source`.
+ Use it to store events, orders or other records as documents in MongoDB,
for example each message that arrives on a Kafka topic, to keep a copy of
records by their ID (`save`), or to change fields of a document, such as the
stock of a product (`update`). To read documents, use `mongodb-source` or
`mongodb-changes-stream-source`.
- Input: the body is one document, as a JSON object in text, such as
`{"item":"book","qty":2}`. When it has no `_id` field, MongoDB gives the
document a new ObjectId. A JSON array, or text that is not JSON, fails.
+ Input: with `insert` (the default) and `save`, the body is one document,
as a JSON object in text, such as `{"item":"book","qty":2}`. When it has no
`_id` field, MongoDB gives the document a new ObjectId. With `update`, the body
is a JSON array in text of two objects: the filter that selects the document
and the changes, such as `[{"sku":"A1"},{"$set":{"qty":3}}]`. An ObjectId is
written as `{"$oid":"..."}`. Text that is not JSON fails.
- Output: the body afterwards is the document that was stored, as a Map
with its `_id`, and the `CamelMongoOid` header holds that `_id`.
+ Output: with `insert`, the body afterwards is the document that was
stored, as a Map with its `_id`. With `save` and `update`, it is the result
that MongoDB returns. With `insert` and `save`, the `CamelMongoOid` header
holds the `_id` of the document. With `update`, the
`CamelMongoDbRecordsMatched` and `CamelMongoDbRecordsAffected` headers say how
many documents matched the filter and how many were changed.
- Good to know: a document whose `_id` is already in the collection fails:
the Kamelet only inserts, and the `db-upsert` and `ce-dbupsert` headers do not
change that. With `createCollection` false, the default, the Kamelet fails to
start when the collection does not exist yet. Set either `hosts` or
`connectionString`. With `hosts`, it connects with TLS by default; set `ssl` to
false for a MongoDB without TLS, such as a local one. With `connectionString`,
such as the `mongodb+srv://` [...]
+ Good to know: with `insert`, a document whose `_id` is already in the
collection fails; `save` replaces it instead. `save` replaces the whole
document, so fields that are not in the body are removed. `update` changes only
the first document that matches, and nothing when none matches, unless `upsert`
is true. The `db-upsert` and `ce-dbupsert` headers are not used. With
`createCollection` false, the default, the Kamelet fails to start when the
collection does not exist yet. Set eith [...]
required:
- collection
- database
@@ -60,7 +60,7 @@ spec:
example:
"mongodb+srv://myuser:[email protected]/?retryWrites=true&w=majority"
collection:
title: MongoDB Collection
- description: The name of the collection to insert the documents into.
+ description: The name of the collection to write the documents to.
type: string
example: orders
password:
@@ -93,7 +93,7 @@ spec:
example: shop
writeConcern:
title: Write Concern
- description: How many MongoDB servers must confirm an insert before it
counts as done, such as `MAJORITY` for most members of a replica set. When it
is not set, MongoDB acknowledges each insert (`ACKNOWLEDGED`).
+ description: How many MongoDB servers must confirm a write before it
counts as done, such as `MAJORITY` for most members of a replica set. When it
is not set, MongoDB acknowledges each write (`ACKNOWLEDGED`).
type: string
enum: ["ACKNOWLEDGED", "W1", "W2", "W3", "UNACKNOWLEDGED",
"JOURNALED", "MAJORITY"]
createCollection:
@@ -101,6 +101,16 @@ spec:
description: If true, the collection is created when the Kamelet
starts, if it does not exist. If false, the Kamelet fails to start when the
collection does not exist.
type: boolean
default: false
+ operation:
+ title: Operation
+ description: "What to do with each message. `insert` adds the body as
a new document, and fails when its `_id` is already in the collection. `save`
replaces the document that has the `_id` of the body as a whole, or adds the
body when there is none or the body has no `_id`. `update` changes the first
document that matches a filter: the body is the filter and the changes, such as
`[{\"sku\":\"A1\"},{\"$set\":{\"qty\":3}}]`, and the changes must use update
operators such as `$set`, [...]
+ type: string
+ enum: ["insert", "save", "update"]
+ default: insert
+ upsert:
+ title: Insert When Missing
+ description: Only used when `operation` is `update`. If true, a filter
that matches no document inserts a new one, made of the fields of the filter
and the changes, such as `{"sku":"A1","qty":3}`, and the next message with the
same filter updates it. If false or not set, a filter that matches no document
changes nothing.
+ type: boolean
types:
in:
mediaType: application/json
@@ -109,8 +119,16 @@ spec:
headers:
CamelMongoOid:
title: Document ID
- description: The `_id` of the document that was inserted, an
ObjectId when MongoDB made it.
+ description: The `_id` of the document that was inserted or saved,
an ObjectId when MongoDB made it. Only set by `insert` and `save`.
type: string
+ CamelMongoDbRecordsMatched:
+ title: Documents Matched
+ description: How many documents matched the filter of an `update`, 0
or 1. Only set by `update`.
+ type: integer
+ CamelMongoDbRecordsAffected:
+ title: Documents Changed
+ description: How many documents an `update` changed, 0 or 1. It is 0
when the document already had the values, and when `upsert` inserted a new one.
Only set by `update`.
+ type: integer
dependencies:
- "camel:kamelet"
- "camel:mongodb"
@@ -132,5 +150,24 @@ spec:
from:
uri: kamelet:source
steps:
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{operation}}' == 'update' && ${body} !is
'java.util.List'"
+ steps:
+ - convertBodyTo:
+ type: java.lang.String
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{operation}}' == 'update' && '{{?upsert}}'
!= ''"
+ steps:
+ - setHeader:
+ name: CamelMongoDbUpsert
+ expression:
+ constant:
+ expression: "{{?upsert}}"
- to:
- uri:
"{{local-mongodb}}:test?createCollection={{?createCollection}}&writeConcern={{?writeConcern}}&collection={{collection}}&database={{database}}&operation=insert&connectionUriString=RAW({{?connectionString}})"
+ uri:
"{{local-mongodb}}:test?createCollection={{?createCollection}}&writeConcern={{?writeConcern}}&collection={{collection}}&database={{database}}&operation={{operation}}&connectionUriString=RAW({{?connectionString}})"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-source.kamelet.yaml
index e551dc2be..70f1f58da 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is one document as JSON text. An `_id` that is an
ObjectId comes out as an object with only its time (`timestamp` and `date`),
not as the ID text, and the format of dates depends on the Camel version. The
`CamelMongoDbDatabase` and `CamelMongoDbCollection` headers name where the
document comes from.
- Good to know: the collection must be capped (created with a fixed size),
or the Kamelet fails to start. Each time the integration starts, it reads the
whole collection again from the start, unless `persistentTailTracking` is true.
Set either `hosts` or `connectionString`. With `hosts`, it connects with TLS by
default; set `ssl` to false for a MongoDB without TLS, such as a local one.
With `connectionString`, such as the `mongodb+srv://` string of MongoDB Atlas,
it connects only as [...]
+ Good to know: the collection must be capped (created with a fixed size),
or the Kamelet fails to start. Each time the integration starts, it reads the
whole collection again from the start, unless `persistentTailTracking` is true.
It stores its place only when the integration stops, so after a crash it reads
again the documents since the last stop. Set either `hosts` or
`connectionString`. With `hosts`, it connects with TLS by default; set `ssl` to
false for a MongoDB without TLS, [...]
required:
- collection
- database
@@ -50,7 +50,7 @@ spec:
example: mongo1.example.com:27017,mongo2.example.com:27017
connectionString:
title: Connection String
- description: "The MongoDB connection string, which starts with
`mongodb://` or `mongodb+srv://`, such as the one MongoDB Atlas shows under
Connect. Use it instead of `hosts` for MongoDB Atlas, for a user defined in a
database other than `admin` (add `authSource`, such as `?authSource=shop`), or
for settings such as `replicaSet` or `tls`. When it is set, `hosts`,
`username`, `password`, `ssl` and `sslValidationEnabled` are not used. TLS is
used as the string says: always with `mon [...]
+ description: "The MongoDB connection string, which starts with
`mongodb://` or `mongodb+srv://`, such as the one MongoDB Atlas shows under
Connect. Use it instead of `hosts` for MongoDB Atlas, for a user defined in a
database other than `admin` (add `authSource`, such as `?authSource=shop`), or
for settings such as `replicaSet`, `tls` or `readPreference`, such as
`readPreference=secondaryPreferred` to read from a secondary member of a
replica set. When it is set, `hosts`, `userna [...]
type: string
format: password
x-descriptors:
@@ -91,7 +91,7 @@ spec:
example: shop
persistentTailTracking:
title: Remember Last Document
- description: If true, the Kamelet stores the
`tailTrackIncreasingField` value of the last document it read in the
`camelTailTracking` collection of the same database, and after a restart goes
on from there instead of reading the whole collection again. Needs
`tailTrackIncreasingField`.
+ description: If true, the Kamelet stores the
`tailTrackIncreasingField` value of the last document it read, and after a
restart goes on from there instead of reading the whole collection again. It
stores it when the integration stops, so after a crash the documents since the
last stop are read again. It is stored in the `camelTailTracking` collection of
the same database (see `tailTrackDb` and `tailTrackCollection`), under the name
in `persistentId`. Needs `tailTrackIncreasingField`.
type: boolean
default: false
tailTrackIncreasingField:
@@ -99,6 +99,24 @@ spec:
description: The name of a field whose value grows with each new
document, such as a sequence number or a time, used to find the place to go on
reading from. Required when `persistentTailTracking` is true.
type: string
example: seq
+ persistentId:
+ title: Tail Tracking Name
+ description: The name under which the place of this Kamelet is stored
when `persistentTailTracking` is true. Give each source that stores its place
in the same collection its own name. When it is not set, all of them share one
place, so a restarted source reads documents again or skips them, as it goes on
from the place another source stored.
+ type: string
+ pattern: "^[A-Za-z0-9_.:-]+$"
+ example: events-to-kafka
+ tailTrackDb:
+ title: Tail Tracking Database
+ description: The database to store the place in when
`persistentTailTracking` is true. When it is not set, the place is stored in
`database`. Use it when the MongoDB user may only read `database`; the user
must be allowed to write to this one.
+ type: string
+ pattern: "^[A-Za-z0-9_-]+$"
+ example: camelstate
+ tailTrackCollection:
+ title: Tail Tracking Collection
+ description: The collection to store the place in when
`persistentTailTracking` is true. When it is not set, `camelTailTracking`.
MongoDB creates it, if it does not exist, when the Kamelet starts.
+ type: string
+ pattern: "^[A-Za-z0-9_.-]+$"
+ example: trackers
dataTypes:
out:
headers:
@@ -133,7 +151,7 @@ spec:
properties:
mongoConnection: "#{{mongo-client}}"
from:
- uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&persistentTailTracking={{persistentTailTracking}}&tailTrackIncreasingField={{?tailTrackIncreasingField}}&connectionUriString=RAW({{?connectionString}})"
+ uri:
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&persistentTailTracking={{persistentTailTracking}}&tailTrackIncreasingField={{?tailTrackIncreasingField}}&persistentId={{?persistentId}}&tailTrackDb={{?tailTrackDb}}&tailTrackCollection={{?tailTrackCollection}}&connectionUriString=RAW({{?connectionString}})"
steps:
- marshal:
json: {}