This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 51acba2fe chore: the Couchbase Kamelets work with a scope and a
collection, and couchbase-sink can delete (#3170) (#3220)
51acba2fe is described below
commit 51acba2fee65cdc4810174f2db005952494d0311
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:32:57 2026 +0100
chore: the Couchbase Kamelets work with a scope and a collection, and
couchbase-sink can delete (#3170) (#3220)
couchbase-sink and couchbase-source take optional scope and collection,
so documents are written to and read from a collection other than the
default one of the bucket.
couchbase-sink also takes:
- key: the document ID for a message without the CamelCouchbaseId
header, when autoStartId is false;
- operation: CCB_PUT (store, as before) or CCB_DEL (delete);
- persistTo and replicateTo, the durability of a write;
- connectTimeout, and retryAttempts, retryPause and operationTimeout,
mapped to the endpoint options producerRetryAttempts,
producerRetryPause and queryTimeout. A write or delete fails after
retryAttempts x retryPause ms, or operationTimeout when that is longer.
couchbase-source also takes processedStrategy: with delete, each document
is deleted once its message has been processed, so it is sent only once.
All new options are optional with no default; with them unset the
endpoint URIs are the same as before. scope, collection and key are
passed in RAW() and have a pattern.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../ROOT/partials/couchbase-sink-description.adoc | 72 ++++++++++++++++++-
.../partials/couchbase-source-description.adoc | 42 +++++++++++-
kamelets/couchbase-sink.kamelet.yaml | 80 +++++++++++++++++++---
kamelets/couchbase-source.kamelet.yaml | 29 ++++++--
.../main/resources/META-INF/kamelets-index.json | 6 +-
.../resources/kamelets/couchbase-sink.kamelet.yaml | 80 +++++++++++++++++++---
.../kamelets/couchbase-source.kamelet.yaml | 29 ++++++--
7 files changed, 308 insertions(+), 30 deletions(-)
diff --git a/docs/modules/ROOT/partials/couchbase-sink-description.adoc
b/docs/modules/ROOT/partials/couchbase-sink-description.adoc
index 0273539b9..322d7c89d 100644
--- a/docs/modules/ROOT/partials/couchbase-sink-description.adoc
+++ b/docs/modules/ROOT/partials/couchbase-sink-description.adoc
@@ -16,7 +16,77 @@ Optional settings include:
- Port (defaults to 8091)
- Starting ID for document inserts (defaults to 1)
- Auto start ID flag (defaults to true)
+- Scope and collection to store the documents in
+- A fixed document ID (`key`) for messages without the `CamelCouchbaseId`
header
+- The operation, `CCB_PUT` to store or `CCB_DEL` to delete
+- Durability (`persistTo`, `replicateTo`) and timeouts (`connectTimeout`,
`retryAttempts`, `retryPause`, `operationTimeout`)
=== Document Storage
-Documents are stored in the specified Couchbase bucket. The Kamelet can
automatically generate document IDs starting from a configurable starting
point. The count starts again at the starting ID each time the integration
starts, and a document that already has an ID is replaced, so a restart
overwrites the documents stored before. To choose the IDs yourself, set
`autoStartId` to false and give each message its ID in the `CamelCouchbaseId`
header.
\ No newline at end of file
+Documents are stored in the specified Couchbase bucket. The Kamelet can
automatically generate document IDs starting from a configurable starting
point. The count starts again at the starting ID each time the integration
starts, and a document that already has an ID is replaced, so a restart
overwrites the documents stored before. To choose the IDs yourself, set
`autoStartId` to false and give each message its ID in the `CamelCouchbaseId`
header. A message without the header then uses `k [...]
+
+=== Example with a Scope and a Collection
+
+Without `collection`, the documents go to the default collection of the
bucket, and `scope` is not used. Set both to
+store them in a collection of your own. The collection must exist: if it does
not, each message waits and fails with
+`AmbiguousTimeoutException`. Here each order is stored in the `orders`
collection of the `sales` scope, under the
+order number the message carries in its `CamelCouchbaseId` header:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: "orders"
+ bootstrapServers: "kafka.example.com:9092"
+ saslAuthType: "NONE"
+ steps:
+ - setHeader:
+ name: CamelCouchbaseId
+ jq: ".orderId"
+ - to:
+ uri: "kamelet:couchbase-sink"
+ parameters:
+ protocol: "http"
+ couchbaseHostname: "couchbase.example.com"
+ bucket: "shop"
+ scope: "sales"
+ collection: "orders"
+ autoStartId: false
+ username: "{{couchbase.username}}"
+ password: "{{couchbase.password}}"
+----
+
+=== Example Deleting Documents
+
+With `operation` set to `CCB_DEL`, each message deletes the document whose ID
it carries in the `CamelCouchbaseId`
+header, and the body is not used. Also set `autoStartId` to false, or the
Kamelet deletes the documents `1`, `2`, `3`
+and so on. Deleting a document that does not exist fails the message with
`Document with the given id not found`.
+
+[source,yaml,subs='+attributes,macros']
+----
+- to:
+ uri: "kamelet:couchbase-sink"
+ parameters:
+ protocol: "http"
+ couchbaseHostname: "couchbase.example.com"
+ bucket: "shop"
+ scope: "sales"
+ collection: "orders"
+ autoStartId: false
+ operation: "CCB_DEL"
+ username: "{{couchbase.username}}"
+ password: "{{couchbase.password}}"
+----
+
+=== Durability and Timeouts
+
+`persistTo` and `replicateTo` make each write wait until the document is saved
to disk, or copied to replicas, on that
+many nodes. The bucket must have the replicas: if it does not, the document is
still stored, but the message fails with
+`Not enough replicas configured on the bucket`. They do not apply to `CCB_DEL`.
+
+A write or delete fails after `retryAttempts` × `retryPause` milliseconds, 2 ×
5000 = 10 seconds by default, or after
+`operationTimeout` when that is longer. For example, `retryAttempts: 1`,
`retryPause: 1000` and `operationTimeout: 4000`
+make a write that cannot be done fail after 4 seconds, trying again every
second. `connectTimeout` is how long one attempt
+to connect to a node may take, 30 seconds by default.
diff --git a/docs/modules/ROOT/partials/couchbase-source-description.adoc
b/docs/modules/ROOT/partials/couchbase-source-description.adoc
index f4e2e8d75..8fc621907 100644
--- a/docs/modules/ROOT/partials/couchbase-source-description.adoc
+++ b/docs/modules/ROOT/partials/couchbase-source-description.adoc
@@ -10,8 +10,8 @@ poll. Set `useView` to true to poll a MapReduce view instead,
selected with
=== Writing the Statement
-The statement runs in the bucket scope, so the keyspace is the collection
rather than the
-bucket name - use `_default` for the default collection.
+The statement runs in the scope set by `scope`, or in the `_default` scope
when it is not set, so the
+keyspace is a collection of that scope rather than the bucket name - use
`_default` for the default collection.
It must also alias the document id as `__id`, because the consumer skips any
row without
that field. A query that omits it produces a source that polls without
emitting anything:
@@ -19,3 +19,41 @@ that field. A query that omits it produces a source that
polls without emitting
----
SELECT META().id AS __id, * FROM _default
----
+
+=== Example with a Scope and a Collection
+
+Without `statement`, set `scope` and `collection` to read all the documents of
that collection. Each row is
+`{"<collection>":<the document>,"__id":"<document ID>"}`. With
`processedStrategy` set to `delete`, each document is
+deleted from the collection once its message has been processed, so it is sent
only once. A message that fails keeps
+its document, and the next poll sends it again.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:couchbase-source"
+ parameters:
+ protocol: "http"
+ couchbaseHostname: "couchbase.example.com"
+ bucket: "shop"
+ scope: "sales"
+ collection: "inbox"
+ processedStrategy: "delete"
+ delay: 5000
+ username: "{{couchbase.username}}"
+ password: "{{couchbase.password}}"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+A `statement` runs in `scope` too, so it names the collection alone. With
`fullDocument` true, or `processedStrategy`
+`delete`, also set `collection` to the collection the statement reads: the
documents are read, or deleted, in
+`collection`, the default collection when it is not set.
+
+[source,yaml,subs='+attributes,macros']
+----
+scope: "sales"
+collection: "orders"
+statement: "SELECT META().id AS __id, o.* FROM orders o WHERE o.status = 'new'"
+----
diff --git a/kamelets/couchbase-sink.kamelet.yaml
b/kamelets/couchbase-sink.kamelet.yaml
index facde55da..14fd620bd 100644
--- a/kamelets/couchbase-sink.kamelet.yaml
+++ b/kamelets/couchbase-sink.kamelet.yaml
@@ -31,15 +31,15 @@ spec:
definition:
title: "Couchbase Sink"
description: |-
- Stores each message as a document in a Couchbase bucket, under a
document ID that the Kamelet counts up or that a header gives.
+ Stores each message as a document in a Couchbase bucket, or deletes a
document, under a document ID that the Kamelet counts up or that a header gives.
- Use it to save events, orders or other records as JSON documents in
Couchbase, for example each message that arrives on a Kafka topic. To read
documents, use `couchbase-source`.
+ Use it to save events, orders or other records as JSON documents in
Couchbase, for example each message that arrives on a Kafka topic, or to delete
the documents that messages name. To read documents, use `couchbase-source`.
- Input: the body is the document. JSON text is stored as a JSON document;
other text is stored as a JSON string. With `autoStartId` true, the default,
the documents get the IDs `1`, `2`, `3` and so on, counted from `startingId`,
and the `CamelCouchbaseId` header is ignored. With `autoStartId` false, the
`CamelCouchbaseId` header is the ID, and it is required.
+ Input: the body is the document. JSON text is stored as a JSON document;
other text is stored as a JSON string. With `autoStartId` true, the default,
the documents get the IDs `1`, `2`, `3` and so on, counted from `startingId`,
and the `CamelCouchbaseId` header is ignored. With `autoStartId` false, the
`CamelCouchbaseId` header is the ID; a message without it uses `key`, and fails
when `key` is not set either. With `operation` `CCB_DEL`, the body is not used.
- Output: the body afterwards is `true`.
+ Output: the body afterwards is `true`. After a delete, it is the result
of Couchbase as text, such as `MutationResult{cas=0x18e92f09b4bf0000, ...}`.
- Good to know: a document that already has the ID is replaced. The count
starts again at `startingId` each time the integration starts, so with
`autoStartId` true a restart overwrites the documents stored before. Set
`autoStartId` to false and give each message its own ID to avoid that. The
connection is not encrypted, also with `protocol` set to `https`.
+ Good to know: a document that already has the ID is replaced. The count
starts again at `startingId` each time the integration starts, so with
`autoStartId` true a restart overwrites the documents stored before. Set
`autoStartId` to false and give each message its own ID to avoid that. The
documents go to the default collection of the bucket unless `collection` is
set, in `scope`; a collection that does not exist makes each message wait and
fail with `AmbiguousTimeoutException` (10 [...]
required:
- protocol
- couchbaseHostname
@@ -63,7 +63,7 @@ spec:
default: 8091
bucket:
title: Bucket
- description: The name of the bucket to store the documents in, in its
default collection.
+ description: The name of the bucket to store the documents in. They go
to its default collection, unless `collection` is set.
type: string
example: orders
username:
@@ -91,15 +91,69 @@ spec:
example: couchbase://couchbase.example.com:11210
autoStartId:
title: Automatic IDs
- description: If true, the documents get the IDs `startingId`,
`startingId` + 1, and so on, and the `CamelCouchbaseId` header is ignored. If
false, each message must carry its document ID in the `CamelCouchbaseId` header.
+ description: If true, the documents get the IDs `startingId`,
`startingId` + 1, and so on, and the `CamelCouchbaseId` header and `key` are
ignored. If false, each message must carry its document ID in the
`CamelCouchbaseId` header, or `key` must be set.
type: boolean
default: true
+ key:
+ title: Document ID
+ description: The document ID to use for a message without the
`CamelCouchbaseId` header, when `autoStartId` is false. Every such message then
writes, or deletes, the same document. When it is not set, a message without
the header fails with `No document id`.
+ type: string
+ pattern: "^[^\\r\\n]{1,250}$"
+ example: latest-status
+ scope:
+ title: Scope
+ description: The scope of the bucket that holds `collection`. It is
only used together with `collection`. When it is not set, the `_default` scope
is used.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: sales
+ collection:
+ title: Collection
+ description: The collection to store the documents in, or delete them
from, in `scope`. The collection must exist; if it does not, each message waits
and then fails with `AmbiguousTimeoutException`. When it is not set, the
default collection of the bucket is used, and `scope` is ignored.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: orders
+ operation:
+ title: Operation
+ description: "What to do with each message: `CCB_PUT` stores the body
as the document, replacing a document with the same ID, and `CCB_DEL` deletes
the document with the ID. Deleting a document that does not exist fails the
message with `Document with the given id not found`. To delete, also set
`autoStartId` to false, or the Kamelet deletes the documents `startingId`,
`startingId` + 1, and so on. When it is not set, `CCB_PUT` is used."
+ type: string
+ enum: ["CCB_PUT", "CCB_DEL"]
+ example: CCB_DEL
+ persistTo:
+ title: Persist To
+ description: "On how many nodes a stored document must be saved to
disk before the message is done: 0 (not waited for), 1 (the active node), or 2,
3 or 4 (the active node and replicas). A number higher than the bucket can
give, such as 2 for a bucket without replicas, fails every message with `Not
enough replicas configured on the bucket`, but only after the document has been
stored. Not used with `operation` `CCB_DEL`. When it is not set, 0 is used."
+ type: integer
+ enum: [0, 1, 2, 3, 4]
+ replicateTo:
+ title: Replicate To
+ description: "To how many replicas a stored document must be copied
before the message is done, from 0 to 3. The bucket must be set up with that
many replicas, on as many other nodes. Otherwise every message fails, such as
with `Not enough replicas configured on the bucket`, but only after the
document has been stored. Not used with `operation` `CCB_DEL`. When it is not
set, 0 is used."
+ type: integer
+ enum: [0, 1, 2, 3]
+ connectTimeout:
+ title: Connect Timeout
+ description: How long one attempt to connect to a node of the cluster
may take, in milliseconds, before the Kamelet tries again. A message that waits
for the connection still fails after the time that `retryAttempts`,
`retryPause` and `operationTimeout` give. When it is not set, 30000 (30
seconds) is used.
+ type: integer
+ example: 10000
+ retryAttempts:
+ title: Retry Attempts
+ description: "Together with `retryPause`, how long a write or delete
may take: it fails after `retryAttempts` × `retryPause` milliseconds, or after
`operationTimeout` if that is longer. Within that time, a write or delete that
fails for a reason that can pass, such as a collection that is not known yet or
a bucket that is still opening, is tried again every `retryPause` milliseconds.
When it is not set, 2 is used, which with the default `retryPause` gives 10
seconds."
+ type: integer
+ example: 3
+ retryPause:
+ title: Retry Pause
+ description: How long to wait before trying a write or delete again,
in milliseconds. It also sets how long a write or delete may take; see
`retryAttempts`. When it is not set, 5000 (5 seconds) is used.
+ type: integer
+ example: 1000
+ operationTimeout:
+ title: Operation Timeout
+ description: How long a write or delete may take, in milliseconds,
when it is longer than `retryAttempts` × `retryPause`. When it is not set, 2500
is used, so the time from `retryAttempts` and `retryPause`, 10 seconds by
default, applies.
+ type: integer
+ example: 30000
dataTypes:
in:
headers:
CamelCouchbaseId:
title: Document ID
- description: The ID of the document to store. Required when
`autoStartId` is false; ignored when it is true.
+ description: The ID of the document to store or delete. Used when
`autoStartId` is false, and then required unless `key` is set; ignored when it
is true.
type: string
dependencies:
- "camel:couchbase"
@@ -120,3 +174,13 @@ spec:
password: "RAW({{password}})"
startingIdForInsertsFrom: "{{startingId}}"
username: "RAW({{username}})"
+ key: "RAW({{?key}})"
+ scope: "RAW({{?scope}})"
+ collection: "RAW({{?collection}})"
+ operation: "{{?operation}}"
+ persistTo: "{{?persistTo}}"
+ replicateTo: "{{?replicateTo}}"
+ connectTimeout: "{{?connectTimeout}}"
+ producerRetryAttempts: "{{?retryAttempts}}"
+ producerRetryPause: "{{?retryPause}}"
+ queryTimeout: "{{?operationTimeout}}"
diff --git a/kamelets/couchbase-source.kamelet.yaml
b/kamelets/couchbase-source.kamelet.yaml
index afe82d709..3a17340c5 100644
--- a/kamelets/couchbase-source.kamelet.yaml
+++ b/kamelets/couchbase-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
Use it to pass the documents of a Couchbase bucket on to another system,
for example to copy them to a search index or a data warehouse. To store
documents, use `couchbase-sink`.
- Output: the body is one result row as JSON text. Without `statement`,
the query reads all the documents of the default collection, and a row is
`{"<bucket>":<the document>,"__id":"<document ID>"}`. The `CamelCouchbaseId`
header holds the ID of the document.
+ Output: the body is one result row as JSON text. Without `statement`,
the query reads all the documents of the default collection, and a row is
`{"<bucket>":<the document>,"__id":"<document ID>"}`. With `collection` set, it
reads that collection of `scope`, and a row is `{"<collection>":<the
document>,"__id":"<document ID>"}`. The `CamelCouchbaseId` header holds the ID
of the document.
- Good to know: the query runs again every `delay` milliseconds and
nothing marks a document as read, so the same documents are sent on every run.
A `statement` must name the document ID `__id` (`SELECT META().id AS __id,
...`), or its rows are skipped. With `fullDocument` set to true, the body is an
object of the Couchbase SDK instead of JSON text. The connection is not
encrypted, also with `protocol` set to `https`.
+ Good to know: the query runs again every `delay` milliseconds and,
unless `processedStrategy` is `delete`, nothing marks a document as read, so
the same documents are sent on every run. With `delete`, each document is
deleted once its message has been processed. A `statement` must name the
document ID `__id` (`SELECT META().id AS __id, ...`), or its rows are skipped.
With `fullDocument` set to true, the body is an object of the Couchbase SDK
instead of JSON text. The connection is [...]
required:
- protocol
- couchbaseHostname
@@ -61,7 +61,7 @@ spec:
default: 8091
bucket:
title: Bucket
- description: The name of the bucket to read the documents from.
+ description: The name of the bucket to read the documents from. They
are read from its default collection, unless `collection` is set.
type: string
example: orders
username:
@@ -84,7 +84,7 @@ spec:
example: couchbase://couchbase.example.com:11210
statement:
title: SQL++ Statement
- description: The SQL++ (N1QL) query to run on each poll, instead of
reading all documents. It runs in the scope of the bucket, so name the
collection, such as `_default`, not the bucket. It must return the document ID
as `__id`, or its rows are skipped. Not used when `useView` is true.
+ description: The SQL++ (N1QL) query to run on each poll, instead of
reading all documents. It runs in `scope`, the `_default` scope of the bucket
when `scope` is not set, so name a collection of that scope, such as `_default`
or `orders`, not the bucket. It must return the document ID as `__id`, or its
rows are skipped. Not used when `useView` is true.
type: string
example: "SELECT META().id AS __id, * FROM _default"
useView:
@@ -127,6 +127,24 @@ spec:
description: How long to wait after one poll before the next, in
milliseconds.
type: integer
default: 500
+ scope:
+ title: Scope
+ description: The scope of the bucket to read from. A `statement` runs
in this scope, so a collection named in it is one of this scope. When it is not
set, the `_default` scope is used.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: sales
+ collection:
+ title: Collection
+ description: The collection to read from, in `scope`. Without
`statement`, the query reads all its documents. With `fullDocument` true or
`processedStrategy` `delete`, the documents are read or deleted in this
collection, also when a `statement` reads another one. When it is not set, the
default collection of the bucket is used.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: orders
+ processedStrategy:
+ title: Processed Strategy
+ description: "What to do with a document once its message has been
processed: `none` leaves it, so the next poll sends it again, and `delete`
deletes it from `collection`. A message that fails keeps its document, so a
later poll sends it again. When it is not set, `none` is used."
+ type: string
+ enum: ["none", "delete"]
+ example: delete
dataTypes:
out:
headers:
@@ -157,6 +175,9 @@ spec:
useView: "{{useView}}"
username: "RAW({{username}})"
viewName: "{{viewName}}"
+ scope: "RAW({{?scope}})"
+ collection: "RAW({{?collection}})"
+ consumerProcessedStrategy: "{{?processedStrategy}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 54f4121fa..93a103e17 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -544,8 +544,8 @@
"type" : "sink",
"supportLevel" : "Stable",
"title" : "Couchbase Sink",
- "summary" : "Stores each message as a document in a Couchbase bucket,
under a document ID that the Kamelet counts up or that a header gives.",
- "description" : "Stores each message as a document in a Couchbase bucket,
under a document ID that the Kamelet counts up or that a header gives.\n\nUse
it to save events, orders or other records as JSON documents in Couchbase, for
example each message that arrives on a Kafka topic. To read documents, use
`couchbase-source`.\n\nInput: the body is the document. JSON text is stored as
a JSON document; other text is stored as a JSON string. With `autoStartId`
true, the default, the docum [...]
+ "summary" : "Stores each message as a document in a Couchbase bucket, or
deletes a document, under a document ID that the Kamelet counts up or that a
header gives.",
+ "description" : "Stores each message as a document in a Couchbase bucket,
or deletes a document, under a document ID that the Kamelet counts up or that a
header gives.\n\nUse it to save events, orders or other records as JSON
documents in Couchbase, for example each message that arrives on a Kafka topic,
or to delete the documents that messages name. To read documents, use
`couchbase-source`.\n\nInput: the body is the document. JSON text is stored as
a JSON document; other text is st [...]
"group" : "Couchbase",
"namespace" : "Nosql",
"provider" : "Apache Software Foundation"
@@ -555,7 +555,7 @@
"supportLevel" : "Preview",
"title" : "Couchbase Source",
"summary" : "Runs a SQL++ (N1QL) query on a Couchbase bucket again and
again, and sends each document or row it finds as a message, as JSON text.",
- "description" : "Runs a SQL++ (N1QL) query on a Couchbase bucket again and
again, and sends each document or row it finds as a message, as JSON
text.\n\nUse it to pass the documents of a Couchbase bucket on to another
system, for example to copy them to a search index or a data warehouse. To
store documents, use `couchbase-sink`.\n\nOutput: the body is one result row as
JSON text. Without `statement`, the query reads all the documents of the
default collection, and a row is `{\"<buck [...]
+ "description" : "Runs a SQL++ (N1QL) query on a Couchbase bucket again and
again, and sends each document or row it finds as a message, as JSON
text.\n\nUse it to pass the documents of a Couchbase bucket on to another
system, for example to copy them to a search index or a data warehouse. To
store documents, use `couchbase-sink`.\n\nOutput: the body is one result row as
JSON text. Without `statement`, the query reads all the documents of the
default collection, and a row is `{\"<buck [...]
"group" : "Couchbase",
"namespace" : "Nosql",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/couchbase-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/couchbase-sink.kamelet.yaml
index facde55da..14fd620bd 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/couchbase-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/couchbase-sink.kamelet.yaml
@@ -31,15 +31,15 @@ spec:
definition:
title: "Couchbase Sink"
description: |-
- Stores each message as a document in a Couchbase bucket, under a
document ID that the Kamelet counts up or that a header gives.
+ Stores each message as a document in a Couchbase bucket, or deletes a
document, under a document ID that the Kamelet counts up or that a header gives.
- Use it to save events, orders or other records as JSON documents in
Couchbase, for example each message that arrives on a Kafka topic. To read
documents, use `couchbase-source`.
+ Use it to save events, orders or other records as JSON documents in
Couchbase, for example each message that arrives on a Kafka topic, or to delete
the documents that messages name. To read documents, use `couchbase-source`.
- Input: the body is the document. JSON text is stored as a JSON document;
other text is stored as a JSON string. With `autoStartId` true, the default,
the documents get the IDs `1`, `2`, `3` and so on, counted from `startingId`,
and the `CamelCouchbaseId` header is ignored. With `autoStartId` false, the
`CamelCouchbaseId` header is the ID, and it is required.
+ Input: the body is the document. JSON text is stored as a JSON document;
other text is stored as a JSON string. With `autoStartId` true, the default,
the documents get the IDs `1`, `2`, `3` and so on, counted from `startingId`,
and the `CamelCouchbaseId` header is ignored. With `autoStartId` false, the
`CamelCouchbaseId` header is the ID; a message without it uses `key`, and fails
when `key` is not set either. With `operation` `CCB_DEL`, the body is not used.
- Output: the body afterwards is `true`.
+ Output: the body afterwards is `true`. After a delete, it is the result
of Couchbase as text, such as `MutationResult{cas=0x18e92f09b4bf0000, ...}`.
- Good to know: a document that already has the ID is replaced. The count
starts again at `startingId` each time the integration starts, so with
`autoStartId` true a restart overwrites the documents stored before. Set
`autoStartId` to false and give each message its own ID to avoid that. The
connection is not encrypted, also with `protocol` set to `https`.
+ Good to know: a document that already has the ID is replaced. The count
starts again at `startingId` each time the integration starts, so with
`autoStartId` true a restart overwrites the documents stored before. Set
`autoStartId` to false and give each message its own ID to avoid that. The
documents go to the default collection of the bucket unless `collection` is
set, in `scope`; a collection that does not exist makes each message wait and
fail with `AmbiguousTimeoutException` (10 [...]
required:
- protocol
- couchbaseHostname
@@ -63,7 +63,7 @@ spec:
default: 8091
bucket:
title: Bucket
- description: The name of the bucket to store the documents in, in its
default collection.
+ description: The name of the bucket to store the documents in. They go
to its default collection, unless `collection` is set.
type: string
example: orders
username:
@@ -91,15 +91,69 @@ spec:
example: couchbase://couchbase.example.com:11210
autoStartId:
title: Automatic IDs
- description: If true, the documents get the IDs `startingId`,
`startingId` + 1, and so on, and the `CamelCouchbaseId` header is ignored. If
false, each message must carry its document ID in the `CamelCouchbaseId` header.
+ description: If true, the documents get the IDs `startingId`,
`startingId` + 1, and so on, and the `CamelCouchbaseId` header and `key` are
ignored. If false, each message must carry its document ID in the
`CamelCouchbaseId` header, or `key` must be set.
type: boolean
default: true
+ key:
+ title: Document ID
+ description: The document ID to use for a message without the
`CamelCouchbaseId` header, when `autoStartId` is false. Every such message then
writes, or deletes, the same document. When it is not set, a message without
the header fails with `No document id`.
+ type: string
+ pattern: "^[^\\r\\n]{1,250}$"
+ example: latest-status
+ scope:
+ title: Scope
+ description: The scope of the bucket that holds `collection`. It is
only used together with `collection`. When it is not set, the `_default` scope
is used.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: sales
+ collection:
+ title: Collection
+ description: The collection to store the documents in, or delete them
from, in `scope`. The collection must exist; if it does not, each message waits
and then fails with `AmbiguousTimeoutException`. When it is not set, the
default collection of the bucket is used, and `scope` is ignored.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: orders
+ operation:
+ title: Operation
+ description: "What to do with each message: `CCB_PUT` stores the body
as the document, replacing a document with the same ID, and `CCB_DEL` deletes
the document with the ID. Deleting a document that does not exist fails the
message with `Document with the given id not found`. To delete, also set
`autoStartId` to false, or the Kamelet deletes the documents `startingId`,
`startingId` + 1, and so on. When it is not set, `CCB_PUT` is used."
+ type: string
+ enum: ["CCB_PUT", "CCB_DEL"]
+ example: CCB_DEL
+ persistTo:
+ title: Persist To
+ description: "On how many nodes a stored document must be saved to
disk before the message is done: 0 (not waited for), 1 (the active node), or 2,
3 or 4 (the active node and replicas). A number higher than the bucket can
give, such as 2 for a bucket without replicas, fails every message with `Not
enough replicas configured on the bucket`, but only after the document has been
stored. Not used with `operation` `CCB_DEL`. When it is not set, 0 is used."
+ type: integer
+ enum: [0, 1, 2, 3, 4]
+ replicateTo:
+ title: Replicate To
+ description: "To how many replicas a stored document must be copied
before the message is done, from 0 to 3. The bucket must be set up with that
many replicas, on as many other nodes. Otherwise every message fails, such as
with `Not enough replicas configured on the bucket`, but only after the
document has been stored. Not used with `operation` `CCB_DEL`. When it is not
set, 0 is used."
+ type: integer
+ enum: [0, 1, 2, 3]
+ connectTimeout:
+ title: Connect Timeout
+ description: How long one attempt to connect to a node of the cluster
may take, in milliseconds, before the Kamelet tries again. A message that waits
for the connection still fails after the time that `retryAttempts`,
`retryPause` and `operationTimeout` give. When it is not set, 30000 (30
seconds) is used.
+ type: integer
+ example: 10000
+ retryAttempts:
+ title: Retry Attempts
+ description: "Together with `retryPause`, how long a write or delete
may take: it fails after `retryAttempts` × `retryPause` milliseconds, or after
`operationTimeout` if that is longer. Within that time, a write or delete that
fails for a reason that can pass, such as a collection that is not known yet or
a bucket that is still opening, is tried again every `retryPause` milliseconds.
When it is not set, 2 is used, which with the default `retryPause` gives 10
seconds."
+ type: integer
+ example: 3
+ retryPause:
+ title: Retry Pause
+ description: How long to wait before trying a write or delete again,
in milliseconds. It also sets how long a write or delete may take; see
`retryAttempts`. When it is not set, 5000 (5 seconds) is used.
+ type: integer
+ example: 1000
+ operationTimeout:
+ title: Operation Timeout
+ description: How long a write or delete may take, in milliseconds,
when it is longer than `retryAttempts` × `retryPause`. When it is not set, 2500
is used, so the time from `retryAttempts` and `retryPause`, 10 seconds by
default, applies.
+ type: integer
+ example: 30000
dataTypes:
in:
headers:
CamelCouchbaseId:
title: Document ID
- description: The ID of the document to store. Required when
`autoStartId` is false; ignored when it is true.
+ description: The ID of the document to store or delete. Used when
`autoStartId` is false, and then required unless `key` is set; ignored when it
is true.
type: string
dependencies:
- "camel:couchbase"
@@ -120,3 +174,13 @@ spec:
password: "RAW({{password}})"
startingIdForInsertsFrom: "{{startingId}}"
username: "RAW({{username}})"
+ key: "RAW({{?key}})"
+ scope: "RAW({{?scope}})"
+ collection: "RAW({{?collection}})"
+ operation: "{{?operation}}"
+ persistTo: "{{?persistTo}}"
+ replicateTo: "{{?replicateTo}}"
+ connectTimeout: "{{?connectTimeout}}"
+ producerRetryAttempts: "{{?retryAttempts}}"
+ producerRetryPause: "{{?retryPause}}"
+ queryTimeout: "{{?operationTimeout}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/couchbase-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/couchbase-source.kamelet.yaml
index afe82d709..3a17340c5 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/couchbase-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/couchbase-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
Use it to pass the documents of a Couchbase bucket on to another system,
for example to copy them to a search index or a data warehouse. To store
documents, use `couchbase-sink`.
- Output: the body is one result row as JSON text. Without `statement`,
the query reads all the documents of the default collection, and a row is
`{"<bucket>":<the document>,"__id":"<document ID>"}`. The `CamelCouchbaseId`
header holds the ID of the document.
+ Output: the body is one result row as JSON text. Without `statement`,
the query reads all the documents of the default collection, and a row is
`{"<bucket>":<the document>,"__id":"<document ID>"}`. With `collection` set, it
reads that collection of `scope`, and a row is `{"<collection>":<the
document>,"__id":"<document ID>"}`. The `CamelCouchbaseId` header holds the ID
of the document.
- Good to know: the query runs again every `delay` milliseconds and
nothing marks a document as read, so the same documents are sent on every run.
A `statement` must name the document ID `__id` (`SELECT META().id AS __id,
...`), or its rows are skipped. With `fullDocument` set to true, the body is an
object of the Couchbase SDK instead of JSON text. The connection is not
encrypted, also with `protocol` set to `https`.
+ Good to know: the query runs again every `delay` milliseconds and,
unless `processedStrategy` is `delete`, nothing marks a document as read, so
the same documents are sent on every run. With `delete`, each document is
deleted once its message has been processed. A `statement` must name the
document ID `__id` (`SELECT META().id AS __id, ...`), or its rows are skipped.
With `fullDocument` set to true, the body is an object of the Couchbase SDK
instead of JSON text. The connection is [...]
required:
- protocol
- couchbaseHostname
@@ -61,7 +61,7 @@ spec:
default: 8091
bucket:
title: Bucket
- description: The name of the bucket to read the documents from.
+ description: The name of the bucket to read the documents from. They
are read from its default collection, unless `collection` is set.
type: string
example: orders
username:
@@ -84,7 +84,7 @@ spec:
example: couchbase://couchbase.example.com:11210
statement:
title: SQL++ Statement
- description: The SQL++ (N1QL) query to run on each poll, instead of
reading all documents. It runs in the scope of the bucket, so name the
collection, such as `_default`, not the bucket. It must return the document ID
as `__id`, or its rows are skipped. Not used when `useView` is true.
+ description: The SQL++ (N1QL) query to run on each poll, instead of
reading all documents. It runs in `scope`, the `_default` scope of the bucket
when `scope` is not set, so name a collection of that scope, such as `_default`
or `orders`, not the bucket. It must return the document ID as `__id`, or its
rows are skipped. Not used when `useView` is true.
type: string
example: "SELECT META().id AS __id, * FROM _default"
useView:
@@ -127,6 +127,24 @@ spec:
description: How long to wait after one poll before the next, in
milliseconds.
type: integer
default: 500
+ scope:
+ title: Scope
+ description: The scope of the bucket to read from. A `statement` runs
in this scope, so a collection named in it is one of this scope. When it is not
set, the `_default` scope is used.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: sales
+ collection:
+ title: Collection
+ description: The collection to read from, in `scope`. Without
`statement`, the query reads all its documents. With `fullDocument` true or
`processedStrategy` `delete`, the documents are read or deleted in this
collection, also when a `statement` reads another one. When it is not set, the
default collection of the bucket is used.
+ type: string
+ pattern: "^[A-Za-z0-9_%-]{1,251}$"
+ example: orders
+ processedStrategy:
+ title: Processed Strategy
+ description: "What to do with a document once its message has been
processed: `none` leaves it, so the next poll sends it again, and `delete`
deletes it from `collection`. A message that fails keeps its document, so a
later poll sends it again. When it is not set, `none` is used."
+ type: string
+ enum: ["none", "delete"]
+ example: delete
dataTypes:
out:
headers:
@@ -157,6 +175,9 @@ spec:
useView: "{{useView}}"
username: "RAW({{username}})"
viewName: "{{viewName}}"
+ scope: "RAW({{?scope}})"
+ collection: "RAW({{?collection}})"
+ consumerProcessedStrategy: "{{?processedStrategy}}"
steps:
- to:
uri: kamelet:sink