davsclaus opened a new pull request, #3220:
URL: https://github.com/apache/camel-kamelets/pull/3220

   Part of #3170: `couchbase-sink` and `couchbase-source` work with a scope and 
a collection, `couchbase-sink` can delete documents and takes durability and 
timeout settings, and `couchbase-source` can delete each document once it has 
been sent.
   
   ### The problem
   
   Both Kamelets only worked on the default collection of the bucket. The sink 
could only store documents, with the Couchbase defaults for durability and 
timeouts, and the source sent the same documents again on every poll.
   
   ### The options
   
   `couchbase-sink`:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `scope` | `scope: "RAW({{?scope}})"` | The scope that holds `collection`. 
Used only together with `collection`. |
   | `collection` | `collection: "RAW({{?collection}})"` | The collection to 
store the documents in, or delete them from. |
   | `key` | `key: "RAW({{?key}})"` | The document ID for a message without the 
`CamelCouchbaseId` header, when `autoStartId` is false. |
   | `operation` | `operation: "{{?operation}}"` | `CCB_PUT` stores the body 
(as before), `CCB_DEL` deletes the document. |
   | `persistTo` | `persistTo: "{{?persistTo}}"` | On how many nodes a stored 
document must be on disk, 0 to 4. |
   | `replicateTo` | `replicateTo: "{{?replicateTo}}"` | To how many replicas a 
stored document must be copied, 0 to 3. |
   | `connectTimeout` | `connectTimeout: "{{?connectTimeout}}"` | How long one 
attempt to connect to a node may take, in ms. |
   | `retryAttempts` | `producerRetryAttempts: "{{?retryAttempts}}"` | With 
`retryPause`, how long a write or delete may take. |
   | `retryPause` | `producerRetryPause: "{{?retryPause}}"` | The pause between 
two tries of a write or delete, in ms. |
   | `operationTimeout` | `queryTimeout: "{{?operationTimeout}}"` | How long a 
write or delete may take, when that is longer than `retryAttempts` × 
`retryPause`. |
   
   `couchbase-source`:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `scope` | `scope: "RAW({{?scope}})"` | The scope the query runs in. |
   | `collection` | `collection: "RAW({{?collection}})"` | The collection to 
read when there is no `statement`, and where `fullDocument` and `delete` read 
and delete the documents. |
   | `processedStrategy` | `consumerProcessedStrategy: 
"{{?processedStrategy}}"` | `none` (as before) or `delete`: each document is 
deleted once its message has been processed. |
   
   - **Defaults:** all options are optional and have no `default`, as in #3174. 
When they are unset, the endpoint URIs are the same as on `main`.
   - **`RAW()` and `pattern:`:** `scope`, `collection` and `key` are passed in 
`RAW()`, so a `+` or a `%` is kept. A collection named `p%2B-x_1` and a key 
`latest status+1%` were written as named. `scope` and `collection` have the 
pattern of Couchbase names, `^[A-Za-z0-9_%-]{1,251}$`, and `key` has 
`^[^\r\n]{1,250}$`.
   - **Names:** `timeout` is reserved by the `kamelet:` component. The Kamelet 
uses `operationTimeout`, `retryAttempts` and `retryPause` instead of the 
component names `queryTimeout`, `producerRetryAttempts` and 
`producerRetryPause`. For the sink, `queryTimeout` is not a query timeout: the 
component uses max(`producerRetryAttempts` × `producerRetryPause`, 
`queryTimeout`) as the timeout of a write or delete, and retries every 
`producerRetryPause` ms. The descriptions say how the three options work 
together.
   - **`operation`:** the enum is `CCB_PUT` and `CCB_DEL`. `CCB_GET` is left 
out because it puts an object of the Couchbase SDK in the body, which is of no 
use in a sink.
   - **Things the descriptions now say, found in the tests:**
     - `scope` without `collection` is ignored, and the default collection is 
used.
     - A collection that does not exist makes each message wait until the write 
timeout, 10 s by default, and fail with `AmbiguousTimeoutException`.
     - `CCB_DEL` with `autoStartId` true, the default, deletes the documents 
`1`, `2`, and so on.
     - Deleting a document that does not exist fails with `Document with the 
given id not found`.
     - `persistTo` and `replicateTo` above what the bucket has fail the message 
with `Not enough replicas configured on the bucket`, but the document is stored 
first.
     - `persistTo` and `replicateTo` are not used by `CCB_DEL`.
     - With `fullDocument` or `delete`, the source reads or deletes the 
documents in `collection`, even when a `statement` reads another collection.
   - **Left out of the source:** `connectTimeout` and `queryTimeout`. The 
component applies `queryTimeout` to SQL++ queries only when it is not 2500 ms. 
Otherwise the SDK uses its own 75 s, so the option would behave in a surprising 
way. `rangeStartKey`, `rangeEndKey`, `consumerRetryPause` and the scheduler 
options were also left out; `delay` and `limit` already cover the polling.
   - **Partial pages:** the sink page has examples for a scope and a collection 
and for deleting, and a section on durability and timeouts. The source page has 
an example for a scope, a collection and `processedStrategy: delete`.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against 
`couchbase/server:community-7.6.2` in Docker. The setup was a single node with 
data, index and query services and the bucket `shop` with 0 replicas. It had 
the scope `sales` with the collections `orders`, `inbox` and `p%2B-x_1`, all 
with primary indexes, set up with `couchbase-cli` and SQL++. The results were 
checked with SQL++ queries on the server.
   
   | Case | Result |
   |---|---|
   | Options unset, both Kamelets, `main` and this branch | `camel get 
endpoint` shows identical `couchbase:` URIs |
   | Sink, `scope: sales`, `collection: orders`, `autoStartId: false`, header 
`CamelCouchbaseId` `order-1..3` | `order-1`, `order-2`, `order-3` in 
`shop.sales.orders`, default collection empty |
   | Sink, `operation: CCB_DEL`, header `order-2` | `order-2` removed; 
`order-1` and `order-3` kept. Body afterwards `MutationResult{cas=..., ...}` |
   | Sink, `CCB_DEL` of a document that does not exist | Fails: 
`DocumentNotFoundException: Document with the given id not found` |
   | Sink, `CCB_DEL` with `autoStartId` unset (true) | Deletes document `1` of 
the default collection; `2` kept |
   | Sink, `operation: CCB_PUT` | Stored as without the option |
   | Sink, no header, `key: "latest status+1%"` | Document `latest status+1%` 
stored |
   | Sink, header `A-101` and `key: fallback` (header from `jq: .orderId`, as 
in the partial) | Stored as `A-101`; the header wins |
   | Sink, no header, no `key`, `autoStartId: false` | Fails: `No document id: 
set the CamelCouchbaseId header or the key option of the endpoint` |
   | Sink, `scope: sales` only | Stored in the default collection |
   | Sink, `collection: orders` only (no such collection in `_default`) | Fails 
after 10 s: `AmbiguousTimeoutException`, retry reason `COLLECTION_NOT_FOUND` |
   | Sink, `collection: "p%2B-x_1"` | Stored in that collection; the name is 
kept by `RAW()` |
   | Sink, `persistTo: 1` | Stored, message OK |
   | Sink, `persistTo: 2` / `replicateTo: 1`, bucket without replicas | Fails: 
`ReplicaNotConfiguredException: Not enough replicas configured on the bucket`; 
the document is stored |
   | Sink, `CCB_DEL` with `persistTo: 2`, `replicateTo: 1` | Deleted, message 
OK (not used by delete) |
   | Sink, missing collection, options unset | Fails after 10.0 s, `timeoutMs` 
10000, retried 2 times |
   | The same, `retryAttempts: 1`, `retryPause: 1000` | Fails after 2.5 s, 
`timeoutMs` 2500, retried 3 times |
   | The same plus `operationTimeout: 4000` | Fails after 4.0 s, `timeoutMs` 
4000, retried 4 times |
   | The same, `retryAttempts: 3`, `retryPause: 2000` | Fails after 6.0 s, 
`timeoutMs` 6000, retried 3 times |
   | Sink to an unreachable host, `connectTimeout` unset / `2000`, 
`operationTimeout: 25000` | Connect attempts time out after 30000 ms / 2000 ms 
(`EndpointConnectionFailedEvent`); both messages fail after 25 s |
   | Source, `scope: sales`, `collection: orders`, no statement | Sends each 
document of `sales.orders` as `{"orders":{...},"__id":"..."}` |
   | Source, `scope: sales`, `statement: SELECT ... FROM orders o WHERE ...` | 
The statement runs in `sales`; matching rows sent |
   | Source, `scope`, `collection`, `fullDocument: true` (with and without a 
statement) | Body is the document read from `sales.orders` |
   | Source, `scope: sales`, statement on `inbox`, `fullDocument: true`, no 
`collection` | Poll fails with `DocumentNotFoundException`: the document is 
read from the default collection |
   | Source, `collection: inbox`, `processedStrategy: delete`, a route that 
fails for `in-bad` | `in-1`, `in-2` sent once and deleted; `in-bad` sent on 
every poll and kept |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the two Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`. The Citrus test `couchbase-source-route` was not run; it 
uses none of the new options.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to