davsclaus opened a new pull request, #3218:
URL: https://github.com/apache/camel-kamelets/pull/3218

   Part of #3170: the Azure Event Hubs Kamelets take a connection string, the 
source can save its position more or less often and the sink can send with a 
partition key; the Cosmos DB sink can upsert and create its database and 
container; the Cosmos DB source takes change feed options and a fixed host 
name; and the Azure Functions sink can send its key in the `x-functions-key` 
header.
   
   ### The options
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | `azure-eventhubs-source` | `connectionString` | 
`connectionString=RAW({{?connectionString}})` | Logs in with a connection 
string instead of `namespaceName`, `eventhubName`, `sharedAccessName` and 
`sharedAccessKey` |
   | `azure-eventhubs-source` | `checkpointBatchSize` | 
`checkpointBatchSize={{?checkpointBatchSize}}` | Saves the position of a 
partition after this many events (500 when not set) |
   | `azure-eventhubs-source` | `checkpointBatchTimeout` | 
`checkpointBatchTimeout={{?checkpointBatchTimeout}}` | ...or this many 
milliseconds after the first event since the last checkpoint (5000 when not 
set) |
   | `azure-eventhubs-sink` | `connectionString` | 
`connectionString=RAW({{?connectionString}})` | As for the source |
   | `azure-eventhubs-sink` | `partitionKey` | 
`partitionKey=RAW({{?partitionKey}})` | Sends all the events with this 
partition key, so they go to one partition, in order |
   | `azure-cosmosdb-sink` | `operation` | `operation: "{{operation}}"` | 
`createItem` (default, as today) or `upsertItem` |
   | `azure-cosmosdb-sink` | `createDatabaseIfNotExists` | 
`createDatabaseIfNotExists: "{{?createDatabaseIfNotExists}}"`, and the 
`CamelAzureCosmosDbCreateDatabaseIfNotExist` header (see below) | Creates the 
database when it is missing |
   | `azure-cosmosdb-sink` | `createContainerIfNotExists` | 
`createContainerIfNotExists: "{{?createContainerIfNotExists}}"`, and the 
`CamelAzureCosmosDbCreateContainerIfNotExist` header | Creates the container 
when it is missing |
   | `azure-cosmosdb-sink` | `containerPartitionKeyPath` | 
`containerPartitionKeyPath: "{{?containerPartitionKeyPath}}"` | The partition 
key of a container it creates, such as `/region` |
   | `azure-cosmosdb-source` | `hostName` | `hostName: "{{?hostName}}"` | The 
name the source marks its leases with |
   | `azure-cosmosdb-source` | `startFromBeginning`, `maxItemCount`, 
`feedPollDelay`, `leasePrefix` | properties of a `ChangeFeedProcessorOptions` 
template bean, as `{{?...}}`; `changeFeedProcessorOptions: 
"#{{changeFeedOptions}}"` | Read all items on the first start; items per 
message; how long to wait between checks; a lease prefix of its own |
   | `azure-functions-sink` | `keyInHeader` | a `choice` with `precondition: 
true`: when true, `setHeader x-functions-key` = `{{key}}`, `vertx-http` without 
`code`, then `removeHeader x-functions-key` | Sends the key in the 
`x-functions-key` header instead of `?code=` |
   
   - **Defaults:** all the new options are optional and have no `default`, 
except `operation`, which needs one: the endpoint must have an operation, and 
`createItem` is the one the Kamelet always used. When the options are unset, 
the endpoint URIs are the same as `main`'s, with one exception on the Cosmos DB 
source that does not change behaviour, explained below.
   - **Connection strings:** `connectionString` is `format: password` with the 
credentials descriptor, passed in `RAW()`. When it is set, the component does 
not use the path, so `namespaceName` and `eventhubName` are no longer required, 
and the template uses `{{?namespaceName}}/{{?eventhubName}}`. With neither, the 
integration fails to start with `ConnectionString, ProducerAsyncClient or 
Namespace and EventHub name must be set`. The connection string must have 
`EntityPath=<event hub>`: the one of a namespace policy, as the Azure portal 
shows it, does not, and then the integration fails to start with 
`'eventHubName' cannot be an empty string`. The descriptions say so.
   - **`partitionKey` and `partition-id`:** the component takes only one of a 
partition id and a partition key, so with `partitionKey` set, a message with a 
`partition-id` header fails with `Both partitionKey and partitionId are set`. 
The descriptions say so.
   - **`createDatabaseIfNotExists` and `createContainerIfNotExists`:** the 
Cosmos DB producer does not use these endpoint options: with only the endpoint 
options set, the database was not created (see the table). The consumer uses 
them correctly. The template therefore also passes them in the form the 
producer reads, `CamelAzureCosmosDbCreateDatabaseIfNotExist` and 
`CamelAzureCosmosDbCreateContainerIfNotExist`, in `choice` blocks with 
`precondition: true` that are removed when the option is not set. This needs a 
fix in Camel (a separate issue); the endpoint options are kept so that the 
Kamelet works the same once it is fixed.
   - **Change feed options:** the component takes them only as a 
`ChangeFeedProcessorOptions` bean. The template creates one, with the four 
options as optional properties, and passes it as `changeFeedProcessorOptions`. 
So the endpoint URI of the source now always has 
`changeFeedProcessorOptions=#changeFeedOptions-1`. With the options unset, this 
does not change behaviour: the Cosmos SDK uses `new 
ChangeFeedProcessorOptions()` when none is given, and that is what the bean is. 
One thing it does change: a `changeFeedProcessorOptions` set on the 
`azure-cosmosdb` component 
(`camel.component.azure-cosmosdb.change-feed-processor-options`) is now 
overridden by the Kamelet's. The four options are the replacement for that. The 
lease intervals and `startTime` are not offered.
   - **`hostName`:** without it, the source takes a random name at each start. 
Its leases stay owned by the previous name after a stop or a crash, so a 
restarted source receives nothing until they expire, about a minute. With a 
fixed name it continues at once. The description says to give each copy that 
runs at the same time its own name.
   - **`keyInHeader`:** when it is not set, or false, the `otherwise` branch is 
the endpoint of `main`, unchanged, and the key is sent as `?code=`. When it is 
true, the address of the request, which error messages show, has no key, and 
the header is removed from the message after a successful call. `keyInHeader` 
with no `key` fails to start with `Property with key [key] not found`.
   - **`eventPosition`** is still not offered, as in #3193.
   - **Partial pages:** a new section with an example for each Kamelet.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT on an arm64 Mac, with:
   - the Azure Event Hubs emulator with Azurite, with an event hub `eh1` of 2 
partitions. The Kamelets now reach it with their own `connectionString` 
(`...;UseDevelopmentEmulator=true;EntityPath=eh1`). The source still needs a 
test-only `BlobCheckpointStore` on Azurite set on the component, as in #3193, 
because the blob address is fixed to `https://<account>.blob.core.windows.net`. 
The checkpoint blobs were listed in Azurite with their sequence numbers.
   - the Cosmos DB emulator, `vnext-preview`, with `--protocol https`. It 
supports only the gateway connection mode, and the component builds its client 
in direct mode, so a test-only gateway-mode client was set on the 
`azure-cosmosdb` component. The Kamelets ran unchanged, and `camel get 
endpoint` showed the URIs they build.
   - a small HTTPS server in Python for the Functions sink, which answers 200 
when the key comes as `code` or as `x-functions-key`, like Azure Functions, and 
logs each request. The integration trusted its certificate through 
`camel.ssl.*` and `useGlobalSslContextParameters`.
   
   | Kamelet | Case | Result |
   |---|---|---|
   | all five | Options unset, Kamelets from `main` and from this PR | Same 
endpoint URIs, except `changeFeedProcessorOptions=#changeFeedOptions-1` on the 
Cosmos DB source, described above. Functions sink with and without `key`: 
identical (`...?code=plainKey123%3D%3D` and `.../api/anon`) |
   | eventhubs-source | Only `connectionString` (no namespace, event hub or 
policy) | Received the events of both partitions; URI 
`azure-eventhubs://?...&connectionString=RAW(...)` |
   | eventhubs-source | Connection string without `EntityPath` | Fails to 
start: `'eventHubName' cannot be an empty string` |
   | eventhubs-source | Neither `connectionString` nor `namespaceName` | Fails 
to start: `ConnectionString, ProducerAsyncClient or Namespace and EventHub name 
must be set` |
   | eventhubs-source | Checkpoint options unset, 5 events to partition 0 | No 
checkpoint 3 s after the send; checkpoint at the last event 8 s after |
   | eventhubs-source | `checkpointBatchSize: 2`, `checkpointBatchTimeout: 
60000`, 5 events | Checkpoint at once at the 4th event (seq 10), and the 5th 
(seq 11) still not saved 14 s later |
   | eventhubs-source | `checkpointBatchTimeout: 1000` | Checkpoint at the last 
event within 1 s |
   | eventhubs-source | `checkpointBatchTimeout: 20000` | No checkpoint at 14 
s; checkpoint at 20 s |
   | eventhubs-sink | Only `connectionString`, 6 messages | 3 events in each 
partition |
   | eventhubs-sink | `connectionString` with wrong `namespaceName`, 
`eventhubName`, `sharedAccessName` and `sharedAccessKey` | Sent: the connection 
string is used and the others are not |
   | eventhubs-sink | Connection string without `EntityPath` | Fails to start: 
`'eventHubName' cannot be an empty string` |
   | eventhubs-sink | `partitionKey: a`, `b`, `c`, 6 messages each | All 6 in 
one partition each time (`b` in 0, `a` and `c` in 1) |
   | eventhubs-sink | `partitionKey: "a+b&c d%20(x)"` | Received with the 
partition key `a+b&c d%20(x)`, as written |
   | eventhubs-sink | `partitionKey` and a `partition-id: 0` header | Each 
message fails: `Both partitionKey and partitionId are set` |
   | eventhubs-sink | `partition-id: 0` header, no `partitionKey` | All 6 in 
partition 0, as before |
   | cosmosdb-sink | Options unset, database missing | Fails: `Collection 
'orders' not found in database 'shop2'` |
   | cosmosdb-sink | Both create options as endpoint options only (before the 
workaround) | Database not created; fails the same way |
   | cosmosdb-sink | `createDatabaseIfNotExists: true` only | Database created; 
fails: container not found |
   | cosmosdb-sink | Both create options, no `containerPartitionKeyPath` | 
Fails: `containerPartitionKeyPath cannot be empty!` |
   | cosmosdb-sink | Both create options, `containerPartitionKeyPath: /region` 
| Database, container and item created; output body is the item with `_rid`, 
`_etag` |
   | cosmosdb-sink | New container with `/customer`, then an item with another 
partition key value | Fails: `PartitionKey extracted from document doesn't 
match`, so the container has `/customer` |
   | cosmosdb-sink | `containerPartitionKeyPath: /other` on an existing 
`/region` container | Stored; the path is not used |
   | cosmosdb-sink | Same `id` again, `operation` unset | Fails: HTTP 409 `The 
document already exists in the collection.` |
   | cosmosdb-sink | Same `id`, `operation: upsertItem`, `v: 2` | Replaced; 
output body has `v=2` |
   | cosmosdb-sink | `upsertItem` with a new `id`; `createItem` set explicitly 
| Both created |
   | cosmosdb-source | Options unset, new lease container | Received only new 
changes (and the latest existing item, which the emulator returns on a first 
start) |
   | cosmosdb-source | `startFromBeginning: true`, new lease container | All 4 
existing items, in one message |
   | cosmosdb-source | `startFromBeginning: true`, lease container with a saved 
position | Only the changes since that position |
   | cosmosdb-source | `startFromBeginning: true`, `maxItemCount: 2` | Two 
messages of 2 items |
   | cosmosdb-source | `feedPollDelay: 200`, unset, `30000`; 3 items written 7 
s apart | Received within 0.2 s; within 5 s; all 3 together 30 s after the 
previous check |
   | cosmosdb-source | `leasePrefix: search-` and `cache-`, same lease 
container | Both received the new item; lease documents 
`search-localhost_shop2_orders..0` and `cache-...` |
   | cosmosdb-source | No `leasePrefix`, two integrations, same lease container 
| Only one received the new item |
   | cosmosdb-source | Unset `hostName`, `kill -9`, restart, write an item | 
Lease owner is a random name (`GnZtWffVMx`); the item arrived 45 s later |
   | cosmosdb-source | `hostName: node-1`, `kill -9`, restart, write an item | 
The item arrived 3 s later |
   | functions-sink | `keyInHeader: true`, key `plainKey123==` | Server got 
`x-functions-key: plainKey123==` and no query; 200, output body the response; 
no `x-functions-key` header on the message afterwards |
   | functions-sink | `keyInHeader: true`, function answers 500 | Fails with 
`HTTP operation failed invoking https://localhost:48443/api/fail`: no key in 
the message |
   | functions-sink | `keyInHeader: false`, and unset | Key sent as 
`?code=plainKey123%3D%3D`; 200 |
   | functions-sink | `keyInHeader: true` without `key` | Fails to start: 
`Property with key [key] not found` |
   | functions-sink | No `key` | No `code` parameter, as on `main` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the five Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to