davsclaus opened a new pull request, #3218:
URL: https://github.com/apache/camel-kamelets/pull/3218
Part of #3170: the Azure Event Hubs Kamelets take a connection string, the
source can save its position more or less often and the sink can send with a
partition key; the Cosmos DB sink can upsert and create its database and
container; the Cosmos DB source takes change feed options and a fixed host
name; and the Azure Functions sink can send its key in the `x-functions-key`
header.
### The options
| Kamelet | Option | Template mapping | What it does |
|---|---|---|---|
| `azure-eventhubs-source` | `connectionString` |
`connectionString=RAW({{?connectionString}})` | Logs in with a connection
string instead of `namespaceName`, `eventhubName`, `sharedAccessName` and
`sharedAccessKey` |
| `azure-eventhubs-source` | `checkpointBatchSize` |
`checkpointBatchSize={{?checkpointBatchSize}}` | Saves the position of a
partition after this many events (500 when not set) |
| `azure-eventhubs-source` | `checkpointBatchTimeout` |
`checkpointBatchTimeout={{?checkpointBatchTimeout}}` | ...or this many
milliseconds after the first event since the last checkpoint (5000 when not
set) |
| `azure-eventhubs-sink` | `connectionString` |
`connectionString=RAW({{?connectionString}})` | As for the source |
| `azure-eventhubs-sink` | `partitionKey` |
`partitionKey=RAW({{?partitionKey}})` | Sends all the events with this
partition key, so they go to one partition, in order |
| `azure-cosmosdb-sink` | `operation` | `operation: "{{operation}}"` |
`createItem` (default, as today) or `upsertItem` |
| `azure-cosmosdb-sink` | `createDatabaseIfNotExists` |
`createDatabaseIfNotExists: "{{?createDatabaseIfNotExists}}"`, and the
`CamelAzureCosmosDbCreateDatabaseIfNotExist` header (see below) | Creates the
database when it is missing |
| `azure-cosmosdb-sink` | `createContainerIfNotExists` |
`createContainerIfNotExists: "{{?createContainerIfNotExists}}"`, and the
`CamelAzureCosmosDbCreateContainerIfNotExist` header | Creates the container
when it is missing |
| `azure-cosmosdb-sink` | `containerPartitionKeyPath` |
`containerPartitionKeyPath: "{{?containerPartitionKeyPath}}"` | The partition
key of a container it creates, such as `/region` |
| `azure-cosmosdb-source` | `hostName` | `hostName: "{{?hostName}}"` | The
name the source marks its leases with |
| `azure-cosmosdb-source` | `startFromBeginning`, `maxItemCount`,
`feedPollDelay`, `leasePrefix` | properties of a `ChangeFeedProcessorOptions`
template bean, as `{{?...}}`; `changeFeedProcessorOptions:
"#{{changeFeedOptions}}"` | Read all items on the first start; items per
message; how long to wait between checks; a lease prefix of its own |
| `azure-functions-sink` | `keyInHeader` | a `choice` with `precondition:
true`: when true, `setHeader x-functions-key` = `{{key}}`, `vertx-http` without
`code`, then `removeHeader x-functions-key` | Sends the key in the
`x-functions-key` header instead of `?code=` |
- **Defaults:** all the new options are optional and have no `default`,
except `operation`, which needs one: the endpoint must have an operation, and
`createItem` is the one the Kamelet always used. When the options are unset,
the endpoint URIs are the same as `main`'s, with one exception on the Cosmos DB
source that does not change behaviour, explained below.
- **Connection strings:** `connectionString` is `format: password` with the
credentials descriptor, passed in `RAW()`. When it is set, the component does
not use the path, so `namespaceName` and `eventhubName` are no longer required,
and the template uses `{{?namespaceName}}/{{?eventhubName}}`. With neither, the
integration fails to start with `ConnectionString, ProducerAsyncClient or
Namespace and EventHub name must be set`. The connection string must have
`EntityPath=<event hub>`: the one of a namespace policy, as the Azure portal
shows it, does not, and then the integration fails to start with
`'eventHubName' cannot be an empty string`. The descriptions say so.
- **`partitionKey` and `partition-id`:** the component takes only one of a
partition id and a partition key, so with `partitionKey` set, a message with a
`partition-id` header fails with `Both partitionKey and partitionId are set`.
The descriptions say so.
- **`createDatabaseIfNotExists` and `createContainerIfNotExists`:** the
Cosmos DB producer does not use these endpoint options: with only the endpoint
options set, the database was not created (see the table). The consumer uses
them correctly. The template therefore also passes them in the form the
producer reads, `CamelAzureCosmosDbCreateDatabaseIfNotExist` and
`CamelAzureCosmosDbCreateContainerIfNotExist`, in `choice` blocks with
`precondition: true` that are removed when the option is not set. This needs a
fix in Camel (a separate issue); the endpoint options are kept so that the
Kamelet works the same once it is fixed.
- **Change feed options:** the component takes them only as a
`ChangeFeedProcessorOptions` bean. The template creates one, with the four
options as optional properties, and passes it as `changeFeedProcessorOptions`.
So the endpoint URI of the source now always has
`changeFeedProcessorOptions=#changeFeedOptions-1`. With the options unset, this
does not change behaviour: the Cosmos SDK uses `new
ChangeFeedProcessorOptions()` when none is given, and that is what the bean is.
One thing it does change: a `changeFeedProcessorOptions` set on the
`azure-cosmosdb` component
(`camel.component.azure-cosmosdb.change-feed-processor-options`) is now
overridden by the Kamelet's. The four options are the replacement for that. The
lease intervals and `startTime` are not offered.
- **`hostName`:** without it, the source takes a random name at each start.
Its leases stay owned by the previous name after a stop or a crash, so a
restarted source receives nothing until they expire, about a minute. With a
fixed name it continues at once. The description says to give each copy that
runs at the same time its own name.
- **`keyInHeader`:** when it is not set, or false, the `otherwise` branch is
the endpoint of `main`, unchanged, and the key is sent as `?code=`. When it is
true, the address of the request, which error messages show, has no key, and
the header is removed from the message after a successful call. `keyInHeader`
with no `key` fails to start with `Property with key [key] not found`.
- **`eventPosition`** is still not offered, as in #3193.
- **Partial pages:** a new section with an example for each Kamelet.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT on an arm64 Mac, with:
- the Azure Event Hubs emulator with Azurite, with an event hub `eh1` of 2
partitions. The Kamelets now reach it with their own `connectionString`
(`...;UseDevelopmentEmulator=true;EntityPath=eh1`). The source still needs a
test-only `BlobCheckpointStore` on Azurite set on the component, as in #3193,
because the blob address is fixed to `https://<account>.blob.core.windows.net`.
The checkpoint blobs were listed in Azurite with their sequence numbers.
- the Cosmos DB emulator, `vnext-preview`, with `--protocol https`. It
supports only the gateway connection mode, and the component builds its client
in direct mode, so a test-only gateway-mode client was set on the
`azure-cosmosdb` component. The Kamelets ran unchanged, and `camel get
endpoint` showed the URIs they build.
- a small HTTPS server in Python for the Functions sink, which answers 200
when the key comes as `code` or as `x-functions-key`, like Azure Functions, and
logs each request. The integration trusted its certificate through
`camel.ssl.*` and `useGlobalSslContextParameters`.
| Kamelet | Case | Result |
|---|---|---|
| all five | Options unset, Kamelets from `main` and from this PR | Same
endpoint URIs, except `changeFeedProcessorOptions=#changeFeedOptions-1` on the
Cosmos DB source, described above. Functions sink with and without `key`:
identical (`...?code=plainKey123%3D%3D` and `.../api/anon`) |
| eventhubs-source | Only `connectionString` (no namespace, event hub or
policy) | Received the events of both partitions; URI
`azure-eventhubs://?...&connectionString=RAW(...)` |
| eventhubs-source | Connection string without `EntityPath` | Fails to
start: `'eventHubName' cannot be an empty string` |
| eventhubs-source | Neither `connectionString` nor `namespaceName` | Fails
to start: `ConnectionString, ProducerAsyncClient or Namespace and EventHub name
must be set` |
| eventhubs-source | Checkpoint options unset, 5 events to partition 0 | No
checkpoint 3 s after the send; checkpoint at the last event 8 s after |
| eventhubs-source | `checkpointBatchSize: 2`, `checkpointBatchTimeout:
60000`, 5 events | Checkpoint at once at the 4th event (seq 10), and the 5th
(seq 11) still not saved 14 s later |
| eventhubs-source | `checkpointBatchTimeout: 1000` | Checkpoint at the last
event within 1 s |
| eventhubs-source | `checkpointBatchTimeout: 20000` | No checkpoint at 14
s; checkpoint at 20 s |
| eventhubs-sink | Only `connectionString`, 6 messages | 3 events in each
partition |
| eventhubs-sink | `connectionString` with wrong `namespaceName`,
`eventhubName`, `sharedAccessName` and `sharedAccessKey` | Sent: the connection
string is used and the others are not |
| eventhubs-sink | Connection string without `EntityPath` | Fails to start:
`'eventHubName' cannot be an empty string` |
| eventhubs-sink | `partitionKey: a`, `b`, `c`, 6 messages each | All 6 in
one partition each time (`b` in 0, `a` and `c` in 1) |
| eventhubs-sink | `partitionKey: "a+b&c d%20(x)"` | Received with the
partition key `a+b&c d%20(x)`, as written |
| eventhubs-sink | `partitionKey` and a `partition-id: 0` header | Each
message fails: `Both partitionKey and partitionId are set` |
| eventhubs-sink | `partition-id: 0` header, no `partitionKey` | All 6 in
partition 0, as before |
| cosmosdb-sink | Options unset, database missing | Fails: `Collection
'orders' not found in database 'shop2'` |
| cosmosdb-sink | Both create options as endpoint options only (before the
workaround) | Database not created; fails the same way |
| cosmosdb-sink | `createDatabaseIfNotExists: true` only | Database created;
fails: container not found |
| cosmosdb-sink | Both create options, no `containerPartitionKeyPath` |
Fails: `containerPartitionKeyPath cannot be empty!` |
| cosmosdb-sink | Both create options, `containerPartitionKeyPath: /region`
| Database, container and item created; output body is the item with `_rid`,
`_etag` |
| cosmosdb-sink | New container with `/customer`, then an item with another
partition key value | Fails: `PartitionKey extracted from document doesn't
match`, so the container has `/customer` |
| cosmosdb-sink | `containerPartitionKeyPath: /other` on an existing
`/region` container | Stored; the path is not used |
| cosmosdb-sink | Same `id` again, `operation` unset | Fails: HTTP 409 `The
document already exists in the collection.` |
| cosmosdb-sink | Same `id`, `operation: upsertItem`, `v: 2` | Replaced;
output body has `v=2` |
| cosmosdb-sink | `upsertItem` with a new `id`; `createItem` set explicitly
| Both created |
| cosmosdb-source | Options unset, new lease container | Received only new
changes (and the latest existing item, which the emulator returns on a first
start) |
| cosmosdb-source | `startFromBeginning: true`, new lease container | All 4
existing items, in one message |
| cosmosdb-source | `startFromBeginning: true`, lease container with a saved
position | Only the changes since that position |
| cosmosdb-source | `startFromBeginning: true`, `maxItemCount: 2` | Two
messages of 2 items |
| cosmosdb-source | `feedPollDelay: 200`, unset, `30000`; 3 items written 7
s apart | Received within 0.2 s; within 5 s; all 3 together 30 s after the
previous check |
| cosmosdb-source | `leasePrefix: search-` and `cache-`, same lease
container | Both received the new item; lease documents
`search-localhost_shop2_orders..0` and `cache-...` |
| cosmosdb-source | No `leasePrefix`, two integrations, same lease container
| Only one received the new item |
| cosmosdb-source | Unset `hostName`, `kill -9`, restart, write an item |
Lease owner is a random name (`GnZtWffVMx`); the item arrived 45 s later |
| cosmosdb-source | `hostName: node-1`, `kill -9`, restart, write an item |
The item arrived 3 s later |
| functions-sink | `keyInHeader: true`, key `plainKey123==` | Server got
`x-functions-key: plainKey123==` and no query; 200, output body the response;
no `x-functions-key` header on the message afterwards |
| functions-sink | `keyInHeader: true`, function answers 500 | Fails with
`HTTP operation failed invoking https://localhost:48443/api/fail`: no key in
the message |
| functions-sink | `keyInHeader: false`, and unset | Key sent as
`?code=plainKey123%3D%3D`; 200 |
| functions-sink | `keyInHeader: true` without `key` | Fails to start:
`Property with key [key] not found` |
| functions-sink | No `key` | No `code` parameter, as on `main` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the five Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]