This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 8f7592afa chore: the Azure Event Hubs, Cosmos DB and Functions
Kamelets take more options (#3170) (#3218)
8f7592afa is described below
commit 8f7592afa0de6433bec492d437891b13a2fa5880
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:32:10 2026 +0100
chore: the Azure Event Hubs, Cosmos DB and Functions Kamelets take more
options (#3170) (#3218)
azure-eventhubs-source and azure-eventhubs-sink take an optional
connectionString, a secret passed in RAW(). The component ignores the
path when it is set, so namespaceName and eventhubName are no longer
required. The string must end with EntityPath=<event hub>, which the
descriptions say. The source takes checkpointBatchSize and
checkpointBatchTimeout, and the sink partitionKey (in RAW()); with
partitionKey set, a message with a partition-id header fails, as the
component does not take both.
azure-cosmosdb-sink takes operation (createItem, the default as before,
or upsertItem), createDatabaseIfNotExists, createContainerIfNotExists
and containerPartitionKeyPath. The producer does not use the two create
options on the endpoint, so the template also passes them in the form
the producer reads, in precondition choices that are removed when the
options are not set. The endpoint options are kept for when Camel is
fixed.
azure-cosmosdb-source takes hostName, so that a restarted source takes
back its leases at once instead of after they expire, and
startFromBeginning, maxItemCount, feedPollDelay and leasePrefix. The
component takes the last four only as a ChangeFeedProcessorOptions
bean, which the template now creates and always passes. With the
options unset it holds the SDK defaults, which the SDK uses when no
options are given, but it overrides options set on the component.
azure-functions-sink takes keyInHeader, to send the key in the
x-functions-key header instead of the code query parameter, so the key
is not in the address of the request that error messages show. The
header is removed after the call. When it is not set, the endpoint is
the same as before.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../partials/azure-cosmosdb-sink-description.adoc | 32 +++++++++++++
.../azure-cosmosdb-source-description.adoc | 31 +++++++++++++
.../partials/azure-eventhubs-sink-description.adoc | 29 +++++++++++-
.../azure-eventhubs-source-description.adoc | 31 +++++++++++++
.../partials/azure-functions-sink-description.adoc | 18 ++++++++
kamelets/azure-cosmosdb-sink.kamelet.yaml | 53 ++++++++++++++++++++--
kamelets/azure-cosmosdb-source.kamelet.yaml | 37 ++++++++++++++-
kamelets/azure-eventhubs-sink.kamelet.yaml | 25 ++++++----
kamelets/azure-eventhubs-source.kamelet.yaml | 27 ++++++++---
kamelets/azure-functions-sink.kamelet.yaml | 38 +++++++++++++---
.../main/resources/META-INF/kamelets-index.json | 12 ++---
.../kamelets/azure-cosmosdb-sink.kamelet.yaml | 53 ++++++++++++++++++++--
.../kamelets/azure-cosmosdb-source.kamelet.yaml | 37 ++++++++++++++-
.../kamelets/azure-eventhubs-sink.kamelet.yaml | 25 ++++++----
.../kamelets/azure-eventhubs-source.kamelet.yaml | 27 ++++++++---
.../kamelets/azure-functions-sink.kamelet.yaml | 38 +++++++++++++---
16 files changed, 456 insertions(+), 57 deletions(-)
diff --git a/docs/modules/ROOT/partials/azure-cosmosdb-sink-description.adoc
b/docs/modules/ROOT/partials/azure-cosmosdb-sink-description.adoc
index 64c376f76..25ed79603 100644
--- a/docs/modules/ROOT/partials/azure-cosmosdb-sink-description.adoc
+++ b/docs/modules/ROOT/partials/azure-cosmosdb-sink-description.adoc
@@ -24,3 +24,35 @@ For more information, see the
https://learn.microsoft.com/en-us/java/api/overvie
In the headers, you can optionally set the `itemPartitionKey` /
`ce-itemPartitionKey` property to specify the partition key for a specific item.
If you do not set the property in the header, you'll need to use the static
property `itemPartitionKey`. A message without a partition key fails.
+
+=== Creating or replacing items
+
+With `operation` `createItem`, the default, an item whose `id` already exists
in the same partition fails with the error `The document already exists`. With
`upsertItem`, the item is created, or replaces the existing item with the same
`id` and partition key value, as a whole.
+
+=== Creating the database and the container
+
+The database and the container must exist, unless `createDatabaseIfNotExists`
and `createContainerIfNotExists` are true. A new container takes
`containerPartitionKeyPath` as its partition key, such as `/region`; without
it, each message fails with the error `containerPartitionKeyPath cannot be
empty!`. For a container that exists, `containerPartitionKeyPath` is not used.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: "orders"
+ bootstrapServers: "kafka.example.com:9092"
+ saslAuthType: "NONE"
+ steps:
+ - to:
+ uri: "kamelet:azure-cosmosdb-sink"
+ parameters:
+ databaseEndpoint: "https://my-account.documents.azure.com:443/"
+ accountKey: "{{cosmos.key}}"
+ databaseName: "shop"
+ containerName: "orders"
+ itemPartitionKey: "EU"
+ operation: "upsertItem"
+ createDatabaseIfNotExists: true
+ createContainerIfNotExists: true
+ containerPartitionKeyPath: "/region"
+----
diff --git a/docs/modules/ROOT/partials/azure-cosmosdb-source-description.adoc
b/docs/modules/ROOT/partials/azure-cosmosdb-source-description.adoc
index 71597a017..ee5594826 100644
--- a/docs/modules/ROOT/partials/azure-cosmosdb-source-description.adoc
+++ b/docs/modules/ROOT/partials/azure-cosmosdb-source-description.adoc
@@ -18,3 +18,34 @@ The order of evaluation for `AZURE_IDENTITY` is the
following:
- Azure Powershell
For more information, see the
https://learn.microsoft.com/en-us/java/api/overview/azure/identity-readme[Azure
Identity documentation]
+
+=== Reading the change feed
+
+The Kamelet keeps its position in the lease container. These options change
how it reads:
+
+- `startFromBeginning`: if true, the first start reads all the items of the
container. Once a position is saved, the source continues from it.
+- `maxItemCount`: the largest number of items in one message, 100 by default.
+- `feedPollDelay`: how long to wait before checking again for new changes, in
milliseconds, 5000 by default. A smaller value receives changes sooner, with
more requests to Cosmos DB.
+- `leasePrefix`: two integrations that read the same container with the same
lease container share one position, and each change goes to only one of them.
Give each integration its own `leasePrefix` to have each receive every change.
+- `hostName`: the name the source marks its leases with. Without it, the
source takes a new random name at each start, and after a crash waits up to a
minute for the leases of its previous run to expire. With a name that stays the
same, it continues at once. Give each copy that runs at the same time its own
name.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:azure-cosmosdb-source"
+ parameters:
+ databaseEndpoint: "https://my-account.documents.azure.com:443/"
+ accountKey: "{{cosmos.key}}"
+ databaseName: "shop"
+ containerName: "orders"
+ createLeaseContainerIfNotExists: true
+ startFromBeginning: true
+ maxItemCount: 10
+ feedPollDelay: 1000
+ leasePrefix: "search-"
+ hostName: "orders-search-1"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
diff --git a/docs/modules/ROOT/partials/azure-eventhubs-sink-description.adoc
b/docs/modules/ROOT/partials/azure-eventhubs-sink-description.adoc
index 827a485ef..004e4b3b7 100644
--- a/docs/modules/ROOT/partials/azure-eventhubs-sink-description.adoc
+++ b/docs/modules/ROOT/partials/azure-eventhubs-sink-description.adoc
@@ -23,4 +23,31 @@ For more information, see the
https://learn.microsoft.com/en-us/java/api/overvie
In the headers, you can optionally set the `partition-id` / `ce-partition-id`
property to specify the partition id for a specific item.
-If you do not set the property in the header, Azure Event Hubs will do that
for you.
+If you set neither the header nor `partitionKey`, Azure Event Hubs chooses the
partition.
+
+=== Sending all events to the same partition
+
+Event Hubs keeps the order of the events only within a partition. To read the
events in the order they were sent, set `partitionKey`: Event Hubs sends all
the events with the same partition key to the same partition, which it chooses
from the key. Do not set the `partition-id` header as well: a message with both
fails with the error `Both partitionKey and partitionId are set`.
+
+=== Connecting with a connection string
+
+Instead of `namespaceName`, `eventhubName`, `sharedAccessName` and
`sharedAccessKey`, you can set `connectionString`. It must end with
`;EntityPath=` and the name of the event hub, which the connection string of a
policy of the namespace does not have: add it.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: "orders"
+ bootstrapServers: "kafka.example.com:9092"
+ saslAuthType: "NONE"
+ steps:
+ - to:
+ uri: "kamelet:azure-eventhubs-sink"
+ parameters:
+ connectionString:
"Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=send-policy;SharedAccessKey=...;EntityPath=orders"
+ partitionKey: "orders"
+----
+
+The connection string holds the key, so keep it in a secret, like
`sharedAccessKey`.
diff --git a/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
b/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
index 992b5468f..7d1d1542d 100644
--- a/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
+++ b/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
@@ -43,3 +43,34 @@ By default the Kamelet reads as the `$Default` consumer
group. To read the same
----
Each consumer group has its own saved position in `blobContainerName`, so the
integrations can share the container. A consumer group without a saved position
starts with the events that arrive after the integration starts. With a
consumer group that does not exist, the integration starts but receives
nothing, and logs a warning that the messaging entity could not be found.
+
+=== Connecting with a connection string
+
+Instead of `namespaceName`, `eventhubName`, `sharedAccessName` and
`sharedAccessKey`, you can set `connectionString`, as the Azure portal shows it
under the namespace or the event hub, then Settings, Shared access policies,
the policy. The connection string must end with `;EntityPath=` and the name of
the event hub. The connection string of a policy of the event hub has it; for
one of a policy of the namespace, add it. Without it, the integration fails to
start with the error `'eventHubN [...]
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:azure-eventhubs-source"
+ parameters:
+ connectionString:
"Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;EntityPath=orders"
+ blobAccountName: "mystorageaccount"
+ blobAccessKey: "{{storage.key}}"
+ blobContainerName: "eventhub-checkpoints"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+The connection string holds the key, so keep it in a secret, like
`sharedAccessKey`.
+
+=== How often the position is saved
+
+For each partition, the Kamelet saves how far it has read (a checkpoint) in
`blobContainerName` after `checkpointBatchSize` events, 500 by default, or
`checkpointBatchTimeout` milliseconds, 5000 by default, after the first event
since the last checkpoint, whichever comes first. After a restart, it reads
again the events that came after the last checkpoint. To read fewer events
again, save the position more often, at the cost of more writes to the storage
account:
+
+[source,yaml,subs='+attributes,macros']
+----
+ checkpointBatchSize: 100
+ checkpointBatchTimeout: 1000
+----
diff --git a/docs/modules/ROOT/partials/azure-functions-sink-description.adoc
b/docs/modules/ROOT/partials/azure-functions-sink-description.adoc
index 211e8958d..7804fa54e 100644
--- a/docs/modules/ROOT/partials/azure-functions-sink-description.adoc
+++ b/docs/modules/ROOT/partials/azure-functions-sink-description.adoc
@@ -3,3 +3,21 @@
=== Authentication methods
In this Kamelet you should access the function on Azure through a key token in
the key parameter.
+
+By default the key is sent as the `code` query parameter of the request. With
`keyInHeader` set to true, it is sent in the `x-functions-key` header instead,
so the address of the request, which error messages show, does not hold the key:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "hello"
+ steps:
+ - to:
+ uri: "kamelet:azure-functions-sink"
+ parameters:
+ url:
"https://azure-function-demo-12234.azurewebsites.net/api/httpexample"
+ key: "{{function.key}}"
+ keyInHeader: true
+----
diff --git a/kamelets/azure-cosmosdb-sink.kamelet.yaml
b/kamelets/azure-cosmosdb-sink.kamelet.yaml
index feaabf3e8..0aada5704 100644
--- a/kamelets/azure-cosmosdb-sink.kamelet.yaml
+++ b/kamelets/azure-cosmosdb-sink.kamelet.yaml
@@ -32,7 +32,7 @@ spec:
definition:
title: "Azure CosmosDB Sink"
description: |-
- Creates an item (a JSON document) in a container of Azure Cosmos DB for
each message, from the JSON object in the body.
+ Creates or replaces an item (a JSON document) in a container of Azure
Cosmos DB for each message, from the JSON object in the body.
Use it to store orders, events or profiles as JSON documents in Cosmos
DB. To receive the changes made to a container, use `azure-cosmosdb-source`.
@@ -40,7 +40,7 @@ spec:
Output: the body afterwards is the item as it was created, as a Map,
with the fields that Cosmos DB adds.
- Good to know: a message without a partition key value fails, and so does
an item whose `id` already exists in the same partition: the sink creates items
and never replaces them.
+ Good to know: a message without a partition key value fails. With
`operation` `createItem`, the default, an item whose `id` already exists in the
same partition fails; with `upsertItem`, it replaces the existing item. The
database and the container must exist, unless `createDatabaseIfNotExists` and
`createContainerIfNotExists` are true; a new container needs
`containerPartitionKeyPath`.
required:
- databaseName
- containerName
@@ -57,6 +57,26 @@ spec:
description: The name of the container in the database to create the
items in.
type: string
example: orders
+ operation:
+ title: Operation
+ description: What to do with each item. `createItem` creates the item,
and fails when an item with the same `id` already exists in the same partition.
`upsertItem` creates the item, or replaces the existing item with the same `id`
in the same partition with it, as a whole.
+ type: string
+ default: createItem
+ enum: ["createItem", "upsertItem"]
+ createDatabaseIfNotExists:
+ title: Autocreate Database
+ description: If true, the database `databaseName` is created when it
does not exist. When not set, or false, a missing database fails each message.
+ type: boolean
+ createContainerIfNotExists:
+ title: Autocreate Container
+ description: If true, the container `containerName` is created when it
does not exist, with `containerPartitionKeyPath` as its partition key; then
`containerPartitionKeyPath` is required. When not set, or false, a missing
container fails each message.
+ type: boolean
+ containerPartitionKeyPath:
+ title: Container Partition Key Path
+ description: The partition key of the container to create when
`createContainerIfNotExists` is true, as the path of a field of the items, such
as `/region`. Not used for a container that exists.
+ type: string
+ pattern: ^/\S+$
+ example: /region
accountKey:
title: Account Key
description: "The primary or secondary key of the Azure Cosmos DB
account, from the Azure portal: the account, then Settings, Keys. Used when
`credentialType` is `SHARED_ACCOUNT_KEY`, the default."
@@ -129,6 +149,30 @@ spec:
expression:
simple:
expression:
"${header[ce-itemPartitionKey]}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${propertiesExist:createDatabaseIfNotExists}"
+ steps:
+ - setHeader:
+ name: CamelAzureCosmosDbCreateDatabaseIfNotExist
+ expression:
+ constant:
+ expression: "{{createDatabaseIfNotExists}}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${propertiesExist:createContainerIfNotExists}"
+ steps:
+ - setHeader:
+ name: CamelAzureCosmosDbCreateContainerIfNotExist
+ expression:
+ constant:
+ expression: "{{createContainerIfNotExists}}"
- unmarshal:
json: {}
- to:
@@ -140,5 +184,8 @@ spec:
credentialType: "{{credentialType}}"
databaseEndpoint: "{{databaseEndpoint}}"
itemPartitionKey: "{{?itemPartitionKey}}"
- operation: createItem
+ operation: "{{operation}}"
+ createDatabaseIfNotExists: "{{?createDatabaseIfNotExists}}"
+ createContainerIfNotExists: "{{?createContainerIfNotExists}}"
+ containerPartitionKeyPath: "{{?containerPartitionKeyPath}}"
diff --git a/kamelets/azure-cosmosdb-source.kamelet.yaml
b/kamelets/azure-cosmosdb-source.kamelet.yaml
index 7b6a417dc..8bc5cd51d 100644
--- a/kamelets/azure-cosmosdb-source.kamelet.yaml
+++ b/kamelets/azure-cosmosdb-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is a JSON array of the items of one batch, as bytes,
each item with the fields that Cosmos DB adds, such as `_etag`.
- Good to know: it reads only the changes made after it first starts, and
it does not see deleted items. It keeps its position in a lease container,
`camel-lease` in the database `databaseName` unless set otherwise, which must
exist unless `createLeaseContainerIfNotExists` is true.
+ Good to know: it reads only the changes made after it first starts,
unless `startFromBeginning` is true, and it does not see deleted items. It
keeps its position in a lease container, `camel-lease` in the database
`databaseName` unless set otherwise, which must exist unless
`createLeaseContainerIfNotExists` is true. Two integrations that read the same
container with the same lease container share one position, and each change
goes to only one of them, unless each has its own `lease [...]
required:
- databaseName
- containerName
@@ -93,6 +93,31 @@ spec:
type: string
default: "SHARED_ACCOUNT_KEY"
enum: ["SHARED_ACCOUNT_KEY", "AZURE_IDENTITY"]
+ hostName:
+ title: Host Name
+ description: A name for this running copy of the integration, which
the source writes into the lease container to mark the leases it holds. When it
is not set, the source takes a new random name at each start, so after a crash
it cannot take back the leases of its previous run until they expire, and
receives no changes for up to a minute. With a name that stays the same across
restarts, it continues at once. Give each copy that runs at the same time its
own name.
+ type: string
+ pattern: ^\S+$
+ example: orders-sync-1
+ startFromBeginning:
+ title: Start From Beginning
+ description: If true, the first start reads all the items of the
container, from the oldest change, instead of only the changes made after it
starts. Used only when the lease container holds no position yet for this
source; afterwards the source continues from its saved position.
+ type: boolean
+ maxItemCount:
+ title: Max Items Per Batch
+ description: The largest number of items in one message, the JSON
array of a batch. When not set, 100.
+ type: integer
+ example: 10
+ feedPollDelay:
+ title: Feed Poll Delay
+ description: How long to wait before checking again for new changes,
when there were none, in milliseconds. A smaller value receives changes sooner,
but makes more requests to Cosmos DB, which use request units. When not set,
5000 (5 seconds).
+ type: integer
+ example: 1000
+ leasePrefix:
+ title: Lease Prefix
+ description: A prefix for the names of the leases of this source in
the lease container, such as `search-`. Give each integration that reads the
same container its own prefix, so that they can share a lease container and
each receives every change. Changing the prefix of a source starts it again
without a saved position.
+ type: string
+ example: search-
dataTypes:
out:
default: json
@@ -134,6 +159,14 @@ spec:
- "camel:kamelet"
- "camel:jackson3"
template:
+ beans:
+ - name: changeFeedOptions
+ type: "#class:com.azure.cosmos.models.ChangeFeedProcessorOptions"
+ properties:
+ startFromBeginning: "{{?startFromBeginning}}"
+ maxItemCount: "{{?maxItemCount}}"
+ feedPollDelay: "{{?feedPollDelay}}"
+ leasePrefix: "{{?leasePrefix}}"
from:
uri: azure-cosmosdb
parameters:
@@ -146,6 +179,8 @@ spec:
databaseEndpoint: "{{databaseEndpoint}}"
leaseContainerName: "{{?leaseContainerName}}"
leaseDatabaseName: "{{?leaseDatabaseName}}"
+ hostName: "{{?hostName}}"
+ changeFeedProcessorOptions: "#{{changeFeedOptions}}"
steps:
- marshal:
json: {}
diff --git a/kamelets/azure-eventhubs-sink.kamelet.yaml
b/kamelets/azure-eventhubs-sink.kamelet.yaml
index 00d61b302..d35ef2840 100644
--- a/kamelets/azure-eventhubs-sink.kamelet.yaml
+++ b/kamelets/azure-eventhubs-sink.kamelet.yaml
@@ -36,24 +36,21 @@ spec:
Use it to publish events, such as clicks, sensor readings or changes,
for stream processing or for other applications to read. To receive them, use
`azure-eventhubs-source`.
- Input: the body becomes the body of the event: text, bytes or a stream.
A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. A List is sent as one event per element. The optional
`partition-id` header (or `ce-partition-id`) is the partition to send to, such
as `0`; without it, Event Hubs chooses one.
+ Input: the body becomes the body of the event: text, bytes or a stream.
A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. A List is sent as one event per element. The optional
`partition-id` header (or `ce-partition-id`) is the partition to send to, such
as `0`; without it, and without `partitionKey`, Event Hubs chooses one.
Output: the message is unchanged.
- Good to know: the shared access policy needs the Send right.
- required:
- - namespaceName
- - eventhubName
+ Good to know: log in with `sharedAccessName` and `sharedAccessKey`, with
`connectionString`, or with `credentialType` `AZURE_IDENTITY`. Set
`namespaceName` and `eventhubName` unless `connectionString` is set. The shared
access policy needs the Send right. With `partitionKey` set, a message with a
`partition-id` or `ce-partition-id` header fails.
type: object
properties:
namespaceName:
title: Event Hubs Namespace
- description: The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`.
+ description: "The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`. Required unless
`connectionString` is set: with neither, the integration fails to start with
the error `ConnectionString, ProducerAsyncClient or Namespace and EventHub name
must be set`."
type: string
example: my-namespace
eventhubName:
title: Event Hub Name
- description: The name of the event hub in the namespace to send the
events to.
+ description: The name of the event hub in the namespace to send the
events to. Required unless `connectionString` is set.
type: string
example: orders
sharedAccessName:
@@ -70,12 +67,24 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ connectionString:
+ title: Connection String
+ description: "A connection string of a shared access policy, such as
`Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;EntityPath=orders`,
from the Azure portal: the namespace or the event hub, then Settings, Shared
access policies, the policy, Connection string. Use it instead of
`namespaceName`, `eventhubName`, `sharedAccessName` and `sharedAccessKey`,
which are then not used. It must end with `;EntityPath=` an [...]
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
credentialType:
title: Credential Type
description: How to log in to Event Hubs. `CONNECTION_STRING`, the
default, uses `sharedAccessName` and `sharedAccessKey`. `AZURE_IDENTITY` uses
the Azure identity of the environment, such as the `AZURE_CLIENT_ID`,
`AZURE_TENANT_ID` and `AZURE_CLIENT_SECRET` environment variables, a managed
identity or the Azure CLI login.
type: string
default: "CONNECTION_STRING"
enum: ["CONNECTION_STRING", "AZURE_IDENTITY"]
+ partitionKey:
+ title: Partition Key
+ description: A partition key for all the events, such as `orders`.
Event Hubs sends all the events with the same partition key to the same
partition, which it chooses from the key, so they are read in the order they
were sent. Use it instead of the `partition-id` header, as a message with both
fails. When not set, each event goes to the partition in its `partition-id`
header, or Event Hubs chooses one.
+ type: string
+ example: orders
dataTypes:
in:
headers:
@@ -115,4 +124,4 @@ spec:
simple:
expression: "${header[ce-partition-id]}"
- to:
- uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?credentialType={{credentialType}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
+ uri:
"azure-eventhubs://{{?namespaceName}}/{{?eventhubName}}?credentialType={{credentialType}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}&connectionString=RAW({{?connectionString}})&partitionKey=RAW({{?partitionKey}})"
diff --git a/kamelets/azure-eventhubs-source.kamelet.yaml
b/kamelets/azure-eventhubs-source.kamelet.yaml
index cc3cb9799..5ed66eb62 100644
--- a/kamelets/azure-eventhubs-source.kamelet.yaml
+++ b/kamelets/azure-eventhubs-source.kamelet.yaml
@@ -38,10 +38,8 @@ spec:
Output: the body is the body of the event, as bytes. The headers give
its partition (`CamelAzureEventHubsPartitionId`), offset, sequence number and
the time it was enqueued, and `CamelAzureEventHubsMetadata` holds the custom
properties of the event.
- Good to know: when no position has been saved yet, it starts with the
events that arrive after it starts, not the older ones. It reads as the
`$Default` consumer group, or as the one in `consumerGroupName`, and each
consumer group has its own saved position. It saves its position (a checkpoint)
in `blobContainerName` every 500 events or every 5 seconds, so after a restart
some events can be read again.
+ Good to know: log in with `sharedAccessName` and `sharedAccessKey`, with
`connectionString`, or with `credentialType` `AZURE_IDENTITY`. Set
`namespaceName` and `eventhubName` unless `connectionString` is set. When no
position has been saved yet, it starts with the events that arrive after it
starts, not the older ones. It reads as the `$Default` consumer group, or as
the one in `consumerGroupName`, and each consumer group has its own saved
position. For each partition, it saves its [...]
required:
- - namespaceName
- - eventhubName
- blobAccountName
- blobAccessKey
- blobContainerName
@@ -49,12 +47,12 @@ spec:
properties:
namespaceName:
title: Event Hubs Namespace
- description: The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`.
+ description: "The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`. Required unless
`connectionString` is set: with neither, the integration fails to start with
the error `ConnectionString, ProducerAsyncClient or Namespace and EventHub name
must be set`."
type: string
example: my-namespace
eventhubName:
title: Event Hub Name
- description: The name of the event hub in the namespace to receive the
events from.
+ description: The name of the event hub in the namespace to receive the
events from. Required unless `connectionString` is set.
type: string
example: orders
consumerGroupName:
@@ -76,6 +74,13 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ connectionString:
+ title: Connection String
+ description: "A connection string of a shared access policy, such as
`Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;EntityPath=orders`,
from the Azure portal: the namespace or the event hub, then Settings, Shared
access policies, the policy, Connection string. Use it instead of
`namespaceName`, `eventhubName`, `sharedAccessName` and `sharedAccessKey`,
which are then not used. It must end with `;EntityPath=` an [...]
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
blobAccountName:
title: Checkpoint Storage Account
description: The name of the storage account where the source saves
how far it has read each partition (its checkpoints).
@@ -99,6 +104,16 @@ spec:
type: string
default: "CONNECTION_STRING"
enum: ["CONNECTION_STRING", "AZURE_IDENTITY"]
+ checkpointBatchSize:
+ title: Checkpoint Batch Size
+ description: After how many events of a partition the source saves its
position (a checkpoint) for that partition. A smaller number reads fewer events
again after a restart, but writes to the blob container more often. When not
set, 500.
+ type: integer
+ example: 100
+ checkpointBatchTimeout:
+ title: Checkpoint Batch Timeout
+ description: How long after the first event since the last checkpoint
of a partition the source saves its position for that partition, when
`checkpointBatchSize` events have not arrived by then, in milliseconds. When
not set, 5000 (5 seconds).
+ type: integer
+ example: 1000
dataTypes:
out:
default: binary
@@ -167,7 +182,7 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&consumerGroupName={{?consumerGroupName}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
+ uri:
"azure-eventhubs://{{?namespaceName}}/{{?eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&consumerGroupName={{?consumerGroupName}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}&connectionString=RAW({{?connectionString}})&checkpointBatchSize={{?checkpointBatchSize}}&checkpointBatchTimeout={{?checkpointBatchTimeout}}"
steps:
- to:
uri: kamelet:sink
diff --git a/kamelets/azure-functions-sink.kamelet.yaml
b/kamelets/azure-functions-sink.kamelet.yaml
index 70b07df6f..d3e96ad0c 100644
--- a/kamelets/azure-functions-sink.kamelet.yaml
+++ b/kamelets/azure-functions-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: when it is used inside a route, the body afterwards is the
response of the function, and the `CamelHttpResponseCode` header its status
code.
- Good to know: a response with a status code outside 200 to 299 fails the
message. The `key` is sent as the `code` query parameter of the request.
+ Good to know: a response with a status code outside 200 to 299 fails the
message. The `key` is sent as the `code` query parameter of the request, or,
when `keyInHeader` is true, in the `x-functions-key` header, which keeps it out
of the address of the request that error messages show.
required:
- url
type: object
@@ -58,11 +58,15 @@ spec:
enum: ["GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS", "TRACE",
"PATCH"]
key:
title: Key
- description: "A function key, or the master key, from the Azure
portal: the function, then Function Keys. Needed when the authorization level
of the function is `function` or `admin`; leave it empty for `anonymous`."
+ description: "A function key, or the master key, from the Azure
portal: the function, then Function Keys. Needed when the authorization level
of the function is `function` or `admin`; leave it empty for `anonymous`. It is
sent as the `code` query parameter, or in the `x-functions-key` header when
`keyInHeader` is true."
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
+ keyInHeader:
+ title: Key in Header
+ description: If true, the `key` is sent in the `x-functions-key` HTTP
header instead of the `code` query parameter. Azure Functions accepts either.
The address of the request, which error messages show, then does not hold the
key, and the header is removed from the message after a successful call.
Requires `key`. When not set, or false, the key is sent as `code`.
+ type: boolean
dependencies:
- "camel:vertx-http"
- "camel:kamelet"
@@ -77,8 +81,28 @@ spec:
expression:
constant:
expression: "{{method}}"
- - to:
- uri: vertx-http
- parameters:
- httpUri: "{{url}}"
- code: "{{?key}}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "{{keyInHeader:false}} == true"
+ steps:
+ - setHeader:
+ name: x-functions-key
+ expression:
+ constant:
+ expression: "{{key}}"
+ - to:
+ uri: vertx-http
+ parameters:
+ httpUri: "{{url}}"
+ - removeHeader:
+ name: x-functions-key
+ otherwise:
+ steps:
+ - to:
+ uri: vertx-http
+ parameters:
+ httpUri: "{{url}}"
+ code: "{{?key}}"
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 513ee0d11..d227ba047 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -334,8 +334,8 @@
"type" : "sink",
"supportLevel" : "Stable",
"title" : "Azure CosmosDB Sink",
- "summary" : "Creates an item (a JSON document) in a container of Azure
Cosmos DB for each message, from the JSON object in the body.",
- "description" : "Creates an item (a JSON document) in a container of Azure
Cosmos DB for each message, from the JSON object in the body.\n\nUse it to
store orders, events or profiles as JSON documents in Cosmos DB. To receive the
changes made to a container, use `azure-cosmosdb-source`.\n\nInput: the body is
one item as JSON text: a JSON object with an `id` field, such as `{\"id\":
\"1001\", \"region\": \"EU\"}`. A Map fails: turn it into JSON text first, for
example with `json-seria [...]
+ "summary" : "Creates or replaces an item (a JSON document) in a container
of Azure Cosmos DB for each message, from the JSON object in the body.",
+ "description" : "Creates or replaces an item (a JSON document) in a
container of Azure Cosmos DB for each message, from the JSON object in the
body.\n\nUse it to store orders, events or profiles as JSON documents in Cosmos
DB. To receive the changes made to a container, use
`azure-cosmosdb-source`.\n\nInput: the body is one item as JSON text: a JSON
object with an `id` field, such as `{\"id\": \"1001\", \"region\": \"EU\"}`. A
Map fails: turn it into JSON text first, for example with [...]
"group" : "Azure CosmosDB",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -345,7 +345,7 @@
"supportLevel" : "Stable",
"title" : "Azure CosmosDB Source",
"summary" : "Receives the items created or changed in a container of Azure
Cosmos DB, through its change feed, as a JSON array per batch.",
- "description" : "Receives the items created or changed in a container of
Azure Cosmos DB, through its change feed, as a JSON array per batch.\n\nUse it
to react to new or updated documents, for example to copy them to a search
index, a cache or Kafka. To create items, use `azure-cosmosdb-sink`.\n\nOutput:
the body is a JSON array of the items of one batch, as bytes, each item with
the fields that Cosmos DB adds, such as `_etag`.\n\nGood to know: it reads only
the changes made after i [...]
+ "description" : "Receives the items created or changed in a container of
Azure Cosmos DB, through its change feed, as a JSON array per batch.\n\nUse it
to react to new or updated documents, for example to copy them to a search
index, a cache or Kafka. To create items, use `azure-cosmosdb-sink`.\n\nOutput:
the body is a JSON array of the items of one batch, as bytes, each item with
the fields that Cosmos DB adds, such as `_etag`.\n\nGood to know: it reads only
the changes made after i [...]
"group" : "Azure CosmosDB",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -355,7 +355,7 @@
"supportLevel" : "Stable",
"title" : "Azure Eventhubs Sink",
"summary" : "Sends each message as an event to an event hub in Azure Event
Hubs, the Azure service for event streams.",
- "description" : "Sends each message as an event to an event hub in Azure
Event Hubs, the Azure service for event streams.\n\nUse it to publish events,
such as clicks, sensor readings or changes, for stream processing or for other
applications to read. To receive them, use `azure-eventhubs-source`.\n\nInput:
the body becomes the body of the event: text, bytes or a stream. A Map fails:
turn it into JSON text first, for example with `json-serialize-action`. A List
is sent as one event p [...]
+ "description" : "Sends each message as an event to an event hub in Azure
Event Hubs, the Azure service for event streams.\n\nUse it to publish events,
such as clicks, sensor readings or changes, for stream processing or for other
applications to read. To receive them, use `azure-eventhubs-source`.\n\nInput:
the body becomes the body of the event: text, bytes or a stream. A Map fails:
turn it into JSON text first, for example with `json-serialize-action`. A List
is sent as one event p [...]
"group" : "Azure Eventhubs",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -365,7 +365,7 @@
"supportLevel" : "Stable",
"title" : "Azure Eventhubs Source",
"summary" : "Receives the events of an event hub in Azure Event Hubs, one
message per event, and saves how far it has read in an Azure Storage blob
container.",
- "description" : "Receives the events of an event hub in Azure Event Hubs,
one message per event, and saves how far it has read in an Azure Storage blob
container.\n\nUse it to process event streams, such as telemetry or changes,
that other applications send to Event Hubs. To send events, use
`azure-eventhubs-sink`.\n\nOutput: the body is the body of the event, as bytes.
The headers give its partition (`CamelAzureEventHubsPartitionId`), offset,
sequence number and the time it was enqu [...]
+ "description" : "Receives the events of an event hub in Azure Event Hubs,
one message per event, and saves how far it has read in an Azure Storage blob
container.\n\nUse it to process event streams, such as telemetry or changes,
that other applications send to Event Hubs. To send events, use
`azure-eventhubs-sink`.\n\nOutput: the body is the body of the event, as bytes.
The headers give its partition (`CamelAzureEventHubsPartitionId`), offset,
sequence number and the time it was enqu [...]
"group" : "Azure Eventhubs",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -375,7 +375,7 @@
"supportLevel" : "Stable",
"title" : "Azure Function Sink",
"summary" : "Calls a function in Azure Functions over HTTP for each
message, with the message as the body of the request.",
- "description" : "Calls a function in Azure Functions over HTTP for each
message, with the message as the body of the request.\n\nUse it to run your own
serverless code on each message, such as an enrichment, a notification or a
calculation. For any other web service, use `http-sink`.\n\nInput: the body is
sent as the request body, as text, bytes or a stream. Turn structured data such
as a Map into JSON text first, for example with `json-serialize-action`. The
`Content-Type` header is [...]
+ "description" : "Calls a function in Azure Functions over HTTP for each
message, with the message as the body of the request.\n\nUse it to run your own
serverless code on each message, such as an enrichment, a notification or a
calculation. For any other web service, use `http-sink`.\n\nInput: the body is
sent as the request body, as text, bytes or a stream. Turn structured data such
as a Map into JSON text first, for example with `json-serialize-action`. The
`Content-Type` header is [...]
"group" : "Azure Functions",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-sink.kamelet.yaml
index feaabf3e8..0aada5704 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-sink.kamelet.yaml
@@ -32,7 +32,7 @@ spec:
definition:
title: "Azure CosmosDB Sink"
description: |-
- Creates an item (a JSON document) in a container of Azure Cosmos DB for
each message, from the JSON object in the body.
+ Creates or replaces an item (a JSON document) in a container of Azure
Cosmos DB for each message, from the JSON object in the body.
Use it to store orders, events or profiles as JSON documents in Cosmos
DB. To receive the changes made to a container, use `azure-cosmosdb-source`.
@@ -40,7 +40,7 @@ spec:
Output: the body afterwards is the item as it was created, as a Map,
with the fields that Cosmos DB adds.
- Good to know: a message without a partition key value fails, and so does
an item whose `id` already exists in the same partition: the sink creates items
and never replaces them.
+ Good to know: a message without a partition key value fails. With
`operation` `createItem`, the default, an item whose `id` already exists in the
same partition fails; with `upsertItem`, it replaces the existing item. The
database and the container must exist, unless `createDatabaseIfNotExists` and
`createContainerIfNotExists` are true; a new container needs
`containerPartitionKeyPath`.
required:
- databaseName
- containerName
@@ -57,6 +57,26 @@ spec:
description: The name of the container in the database to create the
items in.
type: string
example: orders
+ operation:
+ title: Operation
+ description: What to do with each item. `createItem` creates the item,
and fails when an item with the same `id` already exists in the same partition.
`upsertItem` creates the item, or replaces the existing item with the same `id`
in the same partition with it, as a whole.
+ type: string
+ default: createItem
+ enum: ["createItem", "upsertItem"]
+ createDatabaseIfNotExists:
+ title: Autocreate Database
+ description: If true, the database `databaseName` is created when it
does not exist. When not set, or false, a missing database fails each message.
+ type: boolean
+ createContainerIfNotExists:
+ title: Autocreate Container
+ description: If true, the container `containerName` is created when it
does not exist, with `containerPartitionKeyPath` as its partition key; then
`containerPartitionKeyPath` is required. When not set, or false, a missing
container fails each message.
+ type: boolean
+ containerPartitionKeyPath:
+ title: Container Partition Key Path
+ description: The partition key of the container to create when
`createContainerIfNotExists` is true, as the path of a field of the items, such
as `/region`. Not used for a container that exists.
+ type: string
+ pattern: ^/\S+$
+ example: /region
accountKey:
title: Account Key
description: "The primary or secondary key of the Azure Cosmos DB
account, from the Azure portal: the account, then Settings, Keys. Used when
`credentialType` is `SHARED_ACCOUNT_KEY`, the default."
@@ -129,6 +149,30 @@ spec:
expression:
simple:
expression:
"${header[ce-itemPartitionKey]}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${propertiesExist:createDatabaseIfNotExists}"
+ steps:
+ - setHeader:
+ name: CamelAzureCosmosDbCreateDatabaseIfNotExist
+ expression:
+ constant:
+ expression: "{{createDatabaseIfNotExists}}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${propertiesExist:createContainerIfNotExists}"
+ steps:
+ - setHeader:
+ name: CamelAzureCosmosDbCreateContainerIfNotExist
+ expression:
+ constant:
+ expression: "{{createContainerIfNotExists}}"
- unmarshal:
json: {}
- to:
@@ -140,5 +184,8 @@ spec:
credentialType: "{{credentialType}}"
databaseEndpoint: "{{databaseEndpoint}}"
itemPartitionKey: "{{?itemPartitionKey}}"
- operation: createItem
+ operation: "{{operation}}"
+ createDatabaseIfNotExists: "{{?createDatabaseIfNotExists}}"
+ createContainerIfNotExists: "{{?createContainerIfNotExists}}"
+ containerPartitionKeyPath: "{{?containerPartitionKeyPath}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-source.kamelet.yaml
index 7b6a417dc..8bc5cd51d 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-cosmosdb-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is a JSON array of the items of one batch, as bytes,
each item with the fields that Cosmos DB adds, such as `_etag`.
- Good to know: it reads only the changes made after it first starts, and
it does not see deleted items. It keeps its position in a lease container,
`camel-lease` in the database `databaseName` unless set otherwise, which must
exist unless `createLeaseContainerIfNotExists` is true.
+ Good to know: it reads only the changes made after it first starts,
unless `startFromBeginning` is true, and it does not see deleted items. It
keeps its position in a lease container, `camel-lease` in the database
`databaseName` unless set otherwise, which must exist unless
`createLeaseContainerIfNotExists` is true. Two integrations that read the same
container with the same lease container share one position, and each change
goes to only one of them, unless each has its own `lease [...]
required:
- databaseName
- containerName
@@ -93,6 +93,31 @@ spec:
type: string
default: "SHARED_ACCOUNT_KEY"
enum: ["SHARED_ACCOUNT_KEY", "AZURE_IDENTITY"]
+ hostName:
+ title: Host Name
+ description: A name for this running copy of the integration, which
the source writes into the lease container to mark the leases it holds. When it
is not set, the source takes a new random name at each start, so after a crash
it cannot take back the leases of its previous run until they expire, and
receives no changes for up to a minute. With a name that stays the same across
restarts, it continues at once. Give each copy that runs at the same time its
own name.
+ type: string
+ pattern: ^\S+$
+ example: orders-sync-1
+ startFromBeginning:
+ title: Start From Beginning
+ description: If true, the first start reads all the items of the
container, from the oldest change, instead of only the changes made after it
starts. Used only when the lease container holds no position yet for this
source; afterwards the source continues from its saved position.
+ type: boolean
+ maxItemCount:
+ title: Max Items Per Batch
+ description: The largest number of items in one message, the JSON
array of a batch. When not set, 100.
+ type: integer
+ example: 10
+ feedPollDelay:
+ title: Feed Poll Delay
+ description: How long to wait before checking again for new changes,
when there were none, in milliseconds. A smaller value receives changes sooner,
but makes more requests to Cosmos DB, which use request units. When not set,
5000 (5 seconds).
+ type: integer
+ example: 1000
+ leasePrefix:
+ title: Lease Prefix
+ description: A prefix for the names of the leases of this source in
the lease container, such as `search-`. Give each integration that reads the
same container its own prefix, so that they can share a lease container and
each receives every change. Changing the prefix of a source starts it again
without a saved position.
+ type: string
+ example: search-
dataTypes:
out:
default: json
@@ -134,6 +159,14 @@ spec:
- "camel:kamelet"
- "camel:jackson3"
template:
+ beans:
+ - name: changeFeedOptions
+ type: "#class:com.azure.cosmos.models.ChangeFeedProcessorOptions"
+ properties:
+ startFromBeginning: "{{?startFromBeginning}}"
+ maxItemCount: "{{?maxItemCount}}"
+ feedPollDelay: "{{?feedPollDelay}}"
+ leasePrefix: "{{?leasePrefix}}"
from:
uri: azure-cosmosdb
parameters:
@@ -146,6 +179,8 @@ spec:
databaseEndpoint: "{{databaseEndpoint}}"
leaseContainerName: "{{?leaseContainerName}}"
leaseDatabaseName: "{{?leaseDatabaseName}}"
+ hostName: "{{?hostName}}"
+ changeFeedProcessorOptions: "#{{changeFeedOptions}}"
steps:
- marshal:
json: {}
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-sink.kamelet.yaml
index 00d61b302..d35ef2840 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-sink.kamelet.yaml
@@ -36,24 +36,21 @@ spec:
Use it to publish events, such as clicks, sensor readings or changes,
for stream processing or for other applications to read. To receive them, use
`azure-eventhubs-source`.
- Input: the body becomes the body of the event: text, bytes or a stream.
A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. A List is sent as one event per element. The optional
`partition-id` header (or `ce-partition-id`) is the partition to send to, such
as `0`; without it, Event Hubs chooses one.
+ Input: the body becomes the body of the event: text, bytes or a stream.
A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. A List is sent as one event per element. The optional
`partition-id` header (or `ce-partition-id`) is the partition to send to, such
as `0`; without it, and without `partitionKey`, Event Hubs chooses one.
Output: the message is unchanged.
- Good to know: the shared access policy needs the Send right.
- required:
- - namespaceName
- - eventhubName
+ Good to know: log in with `sharedAccessName` and `sharedAccessKey`, with
`connectionString`, or with `credentialType` `AZURE_IDENTITY`. Set
`namespaceName` and `eventhubName` unless `connectionString` is set. The shared
access policy needs the Send right. With `partitionKey` set, a message with a
`partition-id` or `ce-partition-id` header fails.
type: object
properties:
namespaceName:
title: Event Hubs Namespace
- description: The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`.
+ description: "The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`. Required unless
`connectionString` is set: with neither, the integration fails to start with
the error `ConnectionString, ProducerAsyncClient or Namespace and EventHub name
must be set`."
type: string
example: my-namespace
eventhubName:
title: Event Hub Name
- description: The name of the event hub in the namespace to send the
events to.
+ description: The name of the event hub in the namespace to send the
events to. Required unless `connectionString` is set.
type: string
example: orders
sharedAccessName:
@@ -70,12 +67,24 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ connectionString:
+ title: Connection String
+ description: "A connection string of a shared access policy, such as
`Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;EntityPath=orders`,
from the Azure portal: the namespace or the event hub, then Settings, Shared
access policies, the policy, Connection string. Use it instead of
`namespaceName`, `eventhubName`, `sharedAccessName` and `sharedAccessKey`,
which are then not used. It must end with `;EntityPath=` an [...]
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
credentialType:
title: Credential Type
description: How to log in to Event Hubs. `CONNECTION_STRING`, the
default, uses `sharedAccessName` and `sharedAccessKey`. `AZURE_IDENTITY` uses
the Azure identity of the environment, such as the `AZURE_CLIENT_ID`,
`AZURE_TENANT_ID` and `AZURE_CLIENT_SECRET` environment variables, a managed
identity or the Azure CLI login.
type: string
default: "CONNECTION_STRING"
enum: ["CONNECTION_STRING", "AZURE_IDENTITY"]
+ partitionKey:
+ title: Partition Key
+ description: A partition key for all the events, such as `orders`.
Event Hubs sends all the events with the same partition key to the same
partition, which it chooses from the key, so they are read in the order they
were sent. Use it instead of the `partition-id` header, as a message with both
fails. When not set, each event goes to the partition in its `partition-id`
header, or Event Hubs chooses one.
+ type: string
+ example: orders
dataTypes:
in:
headers:
@@ -115,4 +124,4 @@ spec:
simple:
expression: "${header[ce-partition-id]}"
- to:
- uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?credentialType={{credentialType}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
+ uri:
"azure-eventhubs://{{?namespaceName}}/{{?eventhubName}}?credentialType={{credentialType}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}&connectionString=RAW({{?connectionString}})&partitionKey=RAW({{?partitionKey}})"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
index cc3cb9799..5ed66eb62 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
@@ -38,10 +38,8 @@ spec:
Output: the body is the body of the event, as bytes. The headers give
its partition (`CamelAzureEventHubsPartitionId`), offset, sequence number and
the time it was enqueued, and `CamelAzureEventHubsMetadata` holds the custom
properties of the event.
- Good to know: when no position has been saved yet, it starts with the
events that arrive after it starts, not the older ones. It reads as the
`$Default` consumer group, or as the one in `consumerGroupName`, and each
consumer group has its own saved position. It saves its position (a checkpoint)
in `blobContainerName` every 500 events or every 5 seconds, so after a restart
some events can be read again.
+ Good to know: log in with `sharedAccessName` and `sharedAccessKey`, with
`connectionString`, or with `credentialType` `AZURE_IDENTITY`. Set
`namespaceName` and `eventhubName` unless `connectionString` is set. When no
position has been saved yet, it starts with the events that arrive after it
starts, not the older ones. It reads as the `$Default` consumer group, or as
the one in `consumerGroupName`, and each consumer group has its own saved
position. For each partition, it saves its [...]
required:
- - namespaceName
- - eventhubName
- blobAccountName
- blobAccessKey
- blobContainerName
@@ -49,12 +47,12 @@ spec:
properties:
namespaceName:
title: Event Hubs Namespace
- description: The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`.
+ description: "The name of the Event Hubs namespace, the first part of
its address `<namespaceName>.servicebus.windows.net`. Required unless
`connectionString` is set: with neither, the integration fails to start with
the error `ConnectionString, ProducerAsyncClient or Namespace and EventHub name
must be set`."
type: string
example: my-namespace
eventhubName:
title: Event Hub Name
- description: The name of the event hub in the namespace to receive the
events from.
+ description: The name of the event hub in the namespace to receive the
events from. Required unless `connectionString` is set.
type: string
example: orders
consumerGroupName:
@@ -76,6 +74,13 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ connectionString:
+ title: Connection String
+ description: "A connection string of a shared access policy, such as
`Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;EntityPath=orders`,
from the Azure portal: the namespace or the event hub, then Settings, Shared
access policies, the policy, Connection string. Use it instead of
`namespaceName`, `eventhubName`, `sharedAccessName` and `sharedAccessKey`,
which are then not used. It must end with `;EntityPath=` an [...]
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
blobAccountName:
title: Checkpoint Storage Account
description: The name of the storage account where the source saves
how far it has read each partition (its checkpoints).
@@ -99,6 +104,16 @@ spec:
type: string
default: "CONNECTION_STRING"
enum: ["CONNECTION_STRING", "AZURE_IDENTITY"]
+ checkpointBatchSize:
+ title: Checkpoint Batch Size
+ description: After how many events of a partition the source saves its
position (a checkpoint) for that partition. A smaller number reads fewer events
again after a restart, but writes to the blob container more often. When not
set, 500.
+ type: integer
+ example: 100
+ checkpointBatchTimeout:
+ title: Checkpoint Batch Timeout
+ description: How long after the first event since the last checkpoint
of a partition the source saves its position for that partition, when
`checkpointBatchSize` events have not arrived by then, in milliseconds. When
not set, 5000 (5 seconds).
+ type: integer
+ example: 1000
dataTypes:
out:
default: binary
@@ -167,7 +182,7 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&consumerGroupName={{?consumerGroupName}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
+ uri:
"azure-eventhubs://{{?namespaceName}}/{{?eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&consumerGroupName={{?consumerGroupName}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}&connectionString=RAW({{?connectionString}})&checkpointBatchSize={{?checkpointBatchSize}}&checkpointBatchTimeout={{?checkpointBatchTimeout}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-functions-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-functions-sink.kamelet.yaml
index 70b07df6f..d3e96ad0c 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-functions-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-functions-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: when it is used inside a route, the body afterwards is the
response of the function, and the `CamelHttpResponseCode` header its status
code.
- Good to know: a response with a status code outside 200 to 299 fails the
message. The `key` is sent as the `code` query parameter of the request.
+ Good to know: a response with a status code outside 200 to 299 fails the
message. The `key` is sent as the `code` query parameter of the request, or,
when `keyInHeader` is true, in the `x-functions-key` header, which keeps it out
of the address of the request that error messages show.
required:
- url
type: object
@@ -58,11 +58,15 @@ spec:
enum: ["GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS", "TRACE",
"PATCH"]
key:
title: Key
- description: "A function key, or the master key, from the Azure
portal: the function, then Function Keys. Needed when the authorization level
of the function is `function` or `admin`; leave it empty for `anonymous`."
+ description: "A function key, or the master key, from the Azure
portal: the function, then Function Keys. Needed when the authorization level
of the function is `function` or `admin`; leave it empty for `anonymous`. It is
sent as the `code` query parameter, or in the `x-functions-key` header when
`keyInHeader` is true."
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
+ keyInHeader:
+ title: Key in Header
+ description: If true, the `key` is sent in the `x-functions-key` HTTP
header instead of the `code` query parameter. Azure Functions accepts either.
The address of the request, which error messages show, then does not hold the
key, and the header is removed from the message after a successful call.
Requires `key`. When not set, or false, the key is sent as `code`.
+ type: boolean
dependencies:
- "camel:vertx-http"
- "camel:kamelet"
@@ -77,8 +81,28 @@ spec:
expression:
constant:
expression: "{{method}}"
- - to:
- uri: vertx-http
- parameters:
- httpUri: "{{url}}"
- code: "{{?key}}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "{{keyInHeader:false}} == true"
+ steps:
+ - setHeader:
+ name: x-functions-key
+ expression:
+ constant:
+ expression: "{{key}}"
+ - to:
+ uri: vertx-http
+ parameters:
+ httpUri: "{{url}}"
+ - removeHeader:
+ name: x-functions-key
+ otherwise:
+ steps:
+ - to:
+ uri: vertx-http
+ parameters:
+ httpUri: "{{url}}"
+ code: "{{?key}}"