This is an automated email from the ASF dual-hosted git repository.
oscerd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new f3283d95b chore: azure-eventhubs-source can read with its own consumer
group (#3170) (#3193)
f3283d95b is described below
commit f3283d95b39c3be4b92ba8df92ac60abd819a533
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 15:10:54 2026 +0100
chore: azure-eventhubs-source can read with its own consumer group (#3170)
(#3193)
The source always read as the $Default consumer group, so two
integrations could not each receive every event of the same event hub.
Add the optional consumerGroupName option, mapped with an optional
placeholder (consumerGroupName={{?consumerGroupName}}) and no default, so
when it is not set the endpoint URI is the same as before and the
component reads as $Default.
eventPosition is not offered: the component option is a Map of
partition id to EventPosition, which cannot be set from a URI string
("No type converter available to convert from type: java.lang.String to
the required type: java.util.Map"), and there is no simpler option for
the earliest or latest position.
Verified with Camel JBang against the Azure Event Hubs emulator and
Azurite: two consumer groups both receive every event and keep their own
position, the option unset reads as $Default like the Kamelet from main,
and a consumer group that does not exist starts but receives nothing and
logs that the messaging entity could not be found.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../azure-eventhubs-source-description.adoc | 25 ++++++++++++++++++++++
kamelets/azure-eventhubs-source.kamelet.yaml | 9 ++++++--
.../main/resources/META-INF/kamelets-index.json | 2 +-
.../kamelets/azure-eventhubs-source.kamelet.yaml | 9 ++++++--
4 files changed, 40 insertions(+), 5 deletions(-)
diff --git a/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
b/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
index 9ec68e7c8..992b5468f 100644
--- a/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
+++ b/docs/modules/ROOT/partials/azure-eventhubs-source-description.adoc
@@ -18,3 +18,28 @@ The order of evaluation for `AZURE_IDENTITY` is the
following:
- Azure Powershell
For more information, see the
https://learn.microsoft.com/en-us/java/api/overview/azure/identity-readme[Azure
Identity documentation]
+
+=== Reading with your own consumer group
+
+By default the Kamelet reads as the `$Default` consumer group. To read the
same event hub from several integrations, each receiving every event, create a
consumer group for each of them in the Azure portal (the event hub, then
Entities, Consumer groups) and set it in `consumerGroupName`:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:azure-eventhubs-source"
+ parameters:
+ namespaceName: "my-namespace"
+ eventhubName: "orders"
+ sharedAccessName: "RootManageSharedAccessKey"
+ sharedAccessKey: "{{eventhubs.key}}"
+ blobAccountName: "mystorageaccount"
+ blobAccessKey: "{{storage.key}}"
+ blobContainerName: "eventhub-checkpoints"
+ consumerGroupName: "billing"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each consumer group has its own saved position in `blobContainerName`, so the
integrations can share the container. A consumer group without a saved position
starts with the events that arrive after the integration starts. With a
consumer group that does not exist, the integration starts but receives
nothing, and logs a warning that the messaging entity could not be found.
diff --git a/kamelets/azure-eventhubs-source.kamelet.yaml
b/kamelets/azure-eventhubs-source.kamelet.yaml
index 714c694ed..cc3cb9799 100644
--- a/kamelets/azure-eventhubs-source.kamelet.yaml
+++ b/kamelets/azure-eventhubs-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the body of the event, as bytes. The headers give
its partition (`CamelAzureEventHubsPartitionId`), offset, sequence number and
the time it was enqueued, and `CamelAzureEventHubsMetadata` holds the custom
properties of the event.
- Good to know: when no position has been saved yet, it starts with the
events that arrive after it starts, not the older ones. It reads as the
`$Default` consumer group. It saves its position (a checkpoint) in
`blobContainerName` every 500 events or every 5 seconds, so after a restart
some events can be read again.
+ Good to know: when no position has been saved yet, it starts with the
events that arrive after it starts, not the older ones. It reads as the
`$Default` consumer group, or as the one in `consumerGroupName`, and each
consumer group has its own saved position. It saves its position (a checkpoint)
in `blobContainerName` every 500 events or every 5 seconds, so after a restart
some events can be read again.
required:
- namespaceName
- eventhubName
@@ -57,6 +57,11 @@ spec:
description: The name of the event hub in the namespace to receive the
events from.
type: string
example: orders
+ consumerGroupName:
+ title: Consumer Group
+ description: "The consumer group to read as. Each consumer group
receives every event of the event hub and has its own saved position, so two
integrations that read the same event hub with different consumer groups both
receive every event. The consumer group must exist: create it in the Azure
portal, under the event hub, then Entities, Consumer groups. With a consumer
group that does not exist, the integration starts but receives nothing, and
logs a warning that the messaging en [...]
+ type: string
+ example: billing
sharedAccessName:
title: Shared Access Name
description: "The name of a shared access policy of the namespace or
of the event hub, such as `RootManageSharedAccessKey`, from the Azure portal:
the namespace or the event hub, then Settings, Shared access policies. Used
when `credentialType` is `CONNECTION_STRING`, the default."
@@ -162,7 +167,7 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
+ uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&consumerGroupName={{?consumerGroupName}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 274acd112..b67c6a43a 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -365,7 +365,7 @@
"supportLevel" : "Stable",
"title" : "Azure Eventhubs Source",
"summary" : "Receives the events of an event hub in Azure Event Hubs, one
message per event, and saves how far it has read in an Azure Storage blob
container.",
- "description" : "Receives the events of an event hub in Azure Event Hubs,
one message per event, and saves how far it has read in an Azure Storage blob
container.\n\nUse it to process event streams, such as telemetry or changes,
that other applications send to Event Hubs. To send events, use
`azure-eventhubs-sink`.\n\nOutput: the body is the body of the event, as bytes.
The headers give its partition (`CamelAzureEventHubsPartitionId`), offset,
sequence number and the time it was enqu [...]
+ "description" : "Receives the events of an event hub in Azure Event Hubs,
one message per event, and saves how far it has read in an Azure Storage blob
container.\n\nUse it to process event streams, such as telemetry or changes,
that other applications send to Event Hubs. To send events, use
`azure-eventhubs-sink`.\n\nOutput: the body is the body of the event, as bytes.
The headers give its partition (`CamelAzureEventHubsPartitionId`), offset,
sequence number and the time it was enqu [...]
"group" : "Azure Eventhubs",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
index 714c694ed..cc3cb9799 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-eventhubs-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the body of the event, as bytes. The headers give
its partition (`CamelAzureEventHubsPartitionId`), offset, sequence number and
the time it was enqueued, and `CamelAzureEventHubsMetadata` holds the custom
properties of the event.
- Good to know: when no position has been saved yet, it starts with the
events that arrive after it starts, not the older ones. It reads as the
`$Default` consumer group. It saves its position (a checkpoint) in
`blobContainerName` every 500 events or every 5 seconds, so after a restart
some events can be read again.
+ Good to know: when no position has been saved yet, it starts with the
events that arrive after it starts, not the older ones. It reads as the
`$Default` consumer group, or as the one in `consumerGroupName`, and each
consumer group has its own saved position. It saves its position (a checkpoint)
in `blobContainerName` every 500 events or every 5 seconds, so after a restart
some events can be read again.
required:
- namespaceName
- eventhubName
@@ -57,6 +57,11 @@ spec:
description: The name of the event hub in the namespace to receive the
events from.
type: string
example: orders
+ consumerGroupName:
+ title: Consumer Group
+ description: "The consumer group to read as. Each consumer group
receives every event of the event hub and has its own saved position, so two
integrations that read the same event hub with different consumer groups both
receive every event. The consumer group must exist: create it in the Azure
portal, under the event hub, then Entities, Consumer groups. With a consumer
group that does not exist, the integration starts but receives nothing, and
logs a warning that the messaging en [...]
+ type: string
+ example: billing
sharedAccessName:
title: Shared Access Name
description: "The name of a shared access policy of the namespace or
of the event hub, such as `RootManageSharedAccessKey`, from the Azure portal:
the namespace or the event hub, then Settings, Shared access policies. Used
when `credentialType` is `CONNECTION_STRING`, the default."
@@ -162,7 +167,7 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
+ uri:
"azure-eventhubs://{{namespaceName}}/{{eventhubName}}?blobAccessKey=RAW({{blobAccessKey}})&blobAccountName={{blobAccountName}}&blobContainerName={{blobContainerName}}&credentialType={{credentialType}}&consumerGroupName={{?consumerGroupName}}&sharedAccessKey=RAW({{?sharedAccessKey}})&sharedAccessName={{?sharedAccessName}}"
steps:
- to:
uri: kamelet:sink