This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 0f142d540 chore: kafka-source and kafka-sink can use a schema registry
(#3183) (#3203)
0f142d540 is described below
commit 0f142d5407d6f174096287b1229bfa74ec6af55d
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 21:25:44 2026 +0100
chore: kafka-source and kafka-sink can use a schema registry (#3183) (#3203)
* chore: kafka-source and kafka-sink can use a schema registry (#3183)
New schemaRegistryVendor (confluent, apicurio), schemaRegistryUrl and
schemaFormat (avro, json) options pass through to the camel-kafka options of
CAMEL-27130, which set the serde classes and registry properties of the
vendor.
kafka-source turns the value read with its schema into JSON text (an Avro
record by its JSON form, a JSON Schema value with the json data format), so
the
next step gets text as without a registry. kafka-sink with json turns JSON
text
into an object, so the record is written with an object schema rather than a
string. Without schemaRegistryVendor nothing changes.
A Citrus test writes and reads a JSON Schema record through the schema
registry of Redpanda, with the Confluent serdes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* chore: kafka-source and kafka-sink support the aws-glue and azure schema
registries (#3183)
schemaRegistryVendor also takes aws-glue and azure. New options:
schemaRegistryRegion and schemaRegistryName (AWS Glue, passed as the region
and registry.name serde properties) and, on kafka-sink, schemaGroup (Azure,
schema.group). Unset, they are not passed. schemaRegistryUrl also takes the
Event Hubs namespace without a scheme, as Azure expects.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../camel-kamelets-catalog-sbom.json | 4 +-
.../ROOT/partials/kafka-sink-description.adoc | 44 ++++++++++++++
.../ROOT/partials/kafka-source-description.adoc | 39 ++++++++++++
kamelets/kafka-sink.kamelet.yaml | 50 ++++++++++++++-
kamelets/kafka-source.kamelet.yaml | 55 ++++++++++++++++-
.../main/resources/META-INF/kamelets-index.json | 4 +-
.../resources/kamelets/kafka-sink.kamelet.yaml | 50 ++++++++++++++-
.../resources/kamelets/kafka-source.kamelet.yaml | 55 ++++++++++++++++-
.../kafka-schema-registry-route.citrus.it.yaml | 71 ++++++++++++++++++++++
.../kafka/kafka-schema-registry-route.yaml | 48 +++++++++++++++
10 files changed, 404 insertions(+), 16 deletions(-)
diff --git a/camel-kamelets-sbom/camel-kamelets-catalog-sbom.json
b/camel-kamelets-sbom/camel-kamelets-catalog-sbom.json
index 3592d2b9d..c4c16c2d8 100644
--- a/camel-kamelets-sbom/camel-kamelets-catalog-sbom.json
+++ b/camel-kamelets-sbom/camel-kamelets-catalog-sbom.json
@@ -2793,10 +2793,10 @@
"dependsOn" : [
"pkg:maven/io.quarkus/[email protected]",
"pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]" ]
}, {
"ref" : "kamelet:kafka-sink",
- "dependsOn" : [ "pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]" ]
+ "dependsOn" : [
"pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]" ]
}, {
"ref" : "kamelet:kafka-source",
- "dependsOn" : [ "pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]" ]
+ "dependsOn" : [
"pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]" ]
}, {
"ref" : "kamelet:langchain4j-ingest-file-source",
"dependsOn" : [ "pkg:maven/org.apache.camel/[email protected]",
"pkg:maven/org.apache.camel/[email protected]" ]
diff --git a/docs/modules/ROOT/partials/kafka-sink-description.adoc
b/docs/modules/ROOT/partials/kafka-sink-description.adoc
index 75a69807d..984b918d6 100644
--- a/docs/modules/ROOT/partials/kafka-sink-description.adoc
+++ b/docs/modules/ROOT/partials/kafka-sink-description.adoc
@@ -16,6 +16,50 @@ leave the others empty.
| `KERBEROS` | None in the Kamelet: Kerberos is configured in the environment
the integration runs in.
|===
+=== Schema registry
+
+To write the records with a schema kept in a schema registry, set
`schemaRegistryVendor`, `schemaRegistryUrl` and
+`schemaFormat` (the default is `avro`). With `json`, the body must be JSON
text (or a Map); the Kamelet turns it into
+an object, and the record is written with a JSON Schema made from it, which
accepts any JSON object. With `avro`,
+`confluent` and `apicurio` write a text body as an Avro string, while
`aws-glue` needs an Avro record, which a step must
+create in code. With `azure`, also set `schemaGroup`. The sink registers the
+schema it makes, and does not check the message against a schema that is
already registered for the topic.
+
+[width="100%",cols="1,1,3",options="header"]
+|===
+| `schemaRegistryVendor` | `schemaFormat` | Registry
+| `confluent` | `avro`, `json` | Confluent Schema Registry, and the registries
with the same API, such as the one of Redpanda and Karapace.
`schemaRegistryUrl` is the URL of the registry, such as `http://localhost:8081`.
+| `apicurio` | `avro` | Apicurio Registry 3.x. `schemaRegistryUrl` is the URL
of the registry API, such as `http://localhost:8080/apis/registry/v3`.
+| `aws-glue` | `avro`, `json` | AWS Glue Schema Registry. Set
`schemaRegistryRegion`, unless the environment has the region (such as
`AWS_REGION`), and `schemaRegistryName`, unless the registry is
`default-registry`. `schemaRegistryUrl` is not needed. The AWS credentials are
taken from the AWS default credentials chain.
+| `azure` | `avro` | Azure Schema Registry of Event Hubs. `schemaRegistryUrl`
is the fully qualified name of the namespace, such as
`my-namespace.servicebus.windows.net`. The sink needs `schemaGroup`. The
credentials are taken by the Azure `DefaultAzureCredential`, such as from the
`AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and `AZURE_CLIENT_SECRET` variables.
+|===
+
+The serde jars of the vendor are not part of the Kamelet. Camel JBang
downloads them by itself; the Confluent jars are
+in the Confluent Maven repository, not in Maven Central, so run with
`--repos=confluent`. In a Maven project, add the
+jar of the vendor and format, such as `io.confluent:kafka-avro-serializer`,
and for Confluent the repository
+`https://packages.confluent.io/maven/`. A `confluent` or `apicurio` registry
that requires a login is not supported yet.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ period: 10000
+ message: '{"id": 42, "item": "book"}'
+ contentType: application/json
+ steps:
+ - to:
+ uri: "kamelet:kafka-sink"
+ parameters:
+ bootstrapServers: "localhost:9092"
+ topic: orders
+ saslAuthType: NONE
+ schemaRegistryVendor: confluent
+ schemaRegistryUrl: "http://localhost:8081"
+ schemaFormat: json
+----
+
=== Usage example
The route below sends a JSON order every ten seconds to the `orders` topic,
with the order number as the record key.
diff --git a/docs/modules/ROOT/partials/kafka-source-description.adoc
b/docs/modules/ROOT/partials/kafka-source-description.adoc
index d5ed65c0f..39c8729ce 100644
--- a/docs/modules/ROOT/partials/kafka-source-description.adoc
+++ b/docs/modules/ROOT/partials/kafka-source-description.adoc
@@ -43,6 +43,8 @@ deployment allows it.
`true`
- **topicIsPattern**: Treat `topic` as a regular expression rather than a
literal list,
default `false`
+- **schemaRegistryVendor** / **schemaRegistryUrl** / **schemaFormat** /
**schemaRegistryRegion** /
+ **schemaRegistryName**: Read the records with their schema from a schema
registry, see below
=== Output Format
@@ -62,6 +64,43 @@ The `CamelKafka*` headers are left on the exchange as well,
so consumers already
keep working. The Kafka record headers are passed through separately,
controlled by the
`deserializeHeaders` property.
+=== Schema registry
+
+When the records are written with a schema kept in a schema registry, set
`schemaRegistryVendor`, `schemaRegistryUrl`
+and, for JSON Schema, `schemaFormat: json` (the default is `avro`). The
Kamelet reads each value with its schema and
+turns it into JSON text, so the next step gets text as without a registry: an
Avro record becomes, for example,
+`{"id": 7, "item": "camel"}`.
+
+[width="100%",cols="1,1,3",options="header"]
+|===
+| `schemaRegistryVendor` | `schemaFormat` | Registry
+| `confluent` | `avro`, `json` | Confluent Schema Registry, and the registries
with the same API, such as the one of Redpanda and Karapace.
`schemaRegistryUrl` is the URL of the registry, such as `http://localhost:8081`.
+| `apicurio` | `avro` | Apicurio Registry 3.x. `schemaRegistryUrl` is the URL
of the registry API, such as `http://localhost:8080/apis/registry/v3`.
+| `aws-glue` | `avro`, `json` | AWS Glue Schema Registry. Set
`schemaRegistryRegion`, unless the environment has the region (such as
`AWS_REGION`), and `schemaRegistryName`, unless the registry is
`default-registry`. `schemaRegistryUrl` is not needed. The AWS credentials are
taken from the AWS default credentials chain.
+| `azure` | `avro` | Azure Schema Registry of Event Hubs. `schemaRegistryUrl`
is the fully qualified name of the namespace, such as
`my-namespace.servicebus.windows.net`. The sink needs `schemaGroup`. The
credentials are taken by the Azure `DefaultAzureCredential`, such as from the
`AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and `AZURE_CLIENT_SECRET` variables.
+|===
+
+The serde jars of the vendor are not part of the Kamelet. Camel JBang
downloads them by itself; the Confluent jars are
+in the Confluent Maven repository, not in Maven Central, so run with
`--repos=confluent`. In a Maven project, add the
+jar of the vendor and format, such as `io.confluent:kafka-avro-serializer`,
and for Confluent the repository
+`https://packages.confluent.io/maven/`. A `confluent` or `apicurio` registry
that requires a login is not supported yet.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: orders
+ bootstrapServers: "localhost:9092"
+ saslAuthType: NONE
+ schemaRegistryVendor: confluent
+ schemaRegistryUrl: "http://localhost:8081"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
=== Usage Example
[source,yaml,subs='+attributes,macros']
diff --git a/kamelets/kafka-sink.kamelet.yaml b/kamelets/kafka-sink.kamelet.yaml
index 201ceedec..90b6d240a 100644
--- a/kamelets/kafka-sink.kamelet.yaml
+++ b/kamelets/kafka-sink.kamelet.yaml
@@ -33,13 +33,13 @@ spec:
description: |-
Sends each message as a record to a topic in Apache Kafka, or in a
Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
- Use it to publish events or data for other applications to read, such as
orders, sensor readings or changes from a database. When the records must
follow a schema kept in a schema registry, use one of the `kafka-*-registry-*`
sinks instead.
+ Use it to publish events or data for other applications to read, such as
orders, sensor readings or changes from a database. To write the records with a
schema kept in a schema registry, such as Confluent Schema Registry, Redpanda
or Apicurio Registry, set `schemaRegistryVendor`.
- Input: the body becomes the value of the record, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional `key` header (or `kafka-key`, `ce-key`,
`ce-kafkakey`) becomes the key of the record, and the optional `partition-key`
header (or `ce-partitionkey`) is the number of the partition to write to. The
other headers of the message, except the Camel ones, are sent as record headers.
+ Input: the body becomes the value of the record, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. With `schemaRegistryVendor` and `schemaFormat` `json`,
the body must be JSON text (or a Map), and the record is written with a JSON
Schema; with `avro`, `confluent` and `apicurio` write a text body as an Avro
string, while `aws-glue` needs an Avro record. With `azure`, set `schemaGroup`.
The optional `key` header (or `kafka-key [...]
Output: the message is unchanged.
- Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS, as a local Kafka expects. A managed Kafka
service needs one of the other types.
+ Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS, as a local Kafka expects. A managed Kafka
service needs one of the other types. With `schemaRegistryVendor`, the
integration needs the serde jars of the vendor, which Camel JBang downloads by
itself; for `confluent` add the Confluent Maven repository, as in `camel run
--repos=confluent`. With `aws-glue` and `azure`, the serde logs in with the
credentials of the environment the int [...]
required:
- topic
- bootstrapServers
@@ -122,7 +122,36 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ schemaRegistryVendor:
+ title: Schema Registry Vendor
+ description: "The schema registry that holds the schemas of the record
values: `confluent`, also for registries with the Confluent API such as
Redpanda and Karapace; `apicurio` (Apicurio Registry 3.x); `aws-glue` (AWS Glue
Schema Registry); or `azure` (Azure Schema Registry of Event Hubs). It sets the
value serializer and the registry properties, and its jars must be added to the
integration (see below). Leave it unset when the records do not use a schema
registry."
+ type: string
+ enum: ["confluent", "apicurio", "aws-glue", "azure"]
+ schemaRegistryUrl:
+ title: Schema Registry URL
+ description: "The URL of the schema registry: for `confluent` such as
`http://localhost:8081`; for `apicurio` the URL of the registry API, such as
`http://localhost:8080/apis/registry/v3`; for `azure` the fully qualified name
of the Event Hubs namespace, such as `my-namespace.servicebus.windows.net`.
Required when `schemaRegistryVendor` is set, except for `aws-glue`, where it
can override the endpoint of the Glue service."
+ type: string
+ pattern: "^(https?://)?[^\\s]+$"
+ schemaRegistryRegion:
+ title: Schema Registry Region
+ description: "The AWS region of the AWS Glue Schema Registry, such as
`eu-west-1`. When not set, the region is taken from the environment, such as
the `AWS_REGION` variable. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaRegistryName:
+ title: Schema Registry Name
+ description: "The name of the registry in AWS Glue. When not set, it
is `default-registry`. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaGroup:
+ title: Schema Group
+ description: "The schema group in the Azure Schema Registry that the
schemas of the records are registered in. Required when `schemaRegistryVendor`
is `azure`."
+ type: string
+ schemaFormat:
+ title: Schema Format
+ description: "The format of the record values in the schema registry:
`avro` or `json` (JSON Schema). With `apicurio` and `azure`, use `avro`. Only
used when `schemaRegistryVendor` is set."
+ type: string
+ enum: ["avro", "json"]
+ default: avro
dependencies:
+ - "camel:jackson3"
- "camel:kafka"
- "camel:kamelet"
template:
@@ -187,10 +216,22 @@ spec:
expression:
simple:
expression: "${header[ce-partitionkey]}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{?schemaRegistryVendor}}' != '' &&
'{{schemaFormat}}' == 'json' && ${body} != null && ${body} !is 'java.util.Map'
&& ${body} !is 'java.util.List'"
+ steps:
+ - unmarshal:
+ json:
+ library: Jackson
- to:
uri: kafka
parameters:
topic: "{{topic}}"
+ additionalProperties.region: "{{?schemaRegistryRegion}}"
+ additionalProperties.registry.name: "{{?schemaRegistryName}}"
+ additionalProperties.schema.group: "{{?schemaGroup}}"
brokers: "{{bootstrapServers}}"
oauthClientId: "{{?oauthClientId}}"
oauthClientSecret: "RAW({{?oauthClientSecret}})"
@@ -199,6 +240,9 @@ spec:
saslAuthType: "{{saslAuthType}}"
saslPassword: "RAW({{?saslPassword}})"
saslUsername: "{{?saslUsername}}"
+ schemaFormat: "{{schemaFormat}}"
+ schemaRegistryURL: "{{?schemaRegistryUrl}}"
+ schemaRegistryVendor: "{{?schemaRegistryVendor}}"
sslKeyPassword: "RAW({{?sslKeyPassword}})"
sslKeystoreLocation: "{{?sslKeystoreLocation}}"
sslKeystorePassword: "RAW({{?sslKeystorePassword}})"
diff --git a/kamelets/kafka-source.kamelet.yaml
b/kamelets/kafka-source.kamelet.yaml
index 653b1980f..bec2b948c 100644
--- a/kamelets/kafka-source.kamelet.yaml
+++ b/kamelets/kafka-source.kamelet.yaml
@@ -33,11 +33,11 @@ spec:
description: |-
Receives each record from one or more topics in Apache Kafka, or in a
Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
- Use it to react to events or data that other applications publish, such
as orders, sensor readings or changes from a database. To get many records in
one message use `kafka-batch-source`. When the records are written with a
schema kept in a schema registry, use one of the `kafka-*-registry-*` sources
instead.
+ Use it to react to events or data that other applications publish, such
as orders, sensor readings or changes from a database. To get many records in
one message use `kafka-batch-source`. When the records are written with a
schema kept in a schema registry, such as Confluent Schema Registry, Redpanda
or Apicurio Registry, set `schemaRegistryVendor`.
- Output: the body is the value of the record, as text. A record without a
value (a tombstone) gives a message with no body. The `kafka-topic`,
`kafka-key`, `kafka-partition`, `kafka-offset` and `kafka-timestamp` (in
milliseconds since 1970) headers say where the record came from. They are also
set as `ce-kafkatopic`, `ce-kafkakey` and so on, and as the `CamelKafka*`
headers that the Kafka actions read. The record headers become message headers,
as text unless `deserializeHeaders` is false.
+ Output: the body is the value of the record, as text. With
`schemaRegistryVendor`, the value is read with its schema from the registry and
the body is JSON text, such as `{"id": 7, "item": "camel"}` for an Avro record.
A record without a value (a tombstone) gives a message with no body. The
`kafka-topic`, `kafka-key`, `kafka-partition`, `kafka-offset` and
`kafka-timestamp` (in milliseconds since 1970) headers say where the record
came from. They are also set as `ce-kafkatopic`, `ce [...]
- Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS. A new consumer group starts at the newest
records, unless `autoOffsetReset` is `earliest`. Without `consumerGroup`, every
start uses a new group, so the records sent while the integration was stopped
are skipped.
+ Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS. A new consumer group starts at the newest
records, unless `autoOffsetReset` is `earliest`. Without `consumerGroup`, every
start uses a new group, so the records sent while the integration was stopped
are skipped. With `schemaRegistryVendor`, the integration needs the serde jars
of the vendor, which Camel JBang downloads by itself; for `confluent` add the
Confluent Maven repository [...]
required:
- topic
- bootstrapServers
@@ -161,6 +161,30 @@ spec:
description: "If true, `topic` is a regular expression, such as
`orders-.*`, and the Kamelet reads from every topic whose name matches it."
type: boolean
default: false
+ schemaRegistryVendor:
+ title: Schema Registry Vendor
+ description: "The schema registry that holds the schemas of the record
values: `confluent`, also for registries with the Confluent API such as
Redpanda and Karapace; `apicurio` (Apicurio Registry 3.x); `aws-glue` (AWS Glue
Schema Registry); or `azure` (Azure Schema Registry of Event Hubs). It sets the
value deserializer and the registry properties, and its jars must be added to
the integration (see below). Leave it unset when the records do not use a
schema registry."
+ type: string
+ enum: ["confluent", "apicurio", "aws-glue", "azure"]
+ schemaRegistryUrl:
+ title: Schema Registry URL
+ description: "The URL of the schema registry: for `confluent` such as
`http://localhost:8081`; for `apicurio` the URL of the registry API, such as
`http://localhost:8080/apis/registry/v3`; for `azure` the fully qualified name
of the Event Hubs namespace, such as `my-namespace.servicebus.windows.net`.
Required when `schemaRegistryVendor` is set, except for `aws-glue`, where it
can override the endpoint of the Glue service."
+ type: string
+ pattern: "^(https?://)?[^\\s]+$"
+ schemaRegistryRegion:
+ title: Schema Registry Region
+ description: "The AWS region of the AWS Glue Schema Registry, such as
`eu-west-1`. When not set, the region is taken from the environment, such as
the `AWS_REGION` variable. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaRegistryName:
+ title: Schema Registry Name
+ description: "The name of the registry in AWS Glue. When not set, it
is `default-registry`. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaFormat:
+ title: Schema Format
+ description: "The format of the record values in the schema registry:
`avro` or `json` (JSON Schema). With `apicurio` and `azure`, use `avro`. Only
used when `schemaRegistryVendor` is set."
+ type: string
+ enum: ["avro", "json"]
+ default: avro
dataTypes:
out:
headers:
@@ -205,6 +229,7 @@ spec:
description: "The same as `kafka-timestamp`."
type: integer
dependencies:
+ - "camel:jackson3"
- "camel:kafka"
- "camel:kamelet"
template:
@@ -217,6 +242,8 @@ spec:
uri: kafka
parameters:
topic: "{{topic}}"
+ additionalProperties.region: "{{?schemaRegistryRegion}}"
+ additionalProperties.registry.name: "{{?schemaRegistryName}}"
allowManualCommit: "{{allowManualCommit}}"
autoCommitEnable: "{{autoCommitEnable}}"
autoOffsetReset: "{{autoOffsetReset}}"
@@ -230,6 +257,9 @@ spec:
saslAuthType: "{{saslAuthType}}"
saslPassword: "RAW({{?saslPassword}})"
saslUsername: "{{?saslUsername}}"
+ schemaFormat: "{{schemaFormat}}"
+ schemaRegistryURL: "{{?schemaRegistryUrl}}"
+ schemaRegistryVendor: "{{?schemaRegistryVendor}}"
sslKeyPassword: "RAW({{?sslKeyPassword}})"
sslKeystoreLocation: "{{?sslKeystoreLocation}}"
sslKeystorePassword: "RAW({{?sslKeystorePassword}})"
@@ -294,5 +324,24 @@ spec:
expression:
simple:
expression: "${header[CamelKafkaTimestamp]}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{?schemaRegistryVendor}}' != '' &&
'{{schemaFormat}}' == 'json' && ${body} != null && ${body} !is
'java.lang.String'"
+ steps:
+ - marshal:
+ json:
+ library: Jackson
+ - convertBodyTo:
+ type: java.lang.String
+ - expression:
+ simple:
+ expression: "'{{?schemaRegistryVendor}}' != '' && ${body}
!= null && ${body} !is 'java.lang.String'"
+ steps:
+ - setBody:
+ expression:
+ simple:
+ expression: "${body.toString()}"
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index af28c08ed..2624dc5ce 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1405,7 +1405,7 @@
"supportLevel" : "Stable",
"title" : "Kafka Sink",
"summary" : "Sends each message as a record to a topic in Apache Kafka, or
in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.",
- "description" : "Sends each message as a record to a topic in Apache
Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or
Redpanda.\n\nUse it to publish events or data for other applications to read,
such as orders, sensor readings or changes from a database. When the records
must follow a schema kept in a schema registry, use one of the
`kafka-*-registry-*` sinks instead.\n\nInput: the body becomes the value of the
record, as text. Turn structured data such [...]
+ "description" : "Sends each message as a record to a topic in Apache
Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or
Redpanda.\n\nUse it to publish events or data for other applications to read,
such as orders, sensor readings or changes from a database. To write the
records with a schema kept in a schema registry, such as Confluent Schema
Registry, Redpanda or Apicurio Registry, set `schemaRegistryVendor`.\n\nInput:
the body becomes the value of the re [...]
"group" : "Kafka",
"namespace" : "Kafka",
"provider" : "Apache Software Foundation"
@@ -1415,7 +1415,7 @@
"supportLevel" : "Stable",
"title" : "Kafka Source",
"summary" : "Receives each record from one or more topics in Apache Kafka,
or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or
Redpanda.",
- "description" : "Receives each record from one or more topics in Apache
Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or
Redpanda.\n\nUse it to react to events or data that other applications publish,
such as orders, sensor readings or changes from a database. To get many records
in one message use `kafka-batch-source`. When the records are written with a
schema kept in a schema registry, use one of the `kafka-*-registry-*` sources
instead.\n\nOutput: th [...]
+ "description" : "Receives each record from one or more topics in Apache
Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or
Redpanda.\n\nUse it to react to events or data that other applications publish,
such as orders, sensor readings or changes from a database. To get many records
in one message use `kafka-batch-source`. When the records are written with a
schema kept in a schema registry, such as Confluent Schema Registry, Redpanda
or Apicurio Registry, [...]
"group" : "Kafka",
"namespace" : "Kafka",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/kafka-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/kafka-sink.kamelet.yaml
index 201ceedec..90b6d240a 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/kafka-sink.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/kafka-sink.kamelet.yaml
@@ -33,13 +33,13 @@ spec:
description: |-
Sends each message as a record to a topic in Apache Kafka, or in a
Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
- Use it to publish events or data for other applications to read, such as
orders, sensor readings or changes from a database. When the records must
follow a schema kept in a schema registry, use one of the `kafka-*-registry-*`
sinks instead.
+ Use it to publish events or data for other applications to read, such as
orders, sensor readings or changes from a database. To write the records with a
schema kept in a schema registry, such as Confluent Schema Registry, Redpanda
or Apicurio Registry, set `schemaRegistryVendor`.
- Input: the body becomes the value of the record, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional `key` header (or `kafka-key`, `ce-key`,
`ce-kafkakey`) becomes the key of the record, and the optional `partition-key`
header (or `ce-partitionkey`) is the number of the partition to write to. The
other headers of the message, except the Camel ones, are sent as record headers.
+ Input: the body becomes the value of the record, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. With `schemaRegistryVendor` and `schemaFormat` `json`,
the body must be JSON text (or a Map), and the record is written with a JSON
Schema; with `avro`, `confluent` and `apicurio` write a text body as an Avro
string, while `aws-glue` needs an Avro record. With `azure`, set `schemaGroup`.
The optional `key` header (or `kafka-key [...]
Output: the message is unchanged.
- Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS, as a local Kafka expects. A managed Kafka
service needs one of the other types.
+ Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS, as a local Kafka expects. A managed Kafka
service needs one of the other types. With `schemaRegistryVendor`, the
integration needs the serde jars of the vendor, which Camel JBang downloads by
itself; for `confluent` add the Confluent Maven repository, as in `camel run
--repos=confluent`. With `aws-glue` and `azure`, the serde logs in with the
credentials of the environment the int [...]
required:
- topic
- bootstrapServers
@@ -122,7 +122,36 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ schemaRegistryVendor:
+ title: Schema Registry Vendor
+ description: "The schema registry that holds the schemas of the record
values: `confluent`, also for registries with the Confluent API such as
Redpanda and Karapace; `apicurio` (Apicurio Registry 3.x); `aws-glue` (AWS Glue
Schema Registry); or `azure` (Azure Schema Registry of Event Hubs). It sets the
value serializer and the registry properties, and its jars must be added to the
integration (see below). Leave it unset when the records do not use a schema
registry."
+ type: string
+ enum: ["confluent", "apicurio", "aws-glue", "azure"]
+ schemaRegistryUrl:
+ title: Schema Registry URL
+ description: "The URL of the schema registry: for `confluent` such as
`http://localhost:8081`; for `apicurio` the URL of the registry API, such as
`http://localhost:8080/apis/registry/v3`; for `azure` the fully qualified name
of the Event Hubs namespace, such as `my-namespace.servicebus.windows.net`.
Required when `schemaRegistryVendor` is set, except for `aws-glue`, where it
can override the endpoint of the Glue service."
+ type: string
+ pattern: "^(https?://)?[^\\s]+$"
+ schemaRegistryRegion:
+ title: Schema Registry Region
+ description: "The AWS region of the AWS Glue Schema Registry, such as
`eu-west-1`. When not set, the region is taken from the environment, such as
the `AWS_REGION` variable. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaRegistryName:
+ title: Schema Registry Name
+ description: "The name of the registry in AWS Glue. When not set, it
is `default-registry`. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaGroup:
+ title: Schema Group
+ description: "The schema group in the Azure Schema Registry that the
schemas of the records are registered in. Required when `schemaRegistryVendor`
is `azure`."
+ type: string
+ schemaFormat:
+ title: Schema Format
+ description: "The format of the record values in the schema registry:
`avro` or `json` (JSON Schema). With `apicurio` and `azure`, use `avro`. Only
used when `schemaRegistryVendor` is set."
+ type: string
+ enum: ["avro", "json"]
+ default: avro
dependencies:
+ - "camel:jackson3"
- "camel:kafka"
- "camel:kamelet"
template:
@@ -187,10 +216,22 @@ spec:
expression:
simple:
expression: "${header[ce-partitionkey]}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{?schemaRegistryVendor}}' != '' &&
'{{schemaFormat}}' == 'json' && ${body} != null && ${body} !is 'java.util.Map'
&& ${body} !is 'java.util.List'"
+ steps:
+ - unmarshal:
+ json:
+ library: Jackson
- to:
uri: kafka
parameters:
topic: "{{topic}}"
+ additionalProperties.region: "{{?schemaRegistryRegion}}"
+ additionalProperties.registry.name: "{{?schemaRegistryName}}"
+ additionalProperties.schema.group: "{{?schemaGroup}}"
brokers: "{{bootstrapServers}}"
oauthClientId: "{{?oauthClientId}}"
oauthClientSecret: "RAW({{?oauthClientSecret}})"
@@ -199,6 +240,9 @@ spec:
saslAuthType: "{{saslAuthType}}"
saslPassword: "RAW({{?saslPassword}})"
saslUsername: "{{?saslUsername}}"
+ schemaFormat: "{{schemaFormat}}"
+ schemaRegistryURL: "{{?schemaRegistryUrl}}"
+ schemaRegistryVendor: "{{?schemaRegistryVendor}}"
sslKeyPassword: "RAW({{?sslKeyPassword}})"
sslKeystoreLocation: "{{?sslKeystoreLocation}}"
sslKeystorePassword: "RAW({{?sslKeystorePassword}})"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/kafka-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/kafka-source.kamelet.yaml
index 653b1980f..bec2b948c 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/kafka-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/kafka-source.kamelet.yaml
@@ -33,11 +33,11 @@ spec:
description: |-
Receives each record from one or more topics in Apache Kafka, or in a
Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
- Use it to react to events or data that other applications publish, such
as orders, sensor readings or changes from a database. To get many records in
one message use `kafka-batch-source`. When the records are written with a
schema kept in a schema registry, use one of the `kafka-*-registry-*` sources
instead.
+ Use it to react to events or data that other applications publish, such
as orders, sensor readings or changes from a database. To get many records in
one message use `kafka-batch-source`. When the records are written with a
schema kept in a schema registry, such as Confluent Schema Registry, Redpanda
or Apicurio Registry, set `schemaRegistryVendor`.
- Output: the body is the value of the record, as text. A record without a
value (a tombstone) gives a message with no body. The `kafka-topic`,
`kafka-key`, `kafka-partition`, `kafka-offset` and `kafka-timestamp` (in
milliseconds since 1970) headers say where the record came from. They are also
set as `ce-kafkatopic`, `ce-kafkakey` and so on, and as the `CamelKafka*`
headers that the Kafka actions read. The record headers become message headers,
as text unless `deserializeHeaders` is false.
+ Output: the body is the value of the record, as text. With
`schemaRegistryVendor`, the value is read with its schema from the registry and
the body is JSON text, such as `{"id": 7, "item": "camel"}` for an Avro record.
A record without a value (a tombstone) gives a message with no body. The
`kafka-topic`, `kafka-key`, `kafka-partition`, `kafka-offset` and
`kafka-timestamp` (in milliseconds since 1970) headers say where the record
came from. They are also set as `ce-kafkatopic`, `ce [...]
- Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS. A new consumer group starts at the newest
records, unless `autoOffsetReset` is `earliest`. Without `consumerGroup`, every
start uses a new group, so the records sent while the integration was stopped
are skipped.
+ Good to know: `saslAuthType` has no default and must be set; `NONE`
connects with no login and no TLS. A new consumer group starts at the newest
records, unless `autoOffsetReset` is `earliest`. Without `consumerGroup`, every
start uses a new group, so the records sent while the integration was stopped
are skipped. With `schemaRegistryVendor`, the integration needs the serde jars
of the vendor, which Camel JBang downloads by itself; for `confluent` add the
Confluent Maven repository [...]
required:
- topic
- bootstrapServers
@@ -161,6 +161,30 @@ spec:
description: "If true, `topic` is a regular expression, such as
`orders-.*`, and the Kamelet reads from every topic whose name matches it."
type: boolean
default: false
+ schemaRegistryVendor:
+ title: Schema Registry Vendor
+ description: "The schema registry that holds the schemas of the record
values: `confluent`, also for registries with the Confluent API such as
Redpanda and Karapace; `apicurio` (Apicurio Registry 3.x); `aws-glue` (AWS Glue
Schema Registry); or `azure` (Azure Schema Registry of Event Hubs). It sets the
value deserializer and the registry properties, and its jars must be added to
the integration (see below). Leave it unset when the records do not use a
schema registry."
+ type: string
+ enum: ["confluent", "apicurio", "aws-glue", "azure"]
+ schemaRegistryUrl:
+ title: Schema Registry URL
+ description: "The URL of the schema registry: for `confluent` such as
`http://localhost:8081`; for `apicurio` the URL of the registry API, such as
`http://localhost:8080/apis/registry/v3`; for `azure` the fully qualified name
of the Event Hubs namespace, such as `my-namespace.servicebus.windows.net`.
Required when `schemaRegistryVendor` is set, except for `aws-glue`, where it
can override the endpoint of the Glue service."
+ type: string
+ pattern: "^(https?://)?[^\\s]+$"
+ schemaRegistryRegion:
+ title: Schema Registry Region
+ description: "The AWS region of the AWS Glue Schema Registry, such as
`eu-west-1`. When not set, the region is taken from the environment, such as
the `AWS_REGION` variable. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaRegistryName:
+ title: Schema Registry Name
+ description: "The name of the registry in AWS Glue. When not set, it
is `default-registry`. Only used when `schemaRegistryVendor` is `aws-glue`."
+ type: string
+ schemaFormat:
+ title: Schema Format
+ description: "The format of the record values in the schema registry:
`avro` or `json` (JSON Schema). With `apicurio` and `azure`, use `avro`. Only
used when `schemaRegistryVendor` is set."
+ type: string
+ enum: ["avro", "json"]
+ default: avro
dataTypes:
out:
headers:
@@ -205,6 +229,7 @@ spec:
description: "The same as `kafka-timestamp`."
type: integer
dependencies:
+ - "camel:jackson3"
- "camel:kafka"
- "camel:kamelet"
template:
@@ -217,6 +242,8 @@ spec:
uri: kafka
parameters:
topic: "{{topic}}"
+ additionalProperties.region: "{{?schemaRegistryRegion}}"
+ additionalProperties.registry.name: "{{?schemaRegistryName}}"
allowManualCommit: "{{allowManualCommit}}"
autoCommitEnable: "{{autoCommitEnable}}"
autoOffsetReset: "{{autoOffsetReset}}"
@@ -230,6 +257,9 @@ spec:
saslAuthType: "{{saslAuthType}}"
saslPassword: "RAW({{?saslPassword}})"
saslUsername: "{{?saslUsername}}"
+ schemaFormat: "{{schemaFormat}}"
+ schemaRegistryURL: "{{?schemaRegistryUrl}}"
+ schemaRegistryVendor: "{{?schemaRegistryVendor}}"
sslKeyPassword: "RAW({{?sslKeyPassword}})"
sslKeystoreLocation: "{{?sslKeystoreLocation}}"
sslKeystorePassword: "RAW({{?sslKeystorePassword}})"
@@ -294,5 +324,24 @@ spec:
expression:
simple:
expression: "${header[CamelKafkaTimestamp]}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{?schemaRegistryVendor}}' != '' &&
'{{schemaFormat}}' == 'json' && ${body} != null && ${body} !is
'java.lang.String'"
+ steps:
+ - marshal:
+ json:
+ library: Jackson
+ - convertBodyTo:
+ type: java.lang.String
+ - expression:
+ simple:
+ expression: "'{{?schemaRegistryVendor}}' != '' && ${body}
!= null && ${body} !is 'java.lang.String'"
+ steps:
+ - setBody:
+ expression:
+ simple:
+ expression: "${body.toString()}"
- to:
uri: kamelet:sink
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-schema-registry-route.citrus.it.yaml
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-schema-registry-route.citrus.it.yaml
new file mode 100644
index 000000000..7e23d566e
--- /dev/null
+++
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-schema-registry-route.citrus.it.yaml
@@ -0,0 +1,71 @@
+# ---------------------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ---------------------------------------------------------------------------
+
+name: kafka-schema-registry-route-test
+variables:
+ - name: "kafka.deserializeHeaders"
+ value: "true"
+ - name: "kafka.topic"
+ value: "my-topic-registry"
+ - name: "kafka.message"
+ value: '{"id":1,"item":"camel"}'
+actions:
+ # Create Kafka infrastructure, Redpanda has a schema registry with the
Confluent API
+ - testcontainers:
+ start:
+ redpanda: {}
+
+ # Create Camel JBang integration, the Confluent serde jars are in the
Confluent Maven repository
+ - camel:
+ cli:
+ run:
+ args:
+ - "--repos=confluent"
+ integration:
+ file: "kafka/kafka-schema-registry-route.yaml"
+ systemProperties:
+ file: "kafka/application.properties"
+ properties:
+ - name: kafka.schemaRegistryUrl
+ value:
"${CITRUS_TESTCONTAINERS_REDPANDA_SCHEMA_REGISTRY_ADDRESS}"
+ - name: http.sink.url
+ value: "http://localhost:${kafka.sink.server.port}/result"
+
+ # kafka-sink writes the record with a JSON Schema, kafka-source reads it and
emits JSON text
+ - http:
+ server: "kafkaSinkServer"
+ receiveRequest:
+ POST:
+ path: "/result"
+ body:
+ data: |
+ ${kafka.message}
+
+ # The record was written through the schema registry
+ - http:
+ client:
"${CITRUS_TESTCONTAINERS_REDPANDA_SCHEMA_REGISTRY_ADDRESS}/subjects/${kafka.topic}-value/versions/latest"
+ sendRequest:
+ GET: {}
+
+ - http:
+ client:
"${CITRUS_TESTCONTAINERS_REDPANDA_SCHEMA_REGISTRY_ADDRESS}/subjects/${kafka.topic}-value/versions/latest"
+ receiveResponse:
+ response:
+ status: 200
+ body:
+ data: |
+
{"subject":"${kafka.topic}-value","version":"@ignore@","id":"@ignore@","schemaType":"JSON","schema":"@ignore@"}
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-schema-registry-route.yaml
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-schema-registry-route.yaml
new file mode 100644
index 000000000..4f1163c9f
--- /dev/null
+++
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-schema-registry-route.yaml
@@ -0,0 +1,48 @@
+# ---------------------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ---------------------------------------------------------------------------
+
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ period: 10000
+ contentType: application/json
+ message: '{"id":1,"item":"camel"}'
+ steps:
+ - to:
+ uri: "kamelet:kafka-sink"
+ parameters:
+ bootstrapServers: '{{kafka.bootstrapServers}}'
+ saslAuthType: NONE
+ topic: '{{kafka.topic}}'
+ schemaRegistryVendor: confluent
+ schemaRegistryUrl: '{{kafka.schemaRegistryUrl}}'
+ schemaFormat: json
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ bootstrapServers: '{{kafka.bootstrapServers}}'
+ saslAuthType: NONE
+ topic: '{{kafka.topic}}'
+ autoOffsetReset: earliest
+ schemaRegistryVendor: confluent
+ schemaRegistryUrl: '{{kafka.schemaRegistryUrl}}'
+ schemaFormat: json
+ steps:
+ - to:
+ uri: "{{http.sink.url}}"