Claus Ibsen created CAMEL-27136:
-----------------------------------

             Summary: camel-kafka - show and check the schemas of a schema 
registry in the dev console, TUI and Camel JBang
                 Key: CAMEL-27136
                 URL: https://issues.apache.org/jira/browse/CAMEL-27136
             Project: Camel
          Issue Type: New Feature
          Components: camel-jbang, camel-kafka
            Reporter: Claus Ibsen
             Fix For: 4.25.0


With CAMEL-27130 a Kafka endpoint knows its schema registry: the vendor, the 
registry URL, and the registry login (in additionalProperties). Camel could use 
that to show the schemas, and to check data and new schemas against them, in 
the dev consoles, the TUI, Camel JBang and the tooling.

*What the registries offer*

The registries have APIs to read schemas back. The table below is from the 
vendors' documentation and has not been verified for every API version:

|| Registry || API to read schemas || How a record points to its schema ||
| Confluent, and registries that implement its API (Karapace, Redpanda, 
Apicurio through /apis/ccompat/v7) | REST: /subjects, 
/subjects/\{subject\}/versions/latest, /schemas/ids/\{id\}, a compatibility 
check, and the compatibility level | A magic byte and a 4-byte schema id at the 
start of the value |
| Apicurio Registry 3 | Its own REST API, /apis/registry/v3/... | A schema id 
in a record header or a prefix |
| AWS Glue | The AWS API (GetSchemaVersion, ...), with IAM | A schema version 
UUID prefix |
| Azure Schema Registry | REST through the Azure SDK, with Azure AD | A content 
type such as avro/binary+<schemaId> |

The Confluent-compatible REST API covers several registries with plain HTTP, 
and needs no vendor jars.

*What Camel already has*
* The camel-kafka configuration (after CAMEL-27130): schemaRegistryVendor, 
schemaRegistryURL and schemaFormat. By default, the subject of a topic is 
<topic>-value.
* KafkaDevConsole, which keeps the last record of each consumer: topic, 
partition and offset, and the group and commit state.
* With Avro, the body that Camel receives is a GenericRecord that carries its 
Schema, so the schema actually used is known at runtime without calling the 
registry.
* camel-apicurio-registry (new in 4.24), which has getArtifactContent, 
listVersions, testCompatibility and validate operations, for Apicurio only.

*Proposal, in steps*

# *Dev console (and so the TUI and camel get):* for each Kafka endpoint with a 
schemaRegistryVendor, show:
#* the registry and the subject;
#* the latest schema version and id;
#* the compatibility level and the schema text;
#* for consumers, the schema of the last record received, from 
GenericRecord.getSchema().
# *Camel JBang:* a camel kafka schema command, on the Confluent-compatible API 
to begin with:
#* list the subjects, and show a schema;
#* check a local .avsc, JSON Schema or .proto file for compatibility with the 
latest version, before deploying, which is useful in CI.
# *Validation:*
#* At design time, the YAML validator and the MCP server fetch the schema of a 
topic, to check data mappings and to describe the body of a Kamelet for AI 
assistants. This relates to the body type work in CAMEL-24844.
#* At runtime, an opt-in check of the body against the JSON Schema of the 
registry before sending, for clearer errors than the serializer gives.

AWS Glue and Azure need their SDKs and cloud logins, so they can come after the 
Confluent-compatible API and Apicurio.

Related: CAMEL-27130, CAMEL-24844, CAMEL-24172, apache/camel-kamelets#3183.

_Claude Code on behalf of Claus Ibsen_




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to