krickert opened a new issue, #11996: URL: https://github.com/apache/gravitino/issues/11996
### Describe the feature Gravitino can catalog Kafka topics today, but the catalog only shows that a topic exists, not what is inside it. The messages on a topic are Protobuf, Avro, or JSON, usually with a schema registered in a schema registry, and none of that appears in the catalog. An engineer who finds a topic in Gravitino still has to dig through a registry, a wiki, or tribal knowledge to learn what the records look like. This feature implements the reserved `DataLayout` placeholder so topics can carry named message schema metadata, conventionally `key` and `value`. Each layout can describe: - a schema registry reference: `schemaUri`, `schemaSubject`, `schemaVersion`, `schemaId` - inline schema text (Avro JSON, `.proto` source, JSON Schema) - a fully qualified type name (for example a Protobuf message FQN) - a payload format (`protobuf`, `avro`, `json`) and vendor-specific properties Layouts are exposed through the Java API (`Topic#dataLayouts()`, `TopicChange.updateDataLayout` / `removeDataLayout` / `removeDataLayouts`), the REST API (`dataLayouts` on topic create/response, three new update operations), and the Java client. They persist in Gravitino's entity store; the Kafka catalog accepts them as entity-store-only metadata since Kafka has no broker-side schema storage. ### Motivation Topic message schemas are the missing piece of messaging metadata. Without them, Gravitino answers "what topics exist" but not "what data flows through them", which limits its value for discovery, lineage, and governance over streaming data. Example use cases: a discovery tool showing that the `orders` topic carries the Protobuf message `com.example.Order` registered under the `order-value` subject, or a governance process flagging topics with no declared schema. The layout fields map directly onto schema registry concepts (Confluent Schema Registry, Apicurio, AWS Glue). This issue covers the declarative data model only. The intent is to follow up with registry integration: a `schema.registry.url` property on the Kafka catalog enabling automatic import of key/value layouts from registry subjects, validation of referenced subjects, and eventually drift detection. Landing the model first keeps the registry connectivity design (auth, caching, registry-down behavior) reviewable on its own. ### Design note Layouts form a named map because messaging payloads carry separate key and value schemas (Schema Registry's `<topic>-key` / `<topic>-value` subjects). A singular layout cannot express the key schema and would force a breaking migration of a released contract later. Implementing this reshapes `TopicCatalog#createTopic`'s `dataLayout` parameter to `Map<String, DataLayout>`. The existing `DataLayout` is an `@Unstable` empty marker interface documented as "currently not implemented, only reserved as a placeholder", and `TopicCatalog` is `@Evolving`, so this is the intended point in the API lifecycle for such a reshape. An implementation is ready and will be submitted as a PR referencing this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
