mchades commented on code in PR #12424: URL: https://github.com/apache/gravitino/pull/12424#discussion_r3797050047
########## design-docs/gravitino-semantic-model-design.md: ########## @@ -0,0 +1,670 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Semantic Model Support in Gravitino + +## Background + +Business definitions such as revenue, active users, dimensions, and dataset relationships are +shared semantic assets consumed by analytics, BI, and AI applications. When these definitions live +only in individual tools or project files, their identity, discovery, ownership, access control, +auditability, and reuse become fragmented. Gravitino needs a governed and engine-neutral metadata +object for managing these definitions alongside the data entities they reference. + +Unlike a logical View, which defines a fixed query and output schema, a semantic model defines +datasets, relationships, dimensions, and metrics that consumers combine at query time. Gravitino +therefore models it as a dedicated metadata object rather than a relational View. + +Apache Ossie, formerly Open Semantic Interchange (OSI), defines a vendor-neutral structured +`SemanticModel`. + +Gravitino's design direction for semantic metadata has evolved from preserving opaque YAML +documents, to exposing a strongly typed model, and now to managing it as an independent first-class +metadata object. This progression reflects the growing importance of semantics in Gravitino's +metadata and governance model. This design therefore adopts an Ossie-compatible analytical model as +a new schema-scoped Gravitino entity. + +## Goals + +1. **Lifecycle.** Manage Ossie-compatible Semantic Models as first-class, schema-scoped metadata + with stable identity and a dedicated lifecycle. +2. **Governance.** Apply authorization, ownership, audit, events, tags, and policies to Semantic + Models. +3. **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent + validation. +4. **Interoperability.** Define a stable Ossie-compatible contract that can evolve with Apache Ossie. +5. **User experience.** Support discovery and lifecycle management of Semantic Models as a distinct + schema-scoped object category in the Gravitino UI. + +## Non-Goals + +1. **Query compilation and execution.** Semantic query planning, SQL generation, engine execution, + and engine-specific query syntax are separate work. +2. **Non-Ossie native models.** Native dbt, Cube, Databricks, Snowflake, or other vendor-specific + semantic definitions are not modeled by this design. +3. **Document authoring and conversion.** YAML or JSON import, export, formatting, conversion, and + exact textual round trips are not server API contracts. +4. **Ontology management.** Apache Ossie Ontology definitions and mappings are separate metadata + concepts and require their own design. +5. **Materialization.** Metric caches, refresh policies, pre-aggregations, and materialized results + are not defined here. +6. **Member-level authorization.** Datasets, fields, relationships, and metrics are governed as + members of their enclosing Semantic Model rather than as independent securable objects. + +## Proposal + +### Object Model and Namespace + +`SemanticModel` is a new metadata entity under a schema, alongside Tables, Views, Functions, and +other schema-scoped objects. + +```text +metalake + catalog + schema + Table + View + SemanticModel +``` + +The implementation introduces `MetadataObject.Type.SEMANTIC_MODEL` and +`Entity.EntityType.SEMANTIC_MODEL`. Public Java types reside in a dedicated semantic package rather +than the relational View package. + +- **Identity.** The Semantic Model entity name maps to the Ossie `SemanticModel.name` field when + serialized. No separate nested model name or identity is stored. +- **Scope.** A Semantic Model belongs to one schema. Its fully qualified identity is + `metalake.catalog.schema.semanticModel`. +- **Namespace.** Semantic Models have an independent typed namespace. A Semantic Model may have the + same name as a Table, View, Function, or another entity type in the same schema, but two Semantic + Models with the same name cannot coexist. +- **Source of truth.** Semantic Models are always managed by Gravitino and are never persisted in an + underlying catalog. +- **One model per entity.** One Ossie `semantic_model` item maps to one Gravitino Semantic Model. An + external document containing multiple items maps to multiple entities. + +The term `SemanticModel` means an **analytical semantic model** compatible with the Ossie Core +specification. It is not an umbrella for every semantic artifact. The +[Ossie Ontology specification](https://github.com/apache/ossie/blob/88e0011148283302c9a04cd0287e00e0b9d87354/ontology/ontology.json) +defines Ontology as a separate document and maps logical semantic models to ontology concepts. +Gravitino should likewise introduce a separate `Ontology` entity and an explicit binding or mapping +contract if that capability is added later. Such a binding should reference the stable identities of +the Ontology and Semantic Model rather than embed a second copy of either definition. + +`SemanticModel` is a separate metadata type and lifecycle from the existing Gravitino `Model`, which +represents an ML model artifact. + +### Semantic Model Contract + +The API exposes a structured, immutable model. The initial contract follows the +[Apache Ossie Core schema pinned at commit `88e0011`](https://github.com/apache/ossie/blob/88e0011148283302c9a04cd0287e00e0b9d87354/core-spec/osi-schema.json), +whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other +fields are required. Language-specific APIs follow their normal naming conventions, while this +document uses Ossie field names for comparison with the upstream schema. + +The Gravitino entity combines common metadata with the Ossie-compatible definition: + +```text +SemanticModel + name: string + comment?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] + properties: map<string, string> + audit_info: AuditInfo +``` + +- `name` is the Gravitino entity name and maps to Ossie `SemanticModel.name`. +- `comment` follows the common Gravitino entity convention and maps to Ossie + `SemanticModel.description`. +- `properties` stores Gravitino-specific metadata and is not part of an exported Ossie model. +- `custom_extensions` is part of the semantic definition and is preserved for Ossie interchange. +- Collection order is preserved so consumers can produce stable serialized output. + +#### Datasets and Fields + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + datatype?: DataType + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- Dataset names are unique within a Semantic Model. Field names are unique within each Dataset. +- `source` is a three-part `NameIdentifier` in the form `catalog.schema.name`. The request already + identifies the metalake, so cross-catalog references within that metalake are allowed while + cross-metalake references are not. +- A source must resolve to either a Table or a logical View. The semantic definition does not need + to declare which of those two types it references; validation succeeds when either entity exists. +- Apache Ossie does not currently define cross-model dataset references. Therefore, a Semantic Model + is not a valid Dataset source; cross-model composition is deferred until an explicit compatible + reference contract is defined. +- Inline query sources are not supported. A query-backed source must first be created as a logical + View and then referenced by `NameIdentifier`. +- For Table and logical View sources, every column named by `primary_key`, `unique_keys`, + `from_columns`, or `to_columns` must exist in the source metadata. Validation rejects the + definition when column metadata is unavailable or a named column is missing; Gravitino does not + infer these references from free-form field or metric expressions. + +#### Relationships and Metrics + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + datatype?: DataType + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- Relationship and Metric names are unique within a Semantic Model. +- Each relationship endpoint names a Dataset in the same Semantic Model. +- `from_columns` and `to_columns` are non-empty, have equal lengths, and name columns exposed by + their respective Dataset sources. +- Metrics may reference fields and datasets in the same Semantic Model. Cross-model metric + references are not defined by this contract. + +#### Supporting Types + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | AIContextObject + +AIContextObject + instructions?: string + synonyms?: string[] + examples?: string[] + additional properties: allowed and retained losslessly + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" Review Comment: We will model `dialect` as an open string with well-known constants instead of a closed enum. The seven listed values are Ossie-defined constants, while Gravitino also accepts and preserves other non-empty identifiers such as `TRINO` or `SPARK`. It does not normalize, translate, or fall back to `ANSI_SQL`. Gravitino does not compile these expressions. Consumers interpret the dialects they support and can report an unsupported-dialect error otherwise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
