mchades commented on code in PR #12360: URL: https://github.com/apache/gravitino/pull/12360#discussion_r3734847534
########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. Review Comment: Introducing validation state would add unnecessary complexity. Since writes are expected to be infrequent, revalidating sources on each write is an acceptable trade-off and ensures that persisted Metric Views remain validated. I agree that transient catalog unavailability should return `503`, while invalid definitions should remain `400`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
