mchades commented on code in PR #12360: URL: https://github.com/apache/gravitino/pull/12360#discussion_r3726439358
########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,373 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps the root `version` to `MetricRepresentation.osiVersion` and one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + osiVersion: string + semanticModel: MetricModel +``` + +The representation has three fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `osiVersion`: A required string identifying the OSI profile used to interpret and validate the model. The initial supported value is `0.2.0.dev0`. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `osiVersion` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the pinned OSI `0.2.0.dev0` profile. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` with namespace `[catalog, schema]`; `source.name` identifies a `Table` or `View`, and the enclosing object supplies the metalake. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported; register the query as a logical View and reference that View through a `NameIdentifier`. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. Unknown standardized fields are rejected until the declared OSI profile supports them. Review Comment: Yes. For standardized OSI model objects, the schema defines the supported fields and sets `additionalProperties` to `false`. Vendor-specific data remains supported through explicitly defined extension points such as `custom_extensions`. See the [OSI schema definition](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
