mchades commented on code in PR #12360: URL: https://github.com/apache/gravitino/pull/12360#discussion_r3735229486
########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design Review Comment: The complete structured model is stored in the existing `view_version_info.representations` snapshot, and each alter creates a full immutable View version, as already described in the storage section. There is no Metric-View-specific size limit; it follows the existing View storage contract. The entity cache stores only the current View entity and uses the existing View invalidation path. Python and REST are already in scope; OpenAPI updates, View events/listeners, and cache wiring follow the existing View implementation paths. I don’t think a separate feature flag is needed. The background already explains why the structured model was chosen over opaque YAML. I agree that the first reference should define Apache Ossie as the project formerly known as Open Semantic Interchange (OSI). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
