laserninja commented on code in PR #12360: URL: https://github.com/apache/gravitino/pull/12360#discussion_r3733816011
########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. Review Comment: `RepresentationDTO` declares `defaultImpl = SQLRepresentationDTO.class` (`common/src/main/java/org/apache/gravitino/dto/rel/RepresentationDTO.java:37`), so an unrecognized `type` does not fail, it deserializes as a SQL representation. Any rolling upgrade where an older server or client sees `type: "metric"` hits this. Worth calling out removing `defaultImpl` or adding explicit unknown-type rejection. The same class sets `@JsonIgnoreProperties(ignoreUnknown = true)`, which works against strict validation of the metric subtree. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. For standardized OSI model objects, fields not defined by the [pinned OSI schema](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327) are rejected because the schema sets `additionalProperties` to `false`. Gravitino Metric Views enforce the same restriction. + +**Implementation note:** + +Gravitino pins the exact upstream OSI `0.2.0.dev0` JSON Schema used by the structured model and adds Gravitino-specific projection and semantic rules beyond schema validation. The current contract does not persist an OSI version in each Metric View. If a future OSI version introduces an incompatible interpretation, Gravitino can add explicit version metadata and define compatibility behavior then; existing definitions without that metadata retain the initial semantics. Review Comment: Pinning a `.dev0` schema by commit while persisting no version on each Metric View means stored models cannot be interpreted per-version later. Adding version metadata after definitions exist is harder than adding it now, and a `.dev0` schema can be re-cut incompatibly. Suggest persisting the pinned version (or a schema digest) from the start, or marking the first profile explicitly experimental with no compatibility promise until an OSI GA release. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. For standardized OSI model objects, fields not defined by the [pinned OSI schema](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327) are rejected because the schema sets `additionalProperties` to `false`. Gravitino Metric Views enforce the same restriction. + +**Implementation note:** + +Gravitino pins the exact upstream OSI `0.2.0.dev0` JSON Schema used by the structured model and adds Gravitino-specific projection and semantic rules beyond schema validation. The current contract does not persist an OSI version in each Metric View. If a future OSI version introduces an incompatible interpretation, Gravitino can add explicit version metadata and define compatibility behavior then; existing definitions without that metadata retain the initial semantics. + +- All representation and model checks run on `create` and `alter` before a View or View version is persisted. +- Validation checks direct references only. +- Transitive dependency and cycle correctness are not checked; a cyclic definition may be persisted and later rejected by a downstream consumer. +- Catalog changes do not trigger automatic revalidation. +- Catalog-wide revalidation is excluded because it would require a dependency index and potentially global impact analysis. + +### Usage + +Metric Views reuse the existing View lifecycle. Their columns are always empty, and create, list, load, alter, and drop operations use the existing View APIs. + +#### Supported Alter Operations + +Metric Views support the existing `ViewChange` operations: + +- `rename`: Renames the enclosing View only; `MetricModel.name` is unchanged. The target name must be available in the shared View namespace. +- `setProperty`: Adds or replaces a View-level property. +- `removeProperty`: Removes a View-level property. +- `replaceView`: Atomically replaces the View body. The `columns` must remain empty, exactly one `MetricRepresentation` must remain, and changing between metric and logical semantics is rejected. + +Member-level patch operations are not supported; changes to datasets, relationships, fields, or metrics require replacing the complete `MetricModel`. + +Metric Views do not use `defaultCatalog` or `defaultSchema` because dataset sources use `NameIdentifier`; both values must be `null` in create and `replaceView` requests. + +#### Java API + +The Java API uses immutable builders for the structured definition and the existing ViewCatalog lifecycle methods: + +```java +NameIdentifier ident = NameIdentifier.of("mart", "sales_metrics"); +Dataset orders = + Dataset.builder() + .withName("orders") + .withSource(NameIdentifier.of("sales", "mart", "orders")) + .build(); + +MetricModel model = + MetricModel.builder() + .withName("sales_semantic_model") + .withDatasets(List.of(orders)) + .build(); + +MetricRepresentation representation = + MetricRepresentation.builder() + .withSemanticModel(model) + .build(); + +View created = + catalog.createMetricView( + ident, "Sales metric definitions", List.of(representation), + null, null, Map.of()); +``` + +```java +View loaded = catalog.loadView(ident); +NameIdentifier[] views = catalog.listViews(Namespace.of("mart")); + +MetricModel updatedModel = + MetricModel.builder() + .withName("sales_semantic_model") + .withDescription("Updated sales model") + .withDatasets(List.of(orders)) + .build(); +MetricRepresentation updatedRepresentation = + MetricRepresentation.builder() + .withSemanticModel(updatedModel) + .build(); + +View updated = + catalog.alterView( + ident, + ViewChange.replaceView( + new Column[0], + new Representation[] {updatedRepresentation}, + null, null, "Updated sales metric definitions")); + +boolean dropped = catalog.dropView(ident); +``` + +#### Python API + +The Python API exposes the same structured model and View lifecycle: + +```python +ident = NameIdentifier.of("mart", "sales_metrics") +orders = Dataset("orders", NameIdentifier.of("sales", "mart", "orders")) +model = MetricModel("sales_semantic_model", [orders]) +representation = MetricRepresentation(model) + +created = catalog.create_metric_view( + ident, + [representation], + comment="Sales metric definitions", +) + +loaded = catalog.load_view(ident) +views = catalog.list_views(Namespace.of("mart")) + +updated_model = MetricModel( + "sales_semantic_model", + [orders], + _description="Updated sales model", +) +updated_representation = MetricRepresentation(updated_model) +updated = catalog.alter_view( + ident, + ViewChange.replace_view( + columns=[], + representations=[updated_representation], + comment="Updated sales metric definitions", + ), +) + +dropped = catalog.drop_view(ident) +``` + +#### REST API + +REST uses the existing View resources. Create supplies an empty columns array and one Metric representation: + +```http +POST /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views +{ + "name": "sales_metrics", + "comment": "Sales metric definitions", + "columns": [], + "representations": [ + { + "type": "metric", + "semanticModel": { + "name": "sales_semantic_model", + "datasets": [ + { "name": "orders", "source": { "namespace": ["sales", "mart"], "name": "orders" } } + ] + } + } + ] +} +``` + +List, load, alter, and drop use the same resource: + +```http +GET /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views +GET /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics + +PUT /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics +{ + "updates": [ + { + "@type": "replaceView", + "columns": [], + "representations": [ + { + "type": "metric", + "semanticModel": { + "name": "sales_semantic_model", + "description": "Updated sales model", + "datasets": [ + { "name": "orders", "source": { "namespace": ["sales", "mart"], "name": "orders" } } + ] + } + } + ], + "comment": "Updated sales metric definitions" + } + ] +} + +DELETE /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics +``` + +### Storage and Connector Behavior + +- **Source of truth.** Metric Views are stored only in the Gravitino EntityStore. Logical Views remain stored by their underlying catalogs. Review Comment: Two things here. The store-only read path does not exist today. `ViewOperationDispatcher.internalLoadView` starts with `c.doWithViewOps(v -> v.loadView(ident))` and throws `NoSuchViewException` if the catalog lacks the object (`core/src/main/java/org/apache/gravitino/catalog/ViewOperationDispatcher.java:397`); `listViews` (line 84), `dropView`, and `alterView` follow the same pattern. So a Metric View stored only in the EntityStore is unreachable through all of them. Worth stating where the branch lives (dispatcher short circuit vs. a Gravitino-managed view-ops impl selected by `Capability.Scope.VIEW`), whether a Metric View can be created in an Iceberg/HMS-backed schema, and what happens on schema/catalog drop cascade. "Logical Views remain stored by their underlying catalogs" also conflicts with the accepted logical view design, which defines a fully Gravitino-managed tier for catalogs with no native view support (`design-docs/gravitino-logical-view-management.md:152`). Gravitino-stored logical Views already exist, so the merge and conflict rules should be defined against the existing capability tiers. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], Review Comment: The string-or-object union is awkward for Java and Python bindings and for the OpenAPI schema, and the trailing `...` in the object form is undefined. Suggest normalizing on write (accept a bare string, store it as `instructions`) so reads see one shape, and stating whether unknown keys inside `AIContext` are retained or rejected. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. For standardized OSI model objects, fields not defined by the [pinned OSI schema](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327) are rejected because the schema sets `additionalProperties` to `false`. Gravitino Metric Views enforce the same restriction. + +**Implementation note:** + +Gravitino pins the exact upstream OSI `0.2.0.dev0` JSON Schema used by the structured model and adds Gravitino-specific projection and semantic rules beyond schema validation. The current contract does not persist an OSI version in each Metric View. If a future OSI version introduces an incompatible interpretation, Gravitino can add explicit version metadata and define compatibility behavior then; existing definitions without that metadata retain the initial semantics. + +- All representation and model checks run on `create` and `alter` before a View or View version is persisted. +- Validation checks direct references only. +- Transitive dependency and cycle correctness are not checked; a cyclic definition may be persisted and later rejected by a downstream consumer. +- Catalog changes do not trigger automatic revalidation. +- Catalog-wide revalidation is excluded because it would require a dependency index and potentially global impact analysis. + +### Usage + +Metric Views reuse the existing View lifecycle. Their columns are always empty, and create, list, load, alter, and drop operations use the existing View APIs. + +#### Supported Alter Operations + +Metric Views support the existing `ViewChange` operations: + +- `rename`: Renames the enclosing View only; `MetricModel.name` is unchanged. The target name must be available in the shared View namespace. +- `setProperty`: Adds or replaces a View-level property. +- `removeProperty`: Removes a View-level property. +- `replaceView`: Atomically replaces the View body. The `columns` must remain empty, exactly one `MetricRepresentation` must remain, and changing between metric and logical semantics is rejected. + +Member-level patch operations are not supported; changes to datasets, relationships, fields, or metrics require replacing the complete `MetricModel`. + +Metric Views do not use `defaultCatalog` or `defaultSchema` because dataset sources use `NameIdentifier`; both values must be `null` in create and `replaceView` requests. + +#### Java API + +The Java API uses immutable builders for the structured definition and the existing ViewCatalog lifecycle methods: + +```java +NameIdentifier ident = NameIdentifier.of("mart", "sales_metrics"); +Dataset orders = + Dataset.builder() + .withName("orders") + .withSource(NameIdentifier.of("sales", "mart", "orders")) + .build(); + +MetricModel model = + MetricModel.builder() + .withName("sales_semantic_model") + .withDatasets(List.of(orders)) + .build(); + +MetricRepresentation representation = + MetricRepresentation.builder() + .withSemanticModel(model) + .build(); + +View created = + catalog.createMetricView( Review Comment: This contradicts the premise that existing View APIs manage both kinds; load, alter, and drop below and the whole REST section use the generic resource. Suggest `createView(ident, comment, new Column[0], new Representation[] {representation}, null, null, Map.of())`. Related: the design never says how a caller detects a Metric View. A `Representation.TYPE_METRIC` constant next to `TYPE_SQL` (`api/src/main/java/org/apache/gravitino/rel/Representation.java:29`, whose javadoc still says SQL is the only supported type) and a `View.metricRepresentation()` accessor mirroring `sqlFor(dialect)` (`View.java:97`) would help. Empty `columns` cannot be the discriminator: `ViewCreateRequest.validate()` already permits empty columns for logical views (`ViewCreateRequest.java:110`). ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" Review Comment: This creates a second dialect vocabulary alongside the existing `Dialects` type (`trino`, `spark`, `hive`, `flink`) in the same View object, with different naming conventions and different extensibility. A closed enum also makes every new OSI dialect a Gravitino API change plus a hard write-time rejection. Suggest an open string with well-known constants, and a sentence on what a consumer does when its dialect is absent from an `Expression`. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. Review Comment: Since member-level patches are unsupported, changing a description means a full `replaceView`, which revalidates every source, so one unreachable catalog blocks a metadata-only edit. Worth deciding whether source validation is skippable (view marked unvalidated) or config gated, and stating 503 rather than 400 for the retriable case. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. Review Comment: This appears to contradict the next bullet, "It does not infer source-column references from field or metric expressions." Expressions are opaque per-dialect strings, so there is nothing to resolve without parsing them. Either drop the claim or define which structured fields count as an internal reference. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. For standardized OSI model objects, fields not defined by the [pinned OSI schema](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327) are rejected because the schema sets `additionalProperties` to `false`. Gravitino Metric Views enforce the same restriction. + +**Implementation note:** + +Gravitino pins the exact upstream OSI `0.2.0.dev0` JSON Schema used by the structured model and adds Gravitino-specific projection and semantic rules beyond schema validation. The current contract does not persist an OSI version in each Metric View. If a future OSI version introduces an incompatible interpretation, Gravitino can add explicit version metadata and define compatibility behavior then; existing definitions without that metadata retain the initial semantics. + +- All representation and model checks run on `create` and `alter` before a View or View version is persisted. +- Validation checks direct references only. +- Transitive dependency and cycle correctness are not checked; a cyclic definition may be persisted and later rejected by a downstream consumer. +- Catalog changes do not trigger automatic revalidation. +- Catalog-wide revalidation is excluded because it would require a dependency index and potentially global impact analysis. + +### Usage + +Metric Views reuse the existing View lifecycle. Their columns are always empty, and create, list, load, alter, and drop operations use the existing View APIs. + +#### Supported Alter Operations + +Metric Views support the existing `ViewChange` operations: + +- `rename`: Renames the enclosing View only; `MetricModel.name` is unchanged. The target name must be available in the shared View namespace. +- `setProperty`: Adds or replaces a View-level property. +- `removeProperty`: Removes a View-level property. +- `replaceView`: Atomically replaces the View body. The `columns` must remain empty, exactly one `MetricRepresentation` must remain, and changing between metric and logical semantics is rejected. + +Member-level patch operations are not supported; changes to datasets, relationships, fields, or metrics require replacing the complete `MetricModel`. + +Metric Views do not use `defaultCatalog` or `defaultSchema` because dataset sources use `NameIdentifier`; both values must be `null` in create and `replaceView` requests. + +#### Java API + +The Java API uses immutable builders for the structured definition and the existing ViewCatalog lifecycle methods: + +```java +NameIdentifier ident = NameIdentifier.of("mart", "sales_metrics"); +Dataset orders = + Dataset.builder() + .withName("orders") + .withSource(NameIdentifier.of("sales", "mart", "orders")) + .build(); + +MetricModel model = + MetricModel.builder() + .withName("sales_semantic_model") + .withDatasets(List.of(orders)) + .build(); + +MetricRepresentation representation = + MetricRepresentation.builder() + .withSemanticModel(model) + .build(); + +View created = + catalog.createMetricView( + ident, "Sales metric definitions", List.of(representation), + null, null, Map.of()); +``` + +```java +View loaded = catalog.loadView(ident); +NameIdentifier[] views = catalog.listViews(Namespace.of("mart")); + +MetricModel updatedModel = + MetricModel.builder() + .withName("sales_semantic_model") + .withDescription("Updated sales model") + .withDatasets(List.of(orders)) + .build(); +MetricRepresentation updatedRepresentation = + MetricRepresentation.builder() + .withSemanticModel(updatedModel) + .build(); + +View updated = + catalog.alterView( + ident, + ViewChange.replaceView( + new Column[0], + new Representation[] {updatedRepresentation}, + null, null, "Updated sales metric definitions")); + +boolean dropped = catalog.dropView(ident); +``` + +#### Python API + +The Python API exposes the same structured model and View lifecycle: + +```python +ident = NameIdentifier.of("mart", "sales_metrics") +orders = Dataset("orders", NameIdentifier.of("sales", "mart", "orders")) +model = MetricModel("sales_semantic_model", [orders]) +representation = MetricRepresentation(model) + +created = catalog.create_metric_view( + ident, + [representation], + comment="Sales metric definitions", +) + +loaded = catalog.load_view(ident) +views = catalog.list_views(Namespace.of("mart")) + +updated_model = MetricModel( + "sales_semantic_model", + [orders], + _description="Updated sales model", +) +updated_representation = MetricRepresentation(updated_model) +updated = catalog.alter_view( + ident, + ViewChange.replace_view( + columns=[], + representations=[updated_representation], + comment="Updated sales metric definitions", + ), +) + +dropped = catalog.drop_view(ident) +``` + +#### REST API + +REST uses the existing View resources. Create supplies an empty columns array and one Metric representation: + +```http +POST /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views +{ + "name": "sales_metrics", + "comment": "Sales metric definitions", + "columns": [], + "representations": [ + { + "type": "metric", + "semanticModel": { + "name": "sales_semantic_model", + "datasets": [ + { "name": "orders", "source": { "namespace": ["sales", "mart"], "name": "orders" } } + ] + } + } + ] +} +``` + +List, load, alter, and drop use the same resource: + +```http +GET /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views +GET /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics + +PUT /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics +{ + "updates": [ + { + "@type": "replaceView", + "columns": [], + "representations": [ + { + "type": "metric", + "semanticModel": { + "name": "sales_semantic_model", + "description": "Updated sales model", + "datasets": [ + { "name": "orders", "source": { "namespace": ["sales", "mart"], "name": "orders" } } + ] + } + } + ], + "comment": "Updated sales metric definitions" + } + ] +} + +DELETE /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics +``` + +### Storage and Connector Behavior + +- **Source of truth.** Metric Views are stored only in the Gravitino EntityStore. Logical Views remain stored by their underlying catalogs. +- **Listing.** The server merges authorized catalog-backed logical Views with authorized Gravitino-managed Metric Views into the existing View listing. +- **Connector capability.** A connector that does not support Metric Views filters them from `listViews` and returns an explicit unsupported-Metric-View error for a direct `loadView`. Generic REST and Java View APIs continue to expose them. +- **Namespace conflicts.** Create checks both storage sources. If an external client later creates a same-name logical View directly in the catalog, list and load report a conflict and select neither object. The external operation must rename or remove its object; ownership, versions, tags, and policies remain attached to the Gravitino Metric View and never transfer. Review Comment: This failure mode worries me: an external client creating a same-name view in the catalog silently disables `loadView` for a governed Gravitino object, including the operations an owner needs to resolve it. Does the conflict fail the whole `listViews` call or just that entry, and how does an owner rename or drop the Metric View while it is conflicting? I would keep the Gravitino-managed object authoritative and loadable with a conflict indicator instead. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design Review Comment: A few sections reviewers will likely ask for: storage impact (does the whole model live in `view_version_info.representations` JSON, and is there a size limit, given every alter copies it into a new version?); OpenAPI updates under `docs/open-api`; Python client scope; audit and event listeners; feature flag; entity cache interaction (`design-docs/cache-improvement-design.md`). Also worth defining OSI/Ossie on first use, and folding in a short rationale for this shape versus the superseded alternative. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship Review Comment: `Relationship` has no cardinality or join type, so fan-out and double-counting cannot be planned safely; `Metric` has no result type, aggregation semantics, or declared grain; `Dimension` carries only `is_time`, with no granularity, and there is no default time dimension. If these are upstream OSI limits, saying so and naming `custom_extensions` as the interim extension point would set expectations. Separately, field names are unique per `Dataset` and metric names per model, but consumers reference both in one query namespace: worth a cross-collection uniqueness rule or a precedence definition. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. For standardized OSI model objects, fields not defined by the [pinned OSI schema](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327) are rejected because the schema sets `additionalProperties` to `false`. Gravitino Metric Views enforce the same restriction. + +**Implementation note:** + +Gravitino pins the exact upstream OSI `0.2.0.dev0` JSON Schema used by the structured model and adds Gravitino-specific projection and semantic rules beyond schema validation. The current contract does not persist an OSI version in each Metric View. If a future OSI version introduces an incompatible interpretation, Gravitino can add explicit version metadata and define compatibility behavior then; existing definitions without that metadata retain the initial semantics. + +- All representation and model checks run on `create` and `alter` before a View or View version is persisted. +- Validation checks direct references only. +- Transitive dependency and cycle correctness are not checked; a cyclic definition may be persisted and later rejected by a downstream consumer. +- Catalog changes do not trigger automatic revalidation. +- Catalog-wide revalidation is excluded because it would require a dependency index and potentially global impact analysis. + +### Usage + +Metric Views reuse the existing View lifecycle. Their columns are always empty, and create, list, load, alter, and drop operations use the existing View APIs. + +#### Supported Alter Operations + +Metric Views support the existing `ViewChange` operations: + +- `rename`: Renames the enclosing View only; `MetricModel.name` is unchanged. The target name must be available in the shared View namespace. +- `setProperty`: Adds or replaces a View-level property. +- `removeProperty`: Removes a View-level property. +- `replaceView`: Atomically replaces the View body. The `columns` must remain empty, exactly one `MetricRepresentation` must remain, and changing between metric and logical semantics is rejected. Review Comment: `replaceView` has no expected-version or ETag precondition, so read-modify-write of the whole model means two concurrent editors silently lose one. Versioning already exists in `view_version_info`, so an optional `expectedVersion` is cheap. If the risk is accepted deliberately, worth saying so. ########## design-docs/gravitino-metric-view-design.md: ########## @@ -0,0 +1,401 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Metric View Support in Gravitino + +## Background + +Business metrics such as revenue, order count, and active users are shared semantic assets consumed by analytics, BI, and AI applications. When their definitions are kept only in individual semantic-layer tools or project files, discovery, ownership, version history, access control, and consistent reuse become fragmented. Gravitino therefore needs a governed metadata model that manages metric definitions alongside the data entities they reference. + +Semantic-layer definitions are commonly authored and exchanged as YAML. That is convenient for authoring and interoperability, but a raw document does not provide Gravitino consumers with a typed API for datasets, relationships, fields, metrics, and AI context. This design introduces an OSI/Ossie-compatible structured representation while retaining the existing View lifecycle and governance model. + +## Goals + +- **Unified lifecycle.** Represent metric definitions as schema-scoped metadata and manage them through the existing View lifecycle. +- **Structured access.** Expose datasets, relationships, fields, metrics, AI context, and extensions through typed APIs. +- **Governance.** Apply View-level identity, authorization, ownership, audit, tags, policies, and version history to metric definitions. +- **Compatibility.** Preserve existing logical View behavior and provide explicit capability handling for connectors that do not support Metric Views. +- **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent validation. + +## Non-Goals + +- **Non-OSI native models.** Compatibility with dbt, Cube, Databricks, Snowflake, or other non-OSI semantic definitions is outside this design. +- **Document authoring and conversion.** YAML parsing, formatting, conversion, and exact textual round trips are not server API contracts. External tools may provide best-effort stable serialization. +- **Compilation and execution.** Semantic query planning, SQL generation, engine execution, and engine-specific compatibility are separate work. +- **Materialization.** Metric caches, refresh policies, and materialized results are not defined here. +- **Continuous dependency maintenance.** Catalog-wide lineage, automatic revalidation after catalog changes, and transitive cycle analysis are not included. +- **Member-level authorization.** Datasets, fields, and metrics are governed as members of the enclosing Metric View rather than as independently authorized entities. + +## Proposed Design + +### Object Model and Constraints + +A Metric View is a specialized use of the existing View object under a metalake, catalog, and schema. It does not introduce a new top-level metadata object. + +```text +metalake.catalog.schema + View (logical) + SQLRepresentation + View (metric) + MetricRepresentation +``` + +- **Containment and governance.** The enclosing Metric View is the governed object. Datasets, relationships, fields, metrics, AI context, and extensions are members of its representation. +- **Semantic identity.** A logical View defines fixed SQL computation and fixed output columns. A Metric View defines query-time semantic choices, so the two are distinct kinds of definitions. +- **Namespace.** Logical and Metric Views share the same schema-level View namespace and name rules; same-name objects cannot coexist (see Storage and Connector Behavior for conflict resolution). +- **Representation.** A Metric View contains exactly one `MetricRepresentation`. It cannot contain a SQL representation, and alter requests that change a View between logical and metric semantics are rejected. +- **Lifecycle and columns.** Metric Views reuse View create, list, load, alter, drop, and version operations. Their `columns` collection is always empty because the output schema is selected at query time. + +### Representation Model + +The upstream OSI document places its specification version beside an array of semantic models. The abbreviated form is: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_semantic_model + datasets: + - name: orders + source: sales.mart.orders +``` + +Gravitino maps one `semantic_model` item to `semanticModel`. A three-part OSI dataset source maps to a `NameIdentifier`. View identity and lifecycle remain in the surrounding View object. + +```text +MetricRepresentation + type: "metric" + semanticModel: MetricModel +``` + +The representation has two fields: + +- `type`: The fixed value "metric" classifies the View as a Metric View. +- `semanticModel`: The stable, structured Gravitino model exposed through public APIs. +- A Metric View contains exactly one `MetricRepresentation`. +- Its `columns` array is empty. +- It cannot contain a SQL representation. +- Both `type` and `semanticModel` are required. + +#### MetricModel Schema + +The canonical model follows the [Apache Ossie schema pinned at commit `4eb588b`](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json), whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other fields are required. Names below use OSI wire-format spelling, while language bindings use idiomatic accessor names. + +```text +MetricModel + name: string + description?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] +``` + +- `MetricModel` contains at least one `Dataset`. +- Names in each collection follow the uniqueness and reference rules defined with the nested types below. + +Dataset and field definitions: + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Dataset` names are unique within `MetricModel`. +- `Field` names are unique within each `Dataset`. +- Internal field references resolve within the model. +- Each `source` is a `NameIdentifier` in the form `catalog.schema.name`. Gravitino resolves it in the metalake that contains the Metric View. Cross-catalog references are allowed, while cross-metalake references are not supported. +- `source` does not declare whether the referenced entity is a `Table` or `View`. Validation calls `loadTable` first and, if no Table is found, calls `loadView`; it fails if neither entity exists. +- For `Table` and logical `View` sources, Gravitino validates columns explicitly declared in `primary_key`, `unique_keys`, `from_columns`, and `to_columns` against the source schema. It does not infer source-column references from field or metric expressions. +- Metric View sources validate direct existence only. +- Inline query sources are not supported. For example, instead of storing `SELECT * FROM sales.orders WHERE status = 'active'` directly in `Dataset.source`, create a logical View named `sales.mart.active_orders` with that SQL and set `Dataset.source` to `sales.mart.active_orders`. Raw SQL is not stored directly in the Metric View. +- Catalog unavailability is treated as a retriable validation failure. + +Relationship and metric definitions: + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- `Relationship` and `Metric` names are unique within `MetricModel`. +- Each relationship endpoint references an existing `Dataset`. +- `from_columns` and `to_columns` are non-empty and have equal length. +- Each metric expression satisfies the `Expression` rules below. + +Supporting types: + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | { instructions?: string, synonyms?: string[], + examples?: string[], ... } + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" +``` + +- Each `Expression` contains at least one `DialectExpression`. +- Every dialect entry uses a supported `Dialect`. +- Every dialect entry has a non-empty `expression`. +- `Dimension`, `AIContext`, and `CustomExtension` values satisfy the structures above. + +The required `MetricModel.name` is independent of the enclosing View name. This preserves semantic-model identity across imports and View renames. + +Every supported `custom_extensions` array is retained losslessly. For standardized OSI model objects, fields not defined by the [pinned OSI schema](https://github.com/apache/ossie/blob/4eb588bee8340ab66e985433bb7e8af01688d4bb/core-spec/osi-schema.json#L282-L327) are rejected because the schema sets `additionalProperties` to `false`. Gravitino Metric Views enforce the same restriction. + +**Implementation note:** + +Gravitino pins the exact upstream OSI `0.2.0.dev0` JSON Schema used by the structured model and adds Gravitino-specific projection and semantic rules beyond schema validation. The current contract does not persist an OSI version in each Metric View. If a future OSI version introduces an incompatible interpretation, Gravitino can add explicit version metadata and define compatibility behavior then; existing definitions without that metadata retain the initial semantics. + +- All representation and model checks run on `create` and `alter` before a View or View version is persisted. +- Validation checks direct references only. +- Transitive dependency and cycle correctness are not checked; a cyclic definition may be persisted and later rejected by a downstream consumer. +- Catalog changes do not trigger automatic revalidation. +- Catalog-wide revalidation is excluded because it would require a dependency index and potentially global impact analysis. + +### Usage + +Metric Views reuse the existing View lifecycle. Their columns are always empty, and create, list, load, alter, and drop operations use the existing View APIs. + +#### Supported Alter Operations + +Metric Views support the existing `ViewChange` operations: + +- `rename`: Renames the enclosing View only; `MetricModel.name` is unchanged. The target name must be available in the shared View namespace. +- `setProperty`: Adds or replaces a View-level property. +- `removeProperty`: Removes a View-level property. +- `replaceView`: Atomically replaces the View body. The `columns` must remain empty, exactly one `MetricRepresentation` must remain, and changing between metric and logical semantics is rejected. + +Member-level patch operations are not supported; changes to datasets, relationships, fields, or metrics require replacing the complete `MetricModel`. + +Metric Views do not use `defaultCatalog` or `defaultSchema` because dataset sources use `NameIdentifier`; both values must be `null` in create and `replaceView` requests. + +#### Java API + +The Java API uses immutable builders for the structured definition and the existing ViewCatalog lifecycle methods: + +```java +NameIdentifier ident = NameIdentifier.of("mart", "sales_metrics"); +Dataset orders = + Dataset.builder() + .withName("orders") + .withSource(NameIdentifier.of("sales", "mart", "orders")) + .build(); + +MetricModel model = + MetricModel.builder() + .withName("sales_semantic_model") + .withDatasets(List.of(orders)) + .build(); + +MetricRepresentation representation = + MetricRepresentation.builder() + .withSemanticModel(model) + .build(); + +View created = + catalog.createMetricView( + ident, "Sales metric definitions", List.of(representation), + null, null, Map.of()); +``` + +```java +View loaded = catalog.loadView(ident); +NameIdentifier[] views = catalog.listViews(Namespace.of("mart")); + +MetricModel updatedModel = + MetricModel.builder() + .withName("sales_semantic_model") + .withDescription("Updated sales model") + .withDatasets(List.of(orders)) + .build(); +MetricRepresentation updatedRepresentation = + MetricRepresentation.builder() + .withSemanticModel(updatedModel) + .build(); + +View updated = + catalog.alterView( + ident, + ViewChange.replaceView( + new Column[0], + new Representation[] {updatedRepresentation}, + null, null, "Updated sales metric definitions")); + +boolean dropped = catalog.dropView(ident); +``` + +#### Python API + +The Python API exposes the same structured model and View lifecycle: + +```python +ident = NameIdentifier.of("mart", "sales_metrics") +orders = Dataset("orders", NameIdentifier.of("sales", "mart", "orders")) +model = MetricModel("sales_semantic_model", [orders]) +representation = MetricRepresentation(model) + +created = catalog.create_metric_view( + ident, + [representation], + comment="Sales metric definitions", +) + +loaded = catalog.load_view(ident) +views = catalog.list_views(Namespace.of("mart")) + +updated_model = MetricModel( + "sales_semantic_model", + [orders], + _description="Updated sales model", +) +updated_representation = MetricRepresentation(updated_model) +updated = catalog.alter_view( + ident, + ViewChange.replace_view( + columns=[], + representations=[updated_representation], + comment="Updated sales metric definitions", + ), +) + +dropped = catalog.drop_view(ident) +``` + +#### REST API + +REST uses the existing View resources. Create supplies an empty columns array and one Metric representation: + +```http +POST /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views +{ + "name": "sales_metrics", + "comment": "Sales metric definitions", + "columns": [], + "representations": [ + { + "type": "metric", + "semanticModel": { + "name": "sales_semantic_model", + "datasets": [ + { "name": "orders", "source": { "namespace": ["sales", "mart"], "name": "orders" } } + ] + } + } + ] +} +``` + +List, load, alter, and drop use the same resource: + +```http +GET /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views +GET /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics + +PUT /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics +{ + "updates": [ + { + "@type": "replaceView", + "columns": [], + "representations": [ + { + "type": "metric", + "semanticModel": { + "name": "sales_semantic_model", + "description": "Updated sales model", + "datasets": [ + { "name": "orders", "source": { "namespace": ["sales", "mart"], "name": "orders" } } + ] + } + } + ], + "comment": "Updated sales metric definitions" + } + ] +} + +DELETE /metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/views/sales_metrics +``` + +### Storage and Connector Behavior + +- **Source of truth.** Metric Views are stored only in the Gravitino EntityStore. Logical Views remain stored by their underlying catalogs. +- **Listing.** The server merges authorized catalog-backed logical Views with authorized Gravitino-managed Metric Views into the existing View listing. +- **Connector capability.** A connector that does not support Metric Views filters them from `listViews` and returns an explicit unsupported-Metric-View error for a direct `loadView`. Generic REST and Java View APIs continue to expose them. Review Comment: The server performs the listing, so this needs a mechanism: a client capability header, a query parameter, or connector-side filtering after fetch. Without it, Trino and Spark see View objects with empty columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
