mchades commented on code in PR #14038: URL: https://github.com/apache/gravitino/pull/14038#discussion_r4338949030
########## design-docs/gravitino-schema-metrics-discovery-design.md: ########## @@ -0,0 +1,391 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Schema-level Semantic Model Metrics Discovery in Gravitino + +## Background + +Metrics in Gravitino are defined within Semantic Models. To browse or select metrics across a +schema, clients currently list the Semantic Models, load each model, and extract its metrics. + +This requires a separate request to load each model and returns the full definitions, including +datasets and relationships that metric discovery does not need. Clients must also implement the +collection logic themselves. The number of requests and volume of metadata transferred increase +as more models are added or their definitions grow. + +## Goals + +1. **Discovery.** Provide schema-level listing of metric identifiers and complete definitions through + REST, Java, Python, and MCP, without requiring clients to load each Semantic Model separately. +2. **Identity.** Keep internal identities stable for metrics retained during Semantic Model updates. +3. **Consistency.** Keep discovered metric definitions consistent with the current committed + definition of each Semantic Model. +4. **Visibility.** Apply the same visibility rules as the existing Semantic Model listing. + +## Non-Goals + +1. **Independent metric management.** Standalone APIs for creating, updating, deleting, or renaming + metrics; metric-level ownership and privileges. +2. **Public identity and history.** Exposing internal metric IDs or revision identifiers, or querying + historical metric definitions. + +## Proposal + +### API Contract + +#### Java API + +The `SemanticModelCatalog` interface adds separate methods for identifiers and complete definitions: + +```java +NameIdentifier[] listMetrics(Namespace namespace) throws NoSuchSchemaException; + +SemanticModelMetric[] listMetricInfos(Namespace namespace) throws NoSuchSchemaException; +``` + +`listMetrics` returns one identifier per metric. A full identifier has a four-level namespace +`[metalake, catalog, schema, semanticModel]` and the metric name as its `name`. Including the containing +model distinguishes metrics with the same name in different models. Catalog-scoped clients follow +the existing convention: callers pass `Namespace.of(schema)`, and returned identifiers contain the +namespace `[schema, semanticModel]` and the metric name. REST uses full identifiers. + +`listMetricInfos` returns one `SemanticModelMetric` per metric: + +```java +/** A metric's namespace and complete definition returned by discovery. */ +public interface SemanticModelMetric { + + /** + * Returns the metric namespace, whose last level is the containing Semantic Model. + * + * @return The metric namespace. + */ + Namespace namespace(); + + /** + * Returns the complete metric definition. + * + * @return The metric definition. + */ + Metric metric(); +} +``` + +`namespace()` uses the same full or catalog-relative namespace as the identifiers from `listMetrics`. +In this response type, the metric name is stored only in the existing `Metric` definition. +Callers can reconstruct its identifier with `NameIdentifier.of(info.namespace(), info.metric().name())`. +`SemanticModelMetricDTO` contains a `namespace` array and an unchanged `MetricDTO` in `metric`. + +#### REST API + +The new route sits alongside `semantic-models`, avoiding a conflict with a model named `metrics`: + +```http +GET /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/metrics +``` + +The `details` query parameter defaults to `false`. With `details=false`, the endpoint implements +`listMetrics` and returns the existing `EntityListResponse`: + +```json +{ + "code": 0, + "identifiers": [ + { + "namespace": ["example", "sales", "mart", "sales_model"], + "name": "total_revenue" + } + ] +} +``` + +With `details=true`, it implements `listMetricInfos` and returns a `MetricInfoListResponse` containing +a `metrics` array of `SemanticModelMetricDTO` objects: + +```json +{ + "code": 0, + "metrics": [ + { + "namespace": ["example", "sales", "mart", "sales_model"], + "metric": { + "name": "total_revenue", + "expression": { + "dialects": [ + { + "dialect": "ANSI_SQL", + "expression": "SUM(orders.amount)" + } + ] + } + } + } + ] +} +``` + +Both modes sort results by structured model identifier and then case-sensitive metric name. Models +without metrics contribute no entries. Empty results use `identifiers: []` or `metrics: []`, according +to `details`. Preserve existing errors for missing schemas and unsupported catalogs; never truncate Review Comment: Pagination is outside the scope of this proposal. The existing Semantic Model and Function listing APIs are also unpaginated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
