FANNG1 commented on code in PR #11280: URL: https://github.com/apache/gravitino/pull/11280#discussion_r3654781211
########## design-docs/spark-rest-catalog-registration.md: ########## @@ -0,0 +1,338 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design: Spark Iceberg REST Catalog Automatic Registration for Apache Gravitino + +--- + +## Background + +Connecting Spark to Gravitino takes little configuration: + +```text +spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test +``` + +But accessing Iceberg tables through the Gravitino Iceberg REST server still requires hand-written +configuration per catalog, duplicating what the REST server already manages and needing an edit +whenever catalogs are added or removed: + +```text +spark.sql.catalog.iceberg_prod=org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.iceberg_prod.type=rest +spark.sql.catalog.iceberg_prod.uri=http://127.0.0.1:9001/iceberg/ +spark.sql.catalog.iceberg_prod.warehouse=iceberg_prod +``` + +--- + +## Goals + +1. **Automatic Iceberg registration**: A Spark session configured with only the new plugin and the + Iceberg REST server URI registers one Spark Iceberg REST catalog per catalog served by that + server, with no per-catalog configuration. +2. **Server-authoritative catalog list**: The REST server tells Spark which catalogs it serves, so + Spark never guesses catalog names. +3. **User configuration always wins**: A catalog the user configured by hand is never touched, and + this is enforced by mechanism rather than by convention. +4. **Zero impact when disabled**: Users who do not add the new plugin see no behavior change. + +--- + +## Non-Goals + +1. **Lance registration in V1**: The design covers Lance (see [Lance support](#lance-support)), but + V1 implements only Iceberg; Lance registration ships later, once its authorization gap is closed. +2. **Engines beyond Spark**: Flink and Trino may reuse the listing endpoint later. +3. **Iceberg REST specification changes**: The listing endpoint is a Gravitino-private extension. + +--- + +## Proposal + +Two new pieces — a **catalog-listing endpoint** on the Iceberg REST server and a Spark plugin that +consumes it — plus one **ordering rule** that makes the interaction with `GravitinoSparkPlugin` +deterministic. + +### Catalog-listing endpoint + +#### GET `{iceberg-rest-base}/gravitino/v1/catalogs` + +Placed outside the Iceberg REST specification's `/v1/` namespace (default deployment: +`http://<host>:9001/iceberg/gravitino/v1/catalogs`) to mark it as a Gravitino-private extension. + +**Request:** No parameters. + +**Response:** `200 OK` + +```json +{ + "catalogs": [ + { "name": "iceberg_prod", "properties": {} }, + { "name": "iceberg_audit", "properties": {} } + ] +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `catalogs[].name` | string | Catalog name as accepted by this server's `warehouse` parameter | +| `catalogs[].properties` | map | Reserved for non-sensitive, client-relevant metadata; V1 defines no keys. Per-catalog client configuration already arrives via `GET /v1/config?warehouse=<name>`, so this endpoint only enumerates names | + +### GravitinoIcebergRestSparkPlugin Review Comment: Good call — done. The design now uses a single format-agnostic `GravitinoLakehouseRESTDiscoveryPlugin` instead of one plugin per format. Each lakehouse format is a **provider** (Iceberg in V1, Lance later) with its own engine runtime and `<format>Rest.*` config. A provider activates only when its `uri` is set — the URI is the switch, so there's no `enable*` flag — and dependency isolation is kept by loading providers from the classpath (fail-fast if a `uri` is set but its provider jar is missing). So `spark.plugins` lists one plugin and there's a single ordering rule. The registration policy is now a single plugin-level setting shared across providers, with a `format` argument on its methods so one policy can still apply format-specific rules. ########## design-docs/spark-rest-catalog-registration.md: ########## @@ -0,0 +1,338 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design: Spark Iceberg REST Catalog Automatic Registration for Apache Gravitino + +--- + +## Background + +Connecting Spark to Gravitino takes little configuration: + +```text +spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test +``` + +But accessing Iceberg tables through the Gravitino Iceberg REST server still requires hand-written +configuration per catalog, duplicating what the REST server already manages and needing an edit +whenever catalogs are added or removed: + +```text +spark.sql.catalog.iceberg_prod=org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.iceberg_prod.type=rest +spark.sql.catalog.iceberg_prod.uri=http://127.0.0.1:9001/iceberg/ +spark.sql.catalog.iceberg_prod.warehouse=iceberg_prod +``` + +--- + +## Goals + +1. **Automatic Iceberg registration**: A Spark session configured with only the new plugin and the + Iceberg REST server URI registers one Spark Iceberg REST catalog per catalog served by that + server, with no per-catalog configuration. +2. **Server-authoritative catalog list**: The REST server tells Spark which catalogs it serves, so + Spark never guesses catalog names. +3. **User configuration always wins**: A catalog the user configured by hand is never touched, and + this is enforced by mechanism rather than by convention. +4. **Zero impact when disabled**: Users who do not add the new plugin see no behavior change. + +--- + +## Non-Goals + +1. **Lance registration in V1**: The design covers Lance (see [Lance support](#lance-support)), but + V1 implements only Iceberg; Lance registration ships later, once its authorization gap is closed. +2. **Engines beyond Spark**: Flink and Trino may reuse the listing endpoint later. +3. **Iceberg REST specification changes**: The listing endpoint is a Gravitino-private extension. + +--- + +## Proposal + +Two new pieces — a **catalog-listing endpoint** on the Iceberg REST server and a Spark plugin that +consumes it — plus one **ordering rule** that makes the interaction with `GravitinoSparkPlugin` +deterministic. + +### Catalog-listing endpoint + +#### GET `{iceberg-rest-base}/gravitino/v1/catalogs` + +Placed outside the Iceberg REST specification's `/v1/` namespace (default deployment: +`http://<host>:9001/iceberg/gravitino/v1/catalogs`) to mark it as a Gravitino-private extension. + +**Request:** No parameters. + +**Response:** `200 OK` + +```json +{ + "catalogs": [ + { "name": "iceberg_prod", "properties": {} }, + { "name": "iceberg_audit", "properties": {} } + ] +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `catalogs[].name` | string | Catalog name as accepted by this server's `warehouse` parameter | +| `catalogs[].properties` | map | Reserved for non-sensitive, client-relevant metadata; V1 defines no keys. Per-catalog client configuration already arrives via `GET /v1/config?warehouse=<name>`, so this endpoint only enumerates names | Review Comment: It was only a placeholder for future extension, not used in V1, so I removed it. The response returns names only; per-catalog config already comes from `GET /v1/config?warehouse=<name>`, and since JSON extends compatibly a field can be added later if an engine ever needs it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
