FANNG1 commented on code in PR #11280: URL: https://github.com/apache/gravitino/pull/11280#discussion_r3332678367
########## design-docs/unified-engine-access.md: ########## @@ -0,0 +1,361 @@ +--- +title: "Design: Engine-native Catalog Access Mode for Gravitino Connectors" +slug: /unified-engine-access +keywords: + - unified engine access + - spark connector + - lance + - iceberg + - engine-access-mode + - native catalog +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino can manage multiple lakehouse catalogs and lets compute engines access the underlying table +data in various ways. For example, Spark can access some catalogs through the Gravitino Spark +connector, or access Iceberg/Lance tables directly through the Iceberg REST catalog or Lance REST +Namespace. + +In mixed Iceberg-and-Lance query scenarios, the Spark side still requires users to maintain several +sets of configuration manually: + +```text +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test + +spark.sql.catalog.iceberg_rest=org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.iceberg_rest.type=rest +spark.sql.catalog.iceberg_rest.uri=http://127.0.0.1:9001/iceberg/ + +spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog +spark.sql.catalog.lance.impl=rest +spark.sql.catalog.lance.uri=http://127.0.0.1:9101/lance +spark.sql.catalog.lance.parent=lance_catalog +``` + +This creates several problems: + +1. Users must understand Gravitino catalogs, the Iceberg REST catalog, the Lance REST Namespace, + and the catalog configuration of each engine simultaneously. +2. Every time a Gravitino catalog is added or modified, the configuration on the Spark, Flink, + Trino, and other engine sides must be updated in sync. +3. Each engine independently duplicates the translation work from catalog properties to engine + catalog configuration. +4. The value of Gravitino as a unified metadata entry point is diminished. + +This design takes a lightweight approach: no new discovery REST API is introduced; the engine side +declares the access strategy per catalog provider, and native connector configuration is +automatically derived from the catalog's existing properties by each engine connector. + +## Goals + +1. Users only need to configure the Gravitino server address and metalake. +2. Spark can automatically discover and register Iceberg catalogs and Lance native catalogs. +3. The same semantics can be extended to Flink, Trino, Doris, Daft, and other engines. +4. Support controlling the access mode per catalog provider: use the Gravitino connector/API or + the engine's native connector. +5. The access mode is configured on the engine side per catalog provider; native connector + configuration reuses the catalog's existing properties. +6. In the first phase, no new discovery REST API is introduced; the existing + `listCatalogsInfo()` / `loadCatalog()` calls are reused. + +## Non-Goals + +1. In the first phase, full Lance support across all engines is not required simultaneously. + +## Core Design + +A new engine-side, provider-level access mode configuration is introduced: + +```text +spark.sql.gravitino.<provider>.engine-access-mode = auto | gravitino | native +``` + +The semantics are: + +| Value | Meaning | +|-------------|---------| +| `auto` | Default. The Gravitino connector automatically selects the access method based on whether the current engine has a Gravitino connector for the given provider. If a Gravitino connector exists for the provider, it falls back to `gravitino`; otherwise it falls back to `native`. | +| `gravitino` | Force the use of the Gravitino connector/API. | +| `native` | Force the use of the engine's native connector/catalog, for example Spark Iceberg `SparkCatalog`, Spark Lance `LanceNamespaceSparkCatalog`, Trino/Doris native Iceberg catalog, or Lance REST Namespace. | + +### Access Mode Selection + +The engine connector reads the corresponding configuration based on the catalog provider, for example: + +```text +spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native +spark.sql.gravitino.lakehouse-lance.engine-access-mode=native +``` + +If no provider-level configuration is set, `auto` is used. + +| Catalog | `auto` rule | +|---------------|-------------| +| Iceberg | Defaults to `gravitino`, preserving the existing Gravitino Spark connector behavior. Switches to an Iceberg native catalog only when `spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native` is set explicitly. | +| Lance | Defaults to `native`, because there is currently no Lance Gravitino connector. If the conversion to a Lance native catalog fails, an `UnsupportedException` is thrown immediately. | +| Other catalogs | Preserves the existing Gravitino connector behavior. | + +No new native-specific catalog properties are added. The engine connector derives the native +configuration from the existing `provider` and catalog properties, for example: Iceberg uses +`catalog-backend`, `uri`, `warehouse`, and `data-access`; Lance uses `namespace-backend`, `uri`, +and `location`. + +## Catalog Examples + +### Iceberg + +```text +name = iceberg +type = RELATIONAL +provider = lakehouse-iceberg + +catalog-backend = rest +uri = http://127.0.0.1:9001/iceberg/ +warehouse = s3://contacts/raw/iceberg +data-access = vended-credentials +``` + +Notes: + +1. `catalog-backend=rest` indicates the Iceberg catalog backend is an Iceberg REST catalog. +2. `uri` is the Iceberg REST endpoint; it is also used by the Spark connector to generate the + Iceberg Spark catalog `uri`. +3. `data-access=vended-credentials` carries the existing Iceberg REST semantics and is used by the + engine connector to automatically inject the Iceberg REST credential delegation header. + +### Lance + +Whether to introduce a new dedicated `lakehouse-lance` provider or continue expressing Lance +catalogs with the existing `lakehouse-generic + format=lance` is an open question. This document +favors a dedicated `lakehouse-lance` provider because the semantics are clearer and it provides a +better foundation for Lance-specific capability declarations, property validation, and +engine-native configuration translation. + +Example: + +```text +name = lance_catalog +type = RELATIONAL +provider = lakehouse-lance + +namespace-backend = rest +uri = http://127.0.0.1:9101/lance +location = s3://contacts/raw/lance +``` + +Notes: + +1. `namespace-backend=rest` indicates the Lance catalog uses the Lance REST Namespace protocol. +2. `uri` is the Lance REST endpoint; it is also used by the Spark connector to generate the Lance + Spark catalog `uri`. +3. The Lance Spark connector `parent` parameter defaults to the Gravitino catalog name. + +:::note +The use of `type = RELATIONAL` for Lance catalogs is an open question. Lance tables support +columnar/vector storage semantics, which may not cover all relational SQL operations. Community +input is welcome on whether a new catalog type (e.g. `LAKEHOUSE`) or a more relaxed interpretation +of `RELATIONAL` is appropriate here. +::: + +If compatibility with existing implementations is required in the first phase, +`lakehouse-generic` can be used as a compatibility path with the following convention: + +```text +provider = lakehouse-generic +format = lance +namespace-backend = rest +``` + +In the first phase, `lakehouse-lance` is the recommended provider. `lakehouse-generic + format=lance` +is only a transitional path for compatibility with existing catalogs and is not the long-term +recommended modeling approach. + +## Spark Design + +Users only configure: + +```text +spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test +``` + +Optional overrides: + +```text +spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native +spark.sql.gravitino.lakehouse-lance.engine-access-mode=native +spark.sql.gravitino.enableIcebergSupport=true +spark.sql.gravitino.enableLanceSupport=true +``` + +The Spark connector then automatically registers Iceberg and Lance catalogs based on the switches. +Under `auto`, Iceberg uses the Gravitino catalog by default; Lance uses the native catalog. + +`spark.sql.gravitino.enableLanceSupport` defaults to `false` to avoid loading Lance catalogs or +extensions when the user has not explicitly included the Lance Spark connector dependency. Review Comment: Added catalog-level access mode overrides for this case. The design now supports both: - provider default: `spark.sql.gravitino.<provider>.engine-access-mode` - per-catalog override: `spark.sql.gravitino.catalog.<catalog>.engine-access-mode` Resolution order is catalog override > provider default > `auto`, so multiple Iceberg or Lance catalogs can use different access modes in the same Spark session. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
