yuqi1129 commented on code in PR #11280: URL: https://github.com/apache/gravitino/pull/11280#discussion_r3334080142
########## design-docs/unified-engine-access.md: ########## @@ -0,0 +1,401 @@ +--- +title: "Design: Engine-native Catalog Access Mode for Gravitino Connectors" +slug: /unified-engine-access +keywords: + - unified engine access + - spark connector + - lance + - iceberg + - engine-access-mode + - native catalog +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino can manage multiple lakehouse catalogs and lets compute engines access the underlying table +data in various ways. For example, Spark can access some catalogs through the Gravitino Spark +connector, or access Iceberg/Lance tables directly through the Iceberg REST catalog or Lance REST +Namespace. + +In mixed Iceberg-and-Lance query scenarios, the Spark side still requires users to maintain several +sets of configuration manually: + +```text +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test + +spark.sql.catalog.iceberg_rest=org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.iceberg_rest.type=rest +spark.sql.catalog.iceberg_rest.uri=http://127.0.0.1:9001/iceberg/ + +spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog +spark.sql.catalog.lance.impl=rest +spark.sql.catalog.lance.uri=http://127.0.0.1:9101/lance +spark.sql.catalog.lance.parent=lance_catalog +``` + +This creates several problems: + +1. Users must understand Gravitino catalogs, the Iceberg REST catalog, the Lance REST Namespace, + and the catalog configuration of each engine simultaneously. +2. Every time a Gravitino catalog is added or modified, the configuration on the Spark, Flink, + Trino, and other engine sides must be updated in sync. +3. Each engine independently duplicates the translation work from catalog properties to engine + catalog configuration. +4. The value of Gravitino as a unified metadata entry point is diminished. + +This design takes a lightweight approach: no new discovery REST API is introduced; the engine side +declares the access strategy per catalog provider, and native connector configuration is +automatically derived from the catalog's existing properties by each engine connector. + +## Goals + +1. Users only need to configure the Gravitino server address and metalake. +2. Spark can automatically discover and register Iceberg catalogs and Lance native catalogs. +3. The same semantics can be extended to Flink, Trino, Doris, Daft, and other engines. +4. Support controlling the access mode per catalog provider: use the Gravitino connector/API or + the engine's native connector. +5. The access mode is configured on the engine side per catalog provider; native connector + configuration reuses the catalog's existing properties. +6. In the first phase, no new discovery REST API is introduced; the existing + `listCatalogsInfo()` / `loadCatalog()` calls are reused. + +## Non-Goals + +1. In the first phase, full Lance support across all engines is not required simultaneously. + +## Core Design + +A new engine-side, provider-level access mode configuration is introduced: + +```text +spark.sql.gravitino.<provider>.engine-access-mode = auto | gravitino | native Review Comment: You may clarify what `provider` means here? According to the document, it may refer to catalog providers in the Gravitino catalog. Moreover, you may need to add a document about what kinds of providers it supports. ########## design-docs/unified-engine-access.md: ########## @@ -0,0 +1,401 @@ +--- +title: "Design: Engine-native Catalog Access Mode for Gravitino Connectors" +slug: /unified-engine-access +keywords: + - unified engine access + - spark connector + - lance + - iceberg + - engine-access-mode + - native catalog +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino can manage multiple lakehouse catalogs and lets compute engines access the underlying table +data in various ways. For example, Spark can access some catalogs through the Gravitino Spark +connector, or access Iceberg/Lance tables directly through the Iceberg REST catalog or Lance REST +Namespace. + +In mixed Iceberg-and-Lance query scenarios, the Spark side still requires users to maintain several +sets of configuration manually: + +```text +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test + +spark.sql.catalog.iceberg_rest=org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.iceberg_rest.type=rest +spark.sql.catalog.iceberg_rest.uri=http://127.0.0.1:9001/iceberg/ + +spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog +spark.sql.catalog.lance.impl=rest +spark.sql.catalog.lance.uri=http://127.0.0.1:9101/lance +spark.sql.catalog.lance.parent=lance_catalog +``` + +This creates several problems: + +1. Users must understand Gravitino catalogs, the Iceberg REST catalog, the Lance REST Namespace, + and the catalog configuration of each engine simultaneously. +2. Every time a Gravitino catalog is added or modified, the configuration on the Spark, Flink, + Trino, and other engine sides must be updated in sync. +3. Each engine independently duplicates the translation work from catalog properties to engine + catalog configuration. +4. The value of Gravitino as a unified metadata entry point is diminished. + +This design takes a lightweight approach: no new discovery REST API is introduced; the engine side +declares the access strategy per catalog provider, and native connector configuration is +automatically derived from the catalog's existing properties by each engine connector. + +## Goals + +1. Users only need to configure the Gravitino server address and metalake. +2. Spark can automatically discover and register Iceberg catalogs and Lance native catalogs. +3. The same semantics can be extended to Flink, Trino, Doris, Daft, and other engines. +4. Support controlling the access mode per catalog provider: use the Gravitino connector/API or + the engine's native connector. +5. The access mode is configured on the engine side per catalog provider; native connector + configuration reuses the catalog's existing properties. +6. In the first phase, no new discovery REST API is introduced; the existing + `listCatalogsInfo()` / `loadCatalog()` calls are reused. + +## Non-Goals + +1. In the first phase, full Lance support across all engines is not required simultaneously. + +## Core Design + +A new engine-side, provider-level access mode configuration is introduced: + +```text +spark.sql.gravitino.<provider>.engine-access-mode = auto | gravitino | native +``` + +The semantics are: + +| Value | Meaning | +|-------------|---------| +| `auto` | Default. The Gravitino connector automatically selects the access method based on whether the current engine has a Gravitino connector for the given provider. If a Gravitino connector exists for the provider, it falls back to `gravitino`; otherwise it falls back to `native`. | +| `gravitino` | Force the use of the Gravitino connector/API. | +| `native` | Force the use of the engine's native connector/catalog, for example Spark Iceberg `SparkCatalog`, Spark Lance `LanceNamespaceSparkCatalog`, Trino/Doris native Iceberg catalog, or Lance REST Namespace. | + +### Access Mode Selection + +The engine connector reads the corresponding configuration based on the catalog provider, for +example: + +```text +spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native +spark.sql.gravitino.lakehouse-generic.engine-access-mode=native +``` + +If no provider-level configuration is set, `auto` is used. + +| Catalog | `auto` rule | +|---------------|-------------| +| Iceberg | Defaults to `gravitino`, preserving the existing Gravitino Spark connector behavior. Switches to an Iceberg native catalog only when `spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native` is set explicitly. | +| Lance | Defaults to `native`, because there is currently no Lance Gravitino connector. If the conversion to a Lance native catalog fails, an `UnsupportedException` is thrown immediately. | +| Other catalogs | Preserves the existing Gravitino connector behavior. | + +No new native-specific catalog properties are added. The engine connector derives the native +configuration from the existing `provider` and catalog properties, for example: Iceberg uses +`catalog-backend`, `uri`, `warehouse`, and `data-access`; v1 Lance uses `format`, +`namespace-backend`, `uri`, and `location`. + +## Catalog Examples + +### Iceberg + +```text +name = iceberg +type = RELATIONAL +provider = lakehouse-iceberg + +catalog-backend = rest +uri = http://127.0.0.1:9001/iceberg/ +warehouse = iceberg +data-access = vended-credentials Review Comment: Are those configurations for all the catalogs with type `Iceberg`? How can we define catalog-specific configurations? ########## design-docs/unified-engine-access.md: ########## @@ -0,0 +1,401 @@ +--- +title: "Design: Engine-native Catalog Access Mode for Gravitino Connectors" +slug: /unified-engine-access +keywords: + - unified engine access + - spark connector + - lance + - iceberg + - engine-access-mode + - native catalog +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino can manage multiple lakehouse catalogs and lets compute engines access the underlying table +data in various ways. For example, Spark can access some catalogs through the Gravitino Spark +connector, or access Iceberg/Lance tables directly through the Iceberg REST catalog or Lance REST +Namespace. + +In mixed Iceberg-and-Lance query scenarios, the Spark side still requires users to maintain several +sets of configuration manually: + +```text +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test + +spark.sql.catalog.iceberg_rest=org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.iceberg_rest.type=rest +spark.sql.catalog.iceberg_rest.uri=http://127.0.0.1:9001/iceberg/ + +spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog +spark.sql.catalog.lance.impl=rest +spark.sql.catalog.lance.uri=http://127.0.0.1:9101/lance +spark.sql.catalog.lance.parent=lance_catalog +``` + +This creates several problems: + +1. Users must understand Gravitino catalogs, the Iceberg REST catalog, the Lance REST Namespace, + and the catalog configuration of each engine simultaneously. +2. Every time a Gravitino catalog is added or modified, the configuration on the Spark, Flink, + Trino, and other engine sides must be updated in sync. +3. Each engine independently duplicates the translation work from catalog properties to engine + catalog configuration. +4. The value of Gravitino as a unified metadata entry point is diminished. + +This design takes a lightweight approach: no new discovery REST API is introduced; the engine side +declares the access strategy per catalog provider, and native connector configuration is +automatically derived from the catalog's existing properties by each engine connector. + +## Goals + +1. Users only need to configure the Gravitino server address and metalake. +2. Spark can automatically discover and register Iceberg catalogs and Lance native catalogs. +3. The same semantics can be extended to Flink, Trino, Doris, Daft, and other engines. +4. Support controlling the access mode per catalog provider: use the Gravitino connector/API or + the engine's native connector. +5. The access mode is configured on the engine side per catalog provider; native connector + configuration reuses the catalog's existing properties. +6. In the first phase, no new discovery REST API is introduced; the existing + `listCatalogsInfo()` / `loadCatalog()` calls are reused. + +## Non-Goals + +1. In the first phase, full Lance support across all engines is not required simultaneously. + +## Core Design + +A new engine-side, provider-level access mode configuration is introduced: + +```text +spark.sql.gravitino.<provider>.engine-access-mode = auto | gravitino | native +``` + +The semantics are: + +| Value | Meaning | +|-------------|---------| +| `auto` | Default. The Gravitino connector automatically selects the access method based on whether the current engine has a Gravitino connector for the given provider. If a Gravitino connector exists for the provider, it falls back to `gravitino`; otherwise it falls back to `native`. | +| `gravitino` | Force the use of the Gravitino connector/API. | +| `native` | Force the use of the engine's native connector/catalog, for example Spark Iceberg `SparkCatalog`, Spark Lance `LanceNamespaceSparkCatalog`, Trino/Doris native Iceberg catalog, or Lance REST Namespace. | + +### Access Mode Selection + +The engine connector reads the corresponding configuration based on the catalog provider, for +example: + +```text +spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native +spark.sql.gravitino.lakehouse-generic.engine-access-mode=native +``` + +If no provider-level configuration is set, `auto` is used. + +| Catalog | `auto` rule | +|---------------|-------------| +| Iceberg | Defaults to `gravitino`, preserving the existing Gravitino Spark connector behavior. Switches to an Iceberg native catalog only when `spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native` is set explicitly. | +| Lance | Defaults to `native`, because there is currently no Lance Gravitino connector. If the conversion to a Lance native catalog fails, an `UnsupportedException` is thrown immediately. | +| Other catalogs | Preserves the existing Gravitino connector behavior. | + +No new native-specific catalog properties are added. The engine connector derives the native +configuration from the existing `provider` and catalog properties, for example: Iceberg uses +`catalog-backend`, `uri`, `warehouse`, and `data-access`; v1 Lance uses `format`, +`namespace-backend`, `uri`, and `location`. + +## Catalog Examples + +### Iceberg + +```text +name = iceberg +type = RELATIONAL +provider = lakehouse-iceberg + +catalog-backend = rest +uri = http://127.0.0.1:9001/iceberg/ +warehouse = iceberg +data-access = vended-credentials +``` + +Notes: + +1. `catalog-backend=rest` indicates the Iceberg catalog backend is an Iceberg REST catalog. +2. `uri` is the Iceberg REST endpoint; it is also used by the Spark connector to generate the + Iceberg Spark catalog `uri`. +3. `warehouse` selects the target catalog inside the Iceberg REST server when the REST server + supports multiple catalogs. If it is not set, the Spark connector uses the Gravitino catalog + name as the REST catalog selector. +4. `data-access=vended-credentials` carries the existing Iceberg REST semantics and is used by the + engine connector to automatically inject the Iceberg REST credential delegation header. + +### Lance + +The first phase expresses Lance catalogs with the existing `lakehouse-generic + format=lance` +convention. A dedicated `lakehouse-lance` provider can be discussed later, but it is not required +for the v1 Spark-native registration path. + +Example: + +```text +name = lance_catalog +type = RELATIONAL +provider = lakehouse-generic + +format = lance +namespace-backend = rest +uri = http://127.0.0.1:9101/lance +location = s3://contacts/raw/lance +``` + +Notes: + +1. `format=lance` identifies the generic catalog as a Lance catalog for v1 Spark-native + registration. +2. `namespace-backend=rest` indicates the Lance catalog uses the Lance REST Namespace protocol. +3. `uri` is the Lance REST endpoint; it is also used by the Spark connector to generate the Lance + Spark catalog `uri`. +4. The Lance Spark connector `parent` parameter defaults to the Gravitino catalog name. + +:::note +The use of `type = RELATIONAL` for Lance catalogs is an open question. Lance tables support +columnar/vector storage semantics, which may not cover all relational SQL operations. Community +input is welcome on whether a new catalog type (e.g. `LAKEHOUSE`) or a more relaxed interpretation +of `RELATIONAL` is appropriate here. +::: + +Only `format=lance` and `namespace-backend=rest` participate in v1 Spark-native Lance registration. +Other generic catalog formats are ignored by Lance registration. + +## Spark Design + +Users only configure: + +```text +spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin +spark.sql.gravitino.uri=http://127.0.0.1:8090 +spark.sql.gravitino.metalake=test +``` + +Optional overrides: + +```text +spark.sql.gravitino.lakehouse-iceberg.engine-access-mode=native +spark.sql.gravitino.lakehouse-generic.engine-access-mode=native +spark.sql.gravitino.enableIcebergSupport=true Review Comment: Why do we need to add these extra configurations? What's the problem if we set it to constant value `true`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
