FANNG1 opened a new pull request, #11838:
URL: https://github.com/apache/gravitino/pull/11838
### What changes were proposed in this pull request?
When `spark.sql.gravitino.iceberg.enableRestAccess=true` is set, the
Gravitino Spark plugin automatically registers each `lakehouse-iceberg`
catalog as a native Iceberg REST Spark catalog
(`org.apache.iceberg.spark.SparkCatalog` with `type=rest`) instead of the
Gravitino wrapper catalog, removing the need for users to hand-write
per-catalog
Spark configuration.
Key changes:
- **`GravitinoSparkConfig`**: Add
`spark.sql.gravitino.iceberg.enableRestAccess`
and `spark.sql.gravitino.iceberg.restUri` configuration keys.
- **`IcebergRestCatalogRegistrar`** (new): Encapsulates REST URI resolution
(explicit override or inferred from `spark.sql.gravitino.uri`) and
per-catalog
Spark config injection, including credential-vending header and static
S3/OSS/ADLS property mapping.
- **`GravitinoDriverPlugin`**: Routes `lakehouse-iceberg` catalogs through
`IcebergRestCatalogRegistrar` when `enableRestAccess=true`; retains
existing
Gravitino wrapper behavior otherwise.
- **`SparkEnvIT`**: Expose `initIcebergRestServiceEnv()` as `protected` and
add
`getExtraSparkConfigs()` hook for subclass customization.
- **Integration tests** (`SparkIcebergCatalogAutoRestAccessIT` + Spark
3.3/3.4/3.5
runners): Use `dynamic-config-provider` so the Gravitino Iceberg REST
server
loads catalog config from Gravitino at request time. Inherits full DDL/DML
test suite from `SparkIcebergCatalogIT`.
### Why are the changes needed?
Spark users who want to access Iceberg via the REST protocol currently must
hand-write `spark.sql.catalog.*` configuration for every catalog even though
Gravitino already manages the catalog inventory. This change allows the Spark
connector to derive that configuration automatically, keeping Gravitino as
the
single source of truth.
Fix: #11280
### Does this PR introduce _any_ user-facing change?
Yes — new configuration keys:
| Key | Default | Description |
|-----|---------|-------------|
| `spark.sql.gravitino.iceberg.enableRestAccess` | `false` | Register
Iceberg catalogs as native REST catalogs instead of Gravitino wrapper |
| `spark.sql.gravitino.iceberg.restUri` | inferred from
`spark.sql.gravitino.uri` | Explicit URI for the Gravitino Iceberg REST service
|
When `enableRestAccess=true`, `lakehouse-iceberg` catalogs are no longer
registered as `GravitinoIcebergCatalog`; they are registered as
`org.apache.iceberg.spark.SparkCatalog` with `type=rest`. Existing behavior
(controlled by `enableIcebergSupport`) is unchanged when `enableRestAccess`
is
not set.
### How was this patch tested?
- Unit tests: `TestIcebergRestCatalogRegistrar` covers URI inference,
duplicate
registration guard, credential-vending delegation header, static S3
credential
prefix mapping, and storage property forwarding.
- Integration tests: `SparkIcebergCatalogAutoRestAccessIT{33,34,35}` run the
full `SparkIcebergCatalogIT` DDL/DML suite against a real Gravitino server
+
Docker Hive, with the Iceberg REST service configured via
`dynamic-config-provider`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
