FANNG1 opened a new issue, #12525:
URL: https://github.com/apache/gravitino/issues/12525
### Describe the subtask
Add the format-agnostic discovery plugin in `spark-connector/spark-common`,
and ship the Lance provider with it — Lance discovery needs no new server API,
so it can land independently of the Iceberg listing endpoint.
**Plugin core**
- `GravitinoLakehouseRESTDiscoveryPlugin` fails fast if it is listed after
`GravitinoSparkPlugin` in `spark.plugins`. Initializing first is what lets it
treat every `spark.sql.catalog.*` entry it sees as user-written, with no
markers and no state shared between the plugins.
- Snapshots `SparkConf` at init.
- Provider SPI keyed by format; a provider activates only when
`spark.sql.gravitino.<format>REST.uri` is set, so the URI doubles as the
per-format switch and no `enable*` flag is needed. If a `uri` is set but its
provider is not on the classpath, the plugin fails fast.
- `@DeveloperApi CatalogRegistrationPolicy` — `boolean shouldRegister(String
format, String catalogName)` and `default String registeredCatalogName(String
format, String catalogName)` — selected with
`spark.sql.gravitino.REST.registrationPolicy`. The default implementation
registers every advertised catalog under its advertised name.
- Precedence is enforced by the plugin, not by the policy: names already
claimed by a user's `spark.sql.catalog.<name>` are dropped before the policy
runs; returned names are validated (duplicate, collides with user
configuration, or not a valid Spark identifier -> fail fast); user per-catalog
sub-key > plugin-generated key > global `catalogProperties.<key>`.
- `GravitinoDriverPlugin` logs `Register catalog {} failed` with a stack
trace when the name is already registered. With this plugin that fires once per
discovered catalog and reads as an error; distinguish "already registered by
user configuration" from a genuine failure.
**Lance provider**
- Prefix `spark.sql.gravitino.lanceREST.{uri, catalogProperties.<key>}`.
- Lists catalogs with the Lance Namespace root list `GET
{lance-rest-base}/v1/namespace/list` via
`org.lance.namespace.client.apache.api.NamespaceApi` from the lance-spark
bundle, following `page_token`. The server serves exactly the
`lakehouse-generic` catalogs from Gravitino and rejects anything else, so the
advertised names are by construction valid `parent` values.
- Generates, for each accepted catalog:
```text
spark.sql.catalog.<sparkName>=org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.<sparkName>.impl=rest
spark.sql.catalog.<sparkName>.uri=<spark.sql.gravitino.lanceREST.uri>
spark.sql.catalog.<sparkName>.parent=<advertisedName>
```
- Injects `org.lance.spark.extensions.LanceSparkSessionExtensions` into
`spark.sql.extensions`, reusing the existing deduplication so a manually
configured extension is not added twice.
- Generates no storage configuration: `lance.storage.*` static pass-through
is unchanged.
- Unit tests (fake provider for the core; generated-configuration and
precedence assertions), an integration test against the Gravitino Lance REST
server, and a user-facing page under `docs/spark-connector/`.
Design doc:
https://github.com/apache/gravitino/blob/main/design-docs/spark-rest-catalog-registration.md
### Parent issue
https://github.com/apache/gravitino/issues/11064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
