FANNG1 opened a new pull request, #12541: URL: https://github.com/apache/gravitino/pull/12541
### What changes were proposed in this pull request? Add `GravitinoLakehouseRESTDiscoveryPlugin`, a format-agnostic Spark plugin that discovers lakehouse REST catalogs and registers them during driver startup. This change: - Adds provider and registration policy APIs. - Enforces user configuration precedence and atomic Spark configuration updates. - Adds a Lance provider using the Lance Namespace API with pagination support. - Injects the required Lance Spark extension. - Treats catalogs already configured in Spark as normal skips in `GravitinoDriverPlugin`. - Adds unit tests, Lance integration coverage, and user documentation. ### Why are the changes needed? Spark users currently need to configure every Lance REST catalog manually. Automatic discovery allows the Lance REST server to provide the authoritative catalog list while preserving user-defined Spark catalog configuration. Fix: #12525 ### Does this PR introduce _any_ user-facing change? Yes. Users can enable Lance REST catalog discovery with: ```text spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoLakehouseRESTDiscoveryPlugin spark.sql.gravitino.lanceREST.uri=http://127.0.0.1:9101/lance ``` The change also introduces: - `CatalogRegistrationPolicy` - `LakehouseRESTCatalogProvider` - `spark.sql.gravitino.REST.registrationPolicy` - `spark.sql.gravitino.lanceREST.catalogProperties.<key>` Authenticated Lance REST listing is not supported in this change. ### How was this patch tested? - `./gradlew :spark-connector:spark-common:check` - Lance Spark integration matrix with bundle versions 0.2.0, 0.4.0, and 0.5.1. - Spark 3.3, 3.4, and 3.5 runtime shadow JAR builds. - Packaged runtime smoke test using only the shadow runtime and external Lance bundle. - Verified the packaged provider has no shaded Apache HttpClient linkage. - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
