iting0321 opened a new pull request, #3820: URL: https://github.com/apache/polaris/pull/3820
<!-- ๐ Describe what changes you're proposing, especially breaking or user-facing changes. ๐ See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more. --> ## Summary This PR adds integration tests for Apache Paimon table format support in the Polaris Spark catalog, along with the necessary Paimon helper implementation. ## The key change to Paimon architecture The original implementation attempted to configure Polaris as a delegate catalog (following the Delta/Hudi integration pattern). However, this approach is not compatible with Paimon, **as Paimon's SparkCatalog does not implement the delegation model.** Paimon's architecture is fundamentally different: - Delta/Hudi: Extend Spark's `DelegatingCatalogExtension`, allowing them to delegate catalog operations (namespace, table listing) to another catalog while handling format-specific operations themselves. - Paimon: Has its own `SparkCatalog` that directly manages its warehouse. ## Solution Instead of delegation, we implement a hybrid approach: - Paimon operations: A Paimon catalog (NoopPaimonCatalog) handles all Paimon-specific operations directly against its own warehouse. - Polaris registration: After Paimon creates a table, we register it as an external table in Polaris so it appears in the unified catalog view. - Warehouse separation: Paimon tables use a separate warehouse directory (paimon-warehouse) configured via `spark.sql.catalog.<catalog>.paimon-warehouse` ## How to Test ``` ./gradlew :polaris-spark-integration-3.5_2.12:intTest :polaris-spark-integration-3.5_2.13:intTest ``` ## Checklist - [x] ๐ก๏ธ Don't disclose security issues! (contact [email protected]) - [x] ๐ Clearly explained why the changes are needed, or linked related issues: #3723 - [x] ๐งช Added/updated tests with good coverage, or manually tested (and explained how) - [x] ๐ก Added comments for complex logic - [ ] ๐งพ Updated `CHANGELOG.md` (if needed) - [ ] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
