sririshindra opened a new pull request, #3923: URL: https://github.com/apache/polaris/pull/3923
Introduces the ability to define `StorageConfiguration` at the `Namespace` and `Table` levels, rather than restricting it solely to the `Catalog` level. Based on the recent community discussions (https://lists.apache.org/thread/dt8z173709bk4tbb6lg27dkkdxgvtb2b) around table-level storage credential overrides, this implements the "inline model" approach. This feature significantly increases flexibility for users who need to organize tables across different storage configurations, buckets, or cloud providers within a single logical catalog (e.g., keeping US data in AWS and European data in GCP without needing artificial catalog boundaries). The storage configuration resolves hierarchically: `Table` โ `Namespace(s)` โ `Catalog` (default). If a table has a specific configuration, it is used. Otherwise, it falls back to the nearest namespace ancestor, and finally to the catalog. ### Key Changes - **Data Model Extensions**: Added `getStorageConfigurationInfo()` to `NamespaceEntity` and `TableLikeEntity` to allow these entities to expose embedded storage configs. - **Management API Endpoints**: Added 6 new REST endpoints (GET/PUT/DELETE) in the OpenAPI spec and `PolarisServiceImpl` to manage storage configurations at the namespace and table levels. - **Hierarchical Resolution & Hot Path Integration**: - Updated `IcebergCatalog` to build and pass the complete `PolarisResolvedPathWrapper` (catalog โ namespace(s) โ table) into the existing `FileIOUtil.findStorageInfoFromHierarchy()` logic. This ensures existing credential vending paths correctly respect table and namespace overrides. - Reused this same leaf-to-root walk for the new Management API GET endpoints to determine the effective configuration. - Added debug logging to `FileIOUtil` to trace resolution decisions. - **Config Source Header**: Added the `X-Polaris-Storage-Config-Source` HTTP response header to GET endpoints to explicitly indicate which level (`TABLE`, `NAMESPACE`, or `CATALOG`) provided the effective storage configuration. - **`storageName` Preservation**: Fully compatible with the recent storage-scoped AWS credentials feature (#3409). Ensured `storageName` is correctly preserved during API/internal model conversions, guaranteeing isolated server-side credentials work correctly at the namespace and table levels. - **Testing**: Added comprehensive unit tests, service-layer conflict mapping tests (403/409 handling on concurrent updates), HTTP integration tests (`PolarisStorageConfigIntegrationTest`), and a dedicated Spark integration test suite (`RealVendingProfile`) to validate end-to-end STS credential vending across the entire fallback hierarchy. ## Checklist - [ ] ๐ก๏ธ Don't disclose security issues! (contact [email protected]) - [ ] ๐ Clearly explained why the changes are needed, or linked related issues: Fixes # - [ ] ๐งช Added/updated tests with good coverage, or manually tested (and explained how) - [ ] ๐ก Added comments for complex logic - [ ] ๐งพ Updated `CHANGELOG.md` (if needed) - [ ] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
