Hi folks, I would like to discuss a use case and potentially an enhancement on Polaris.
The use case is as follows: 1. I have one Polaris catalog. 2. I have a S3 StorageConfigurationInfo (containing my S3 credentials, allowed locations, region, endpoint). 3. Optionally, at namespace level, I can have base-location, inheriting the catalog's storage credentials. 4. A table level, I can specify a location/base-location, again inheriting the catalog's storage credentials. It means that a table can have its data in s3://bucket-a/table1/, and another table can use s3://bucket-b/table2/, as long as both buckets are accessible with the same catalog-level credentials and are in the catalog's allowed locations. Now, I would like to have (again in a single catalog), one table on a S3, another table on another S3. The credential vending walks up the entity hierarchy (FileIOUtil.findStorageInfoFromHierarchy) and uses the first StorageConfigurationInfo it finds, which is always at the catalog level. The only viable option today is to create separate catalogs, each with its own S3 storage configuration pointing to a different bucket. I would like to introduce an "identifier" on StorageConfigurationInfo and having several StorageConfigurationInfos defined at catalog level. Then, we could add an extra table property to define the StorageConfigurationInfo identifier to us. Thoughts ? I would be happy to work on a proposal about that if there is interest. Regards JB
