Hi Yufei, That is a good point. Deployments may need to customize how FileIO is constructed and we cannot foresee all possible customizations. So, we cannot have a fixed set of properties in the API-level storage config for that.
I'd like to propose this flow: * StorageAccessConfigProvider considers Catalog properties and includes them into StorageAccessConfig as appropriate. * Only relevant properties are includes (e.g. no Azure stuff in S3 access paths) * Well-defined entries in the API-level StorageConfigInfo take precedence over Catalog properties * FileIO is constructed purely from StorageAccessConfig Code benefits: * Default table properties are no longer a factor that has to be considered separately on all access paths. * Customized environments can still use Catalog properties to add configuration to FileIO * Server-side FileIO and client-side FileIO (load table responses) are controlled through the same StorageAccessConfigProvider > StorageAccessConfig and its subclasses currently do not support a > freeform key-value map ATM. StorageAccessConfig does have a Map<String, String>... Did you mean PolarisStorageConfigurationInfo? Cheers, Dmitri. On Wed, Oct 7, 2026 at 5:56 PM Yufei Gu <[email protected]> wrote: > I generally favor a clean boundary, and StorageAccessConfig seems like > the right object for server-side FileIO. > > Currently, as Dmitri mentioned, we also inject the default table > properties from the catalog to server-side FileIO (see: > > https://github.com/apache/polaris/blob/19b54f773d345b3f84abbdb6bc18c3c0e1ff01f4/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java#L334 > , > and > https://github.com/apache/polaris/blob/19b54f773d345b3f84abbdb6bc18c3c0e1ff01f4/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java#L469 > ): > > Given that default table properties are also catalog-level properties, > I don't think it is justified to merge them over the catalog's > StorageAccessConfig. If any customization is needed for catalog > storage access configs, we could add a bag of properties, as > StorageAccessConfig and its subclasses currently do not support a > freeform key-value map ATM. > > On Wed, Oct 7, 2026 at 1:20 PM vignesh a <[email protected]> wrote: > > > > Hi Dmitri, > > > > +1 on using only StorageAccessConfig for server-side FileIO. > > > > I'll take the cleanup PR once this settles. Test/dev paths that today > > put MinIO endpoint and keys on table-default.s3.* when AccessConfig is > > empty will need those on storage config instead. We can cover that in > > the PR. WDYT?? > > > > Thanks, > > *Vignesh* > > > > [5328] https://github.com/apache/polaris/pull/5328 > > > > On Wed, 7 Oct 2026 at 00:21, Dmitri Bourlatchkov <[email protected]> > wrote: > > > > > Cross-referencing related discussion: > > > > > > https://lists.apache.org/thread/49vsx3r803nmf35wsyxl2h3qvxfrrfgs > > > > > > On Tue, Oct 6, 2026 at 2:46 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > This discussion stems [1] from PR [5328], but it is rooted in more > > > > fundamental Polaris behaviours. > > > > > > > > ATM the codebase uses a combination of StorageAccessConfig and > _default_ > > > > table properties for constructing server-side FileIO objects. > > > > > > > > Note that it already deviates from what clients may use. > > > > > > > > Load table responses advertise FileIO-related properties in the REST > > > > API-level config section. These properties include vended > credentials and > > > > endpoints. > > > > > > > > Generally, IRC config properties are sufficient for clients to > construct > > > > operational FileIO objects to access storage files for tables loaded > via > > > > Polaris. > > > > > > > > Therefore, I propose that Polaris code use only StorageAccessConfig > for > > > > constructing server-side FileIO objects. > > > > > > > > Thoughts? > > > > > > > > [1] > https://github.com/apache/polaris/pull/5328#discussion_r4184400135 > > > > > > > > [5328] https://github.com/apache/polaris/pull/5328 > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > > > > >
