Hi, I've been looking into untangling the strong coupling of storage credentials and persistence. Storage (think: S3, GCS, etc), storage credentials and persistence are very different things IMO. This coupling also needs to be removed to allow tasks to be executed "anywhere", without having to load/have any access to persistence.
The only (implementation) place that _really_ needs to do this is o.a.p.service.catalog.iceberg.IcebergCatalog. All other places actually do not need to deal with it in the Polaris code base. The only (implementation) place that _really_ needs the PolarisStorageIntegrationProvider is StorageCredentialCache, as it is the only place that eventually performs credential vending. I'm definitely not against your proposal, but want to add that it's likely worth considering the "whole picture". Robert On Mon, Aug 18, 2025 at 6:09 PM Dmitri Bourlatchkov <di...@apache.org> wrote: > > Hi All, > > Currently Polaris core code has a very specific mechanism for > extracting PolarisStorageConfigurationInfo from entities. The config value > is then passed to a PolarisStorageIntegrationProvider. > > Now, PolarisStorageIntegrationProvider is pluggable and works with the > config extracted from entities by Polaris core. I am proposing to add > another integration point that would handle the construction > of PolarisStorageConfigurationInfo and work in conjunction > with PolarisStorageIntegrationProvider. Having both of these actions > customizable is intended to allow more flexibility (and better code > alignment) in handling storage integrations in downstream projects. > > I'm working on a PR to provide a concrete suggestion. > > Sending this email to proactively ask if there are any concerns with this > approach. > > WDYT? > > Thanks, > Dmitri.
