Hi all, > I do not think it is correct to mix the former with the latter.
I don't think mixing general and prefix-specific configurations is inherently incorrect—it's a grey area. However, I agree that it is likely better practice to separate them. I believe the sensitivity level of the configuration should not be the deciding factor, as any configuration in a LoadTable response should be treated as sensitive. In Iceberg, the distinction in `LoadTableResult` between `config` and `storage-credentials.config` is primarily one of scope: the former applies broadly to the entire FileIO, and the latter is specific to an S3 client tied to a particular prefix. It's important to note, though, that any table-wide properties are ultimately merged with prefix-specific properties when the S3 client is created [1]. > Open API can be leveraged to report them as well-structured objects (JSON) The idea is interesting, especially since we control the specification and thus know the supported properties. However, this approach may complicate the evolution of the specification. Our guidance on evolution [2] states that new releases of Polaris should maintain compatibility with older clients. This requires that any addition or deprecation of properties must be done in a backward-compatible way. Removing or renaming properties would necessitate a major version bump for the specification. I am not convinced that the potential benefits outweigh these consequences. Thanks, Alex [1]: https://github.com/apache/iceberg/blob/fec9800bcc0c4073ca727f3b3bfdc2f34abb26a3/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L411-L415 [2]: https://polaris.apache.org/releases/1.3.0/evolution/ On Sat, Feb 21, 2026 at 1:12 AM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi All, > > I'm transferring some points from my GH comments [3826] here, for a wider > discussion. > > 1) Apparently some of the response properties relate to actual credentials > (key, expiry time), while others are more general configuration items (e.g. > the refresh endpoint). > > I do not think it is corrent to mix the former with the latter. Primarily > because of their different leak sensitivity levels but also because > `StorageAccessCredential` are provided as a list, and I wonder why we would > want to send multiple endpoint config entries (one for each location > prefix). > > 2) Currently properties are defined as an unstructured bag of key/value > pair. I think Open API can be leveraged to report them as well-structured > objects (JSON). > > Server code changes are required to add suport for new properties anyway. > It should not be too difficult to evolve the Open API types at the > same time. We've done it many times already (e.g. to support non-AWS S3 > storage). > > WDYT? > > Thanks, > Dmitri. > > [3826] https://github.com/apache/polaris/pull/3826/changes#r2829649503 > > On Sat, Feb 7, 2026 at 1:07 AM Jack Ye <[email protected]> wrote: > > > > How are clients supposed to know the meaning of credential properties in > > the "config" section? > > > > I think the contract is clear: since we are using the Iceberg REST model > > for StorageCredential, it must use the same configurations as the Iceberg > > credentials vending-related ones [1]. > > > > Based on the current Polaris implementation [2], the full list of configs > > is probably the following: > > > > S3: > > Credentials: > > - s3.access-key-id - AWS access key ID > > - s3.secret-access-key - AWS secret access key > > - s3.session-token - Temporary STS session token > > - s3.session-token-expires-at-ms - Token expiration timestamp (ms) > > Extra Properties: > > - s3.endpoint - S3 endpoint URI (optional) > > - s3.path-style-access - Path-style access flag (optional) > > - client.region - AWS region > > - aws.refresh-credentials-endpoint - Credential refresh endpoint > > (optional) > > > > GCS: > > Credentials: > > - gcs.oauth2.token - Downscoped OAuth2 access token > > - gcs.oauth2.token-expires-at - Token expiration timestamp (ms) > > Extra Properties: > > - gcs.oauth2.refresh-credentials-endpoint - Credential refresh endpoint > > (optional) > > > > Azure: > > Credentials: > > - adls.sas-token.<hostname> - SAS token keyed by storage account hostname > > - adls.sas-token-expires-at-ms.<hostname> - SAS token expiration (ms) > > Extra Properties: > > - adls.refresh-credentials-endpoint - Credential refresh endpoint > > (optional) > > > > It would be helpful to specify those at least in the generic tables spec, > > ideally also in the Iceberg REST spec, which currently only lists S3 > > configs and is already outdated. > > > > Best, > > Jack Ye > > > > [1] > > > > https://github.com/apache/polaris/blob/main/spec/iceberg-rest-catalog-open-api.yaml#L3299C1-L3306C1 > > [2] > > > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java > > > > > > On Fri, Feb 6, 2026 at 11:03 AM Eric Maynard <[email protected]> > > wrote: > > > > > > How are clients supposed to know the meaning of credential properties > > in > > > the "config" section? > > > > > > How are clients supposed to know the meaning of *any* properties written > > to > > > a generic table? > > > > > > That clients need to interpret the payload of a generic table response is > > > already intrinsic to the generic table design. True, adding credential > > > vending pushes generic table support towards the service being slightly > > > more opinionated about the generic table metadata (i.e. the location is > > now > > > implied to be a place that may require credentials to access) but as this > > > would be an opt-in for your generic tables I don't see this as a blocking > > > issue. > > > > > > --EM > > > > > > On Fri, Feb 6, 2026 at 10:57 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Yun, > > > > > > > > The proposal looks reasonable to me in general after a quick review of > > > the > > > > doc. > > > > > > > > I have one concern, though, which may be a whole can of worms, I'm > > afraid > > > > :) > > > > > > > > How are clients supposed to know the meaning of credential properties > > in > > > > the "config" section? The doc proposes to define it as a generic > > property > > > > bag. > > > > > > > > The example appears to use properties that Iceberg (java?) clients > > might > > > > use in a similar situation. However, the Generic Tables API is not > > > related > > > > to Iceberg in any way (AFAIK). > > > > > > > > Plus, I do not think these properties are well-defined even in Iceberg > > > > specifications. > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Fri, Feb 6, 2026 at 1:29 PM yun zou <[email protected]> > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Generic Tables have been available since Polaris 1.0.0 and have seen > > > > > growing interest from an increasing number of customers. > > > > > > > > > > However, the current Generic Table capability has some limitations. > > One > > > > > key gap is the lack of credential vending support. Without credential > > > > > vending, query engines must access tables using long-lived, static > > > cloud > > > > > credentials configured directly in the engine runtime, which limits > > > both > > > > > usability and security. > > > > > > > > > > To address this, we propose adding credential vending support for > > > Generic > > > > > Tables. This enhancement would allow a Polaris catalog to dynamically > > > > vend > > > > > short-lived, scoped storage credentials to query engines at runtime > > > when > > > > > accessing Generic Tables. > > > > > > > > > > The goals of this proposal are to: > > > > > > > > > > 1. > > > > > > > > > > Enable credential vending support for Generic Tables in Polaris > > > > > 2. > > > > > > > > > > Deliver an end-to-end experience for currently supported table > > > > > formats, including Delta, Hudi, and Lance > > > > > 3. > > > > > > > > > > Maintain consistency with the existing Iceberg credential vending > > > > model > > > > > > > > > > Please find the attached short design document with additional > > details. > > > > We > > > > > would appreciate your review and valuable feedback. > > > > > > > > > > link to google doc: > > > > > > > > > > > > > > https://docs.google.com/document/d/1QzTx4tcS23_mF-gc77GbTqtwuRHY5f_Aa_6E4VSKFU4/edit?tab=t.0#heading=h.rpqtaz73xt4v > > > > > > > > > > Best Regards, > > > > > > > > > > Yun > > > > > > > > > > > > > >
