Hi Yun, I think we're in agreement regarding the property bag :)
> — Does the "prefix" in the current API spec apply to non-credential properties? > If I understand correctly, you’re asking whether we should use > something like s3.client.region instead of just client.region. The name "s3.client.region" LGTM, but my concern was different. I specifically mean the storage path "prefix" field inside StorageAccessCredential in the current API spec PR [1] I understand that this prefix is relevant to credentials, which may differ for various storage paths. My question is whether the same location prefix applies to non-credential properties? For example to "client.region" and "client.refresh-credentials-endpoint" ? [1] https://github.com/apache/polaris/pull/3826#discussion_r2878717169 Thanks, Dmitri. On Tue, Mar 3, 2026 at 6:15 PM yun zou <[email protected]> wrote: > Hi Dmirti, > > — Who defines property names such as s3.secret-access-key and their > meaning? > > I understand your perspective — you're looking at this from the angle > of who defines the configuration keys within Polaris. > The angle I’m considering is slightly different: who drives the > evolution of those configurations. In other words, we don’t > fundamentally control which configuration parameters are required to > initiate storage I/O — those are driven by the storage providers. > I think both perspectives are valid. However, using a flexible > property bag approach would give us more adaptability as configuration > requirements evolve over time. > > — It would effectively require a change in Polaris specs to define the > new names and what the corresponding values represent. > Yes, I’m fully aligned with this. We should ensure that all supported > configuration keys are clearly documented. In other words, whenever we > add or deprecate a key, there should be corresponding specification > and documentation updates. > > — Does the "prefix" in the current API spec apply to non-credential > properties? > > If I understand correctly, you’re asking whether we should use > something like s3.client.region instead of just client.region. > From my perspective, we probably should use the storage-specific > prefix (e.g., s3.client.region) since these configurations are > typically storage-type specific. I previously used client.region > because that is what we currently use in Polaris, likely influenced by > Iceberg’s conventions. > For generic tables, however, we are not required to stay 100% aligned > with Iceberg. The only drawback of introducing storage-specific > prefixes would be that we may need to put some special logic while > handling keys for GenericTable response, but I am open for that. > > Best Regards, > Yun > > On Tue, Mar 3, 2026 at 7:07 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > Hi Yun, > > > > Just trying to achieve clarity too: > > > > > I’d like to clarify my point regarding configuration ownership. [...] > > > > Who defines property names such as "s3.secret-access-key" and their > meaning? > > > > My view is that Polaris defines those names and links their meaning to > the > > corresponding concepts in S3 API / scec. Polaris exposes those values to > > its clients, therefore they should have a common understanding of the > > meaning of those values. Since Polaris owns the API that exposes those > > properties, Polaris also controls their definitions. > > > > Even though properties must align with a storage provider's API, Polaris > > still needs to clearly define their meaning. > > > > Therefore, if Polaris were to expose new similar properties for another > > technology, it would effectively require a change in Polaris specs to > > define the new names and what the corresponding values represent. WDYT? > > > > I previously mentioned this in [1] but the emphasis was on using > structured > > OpenAPI types. Now, I agree that a property bag is fine to use as the API > > type, but I still think adding new property names requires a spec change > > (in the description section, given the current PR state). > > > > > >>> you're planning to expose different ordianry configuration for > > different storage prefixes too > > > Could you clarify this question a bit further? > > > > I mean: does the "prefix" in the current API spec [2] apply to > > non-credential properties? > > > > [1] https://github.com/apache/polaris/pull/3826#discussion_r2835541976 > > > > [2] https://github.com/apache/polaris/pull/3826#discussion_r2878717169 > > > > Thanks, > > Dmitri. > > > > On Mon, Mar 2, 2026 at 7:14 PM yun zou <[email protected]> > wrote: > > > > > Hi Dmitri, > > > > > > I’d like to clarify my point regarding configuration ownership. > > > Ultimately, it is the storage providers who determine what > > > configurations are required to access their storage systems. Polaris > > > simply provides a mechanism to capture and pass along those > > > configurations. In that sense, the required configuration is defined > > > and controlled by the storage providers, not by Polaris. > > > > > > From my understanding, Iceberg introduced the new storageCredential > > > field mainly for two reasons: > > > 1) The existing config field contains a mix of different configuration > > > types and lacks clear documentation, making it difficult to reuse or > > > extend for other purposes. > > > 2) A new credentials endpoint was introduced, which requires a clearer > > > and more structured definition of the configurations it accepts. > > > > > > For Polaris, as you mentioned, we have the flexibility to design this > > > correctly from the start. Since we do not plan to introduce a separate > > > credentials endpoint, and there are only very limited additional > > > configurations required beyond the credentials themselves for storage > > > access, it would be more practical to keep everything consolidated to > > > maintain simplicity. > > > > > > Furthermore, because we will clearly document all supported > > > configurations, we can establish a well-defined boundary for what > > > belongs in this field and avoid the ambiguity seen elsewhere. > > > > > > >>> you're planning to expose different ordianry configuration for > > > different storage prefixes too > > > Could you clarify this question a bit further? My proposal remains the > > > same as what the current PR outlines (except the naming): we would > > > have a single configuration bag that contains all configurations > > > required to access storage, and different providers would be > > > distinguished based on the storage prefix. > > > > > > Best Regards, > > > Yun > > > > > > On Mon, Mar 2, 2026 at 3:40 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > > Hi Yun, > > > > > > > > I'm fine with a property bag. > > > > > > > > Still, for the sake of clarity, I'd like to mention that it is not > > > storage > > > > providers who control this configuration. Polaris controls it via > its API > > > > specification. > > > > > > > > Storage Providers control the meaning of values that go into those > > > > properties. Property names remain within the scope of Polaris' > > > > responsibility to define and document for its clients. > > > > > > > > So, adding a new storage provider type will require updating the > Polaris > > > > API spec to clearly define the new property names and how they > relate to > > > > what the storage provider expects for request authentication. > > > > > > > > Re: the separation of plain configuration and credentials, my > reading of > > > > the Iceberg discussion makes me lead toward separating them upfront. > This > > > > is what the Iceberg community seems to prefer too, but they are > burdened > > > by > > > > backward compatibility to older clients. Polaris does not have that > > > burden > > > > since we're exposing vended credentials for the first time in the > Generic > > > > Tables API. WDYT? > > > > > > > > Perhaps I misunderstood this proposal and you're planning to expose > > > > different ordianry configuration for different storage prefixes too > > > > (similar to credentials)... Is that so? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Mar 2, 2026 at 4:37 PM yun zou <[email protected]> > > > wrote: > > > > > > > > > Hi Dmitri, > > > > > > > > > > Thanks for bringing up this discussion. > > > > > > > > > > I believe keeping the storage configuration as a property bag is > more > > > > > beneficial for us, since these settings are fundamentally > controlled > > > > > by the cloud provider rather than Polaris. Leaving it as a property > > > > > bag gives us more flexibility and allows us to onboard new I/O > > > > > providers more quickly. Onboarding one additional provider may not > > > > > seem significant, but if we need to do this more frequently, it > could > > > > > add up to a substantial amount of work. > > > > > > > > > > I’m definitely +1 on having clear documentation. > > > > > > > > > > Regarding other parameters such as client.region, since these are > all > > > > > configurations required to initialize the client and access the > > > > > storage, I don’t think we need to split them into separate fields. > > > > > That said, I agree that the name StorageAccessCredentials can be > > > > > misleading. Perhaps we could rename it to StorageAccessConfigs, to > > > > > better reflect that it includes all configurations needed to access > > > > > remote storage, including credentials. WDYT? > > > > > > > > > > By the way, I also started a thread in the Apache Iceberg > community to > > > > > better understand the historical context behind the > > > > > storage-credentials fields. From the discussion so far, it seems > that > > > > > one of the main reasons is the lack of clear documentation. > > > > > > > > > > Best regards, > > > > > Yun > > > > > > > > > > On Mon, Feb 23, 2026 at 9:48 AM Dmitri Bourlatchkov < > [email protected]> > > > > > wrote: > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > I'm generally fine with not distinguishing credential properties > at > > > the > > > > > API > > > > > > level. So property bags for table-defaults plus path-specific > > > overrides > > > > > > sound reasonable to me. > > > > > > > > > > > > However, Iceberg has a separate credentials section in > > > LoadTableResult. > > > > > Do > > > > > > you know the rationale for that? > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://github.com/apache/iceberg/blob/apache-iceberg-1.10.1/open-api/rest-catalog-open-api.yaml#L3321 > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Feb 23, 2026 at 12:07 PM Alexandre Dutra < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > + 1 for keeping the property bag at the API level and document > > > > > > > well-known properties. > > > > > > > > > > > > > > On the credential vs non-credential topic: my point is that in > > > > > > > Iceberg's `LoadTableResult`, the `config` field could contain > > > storage > > > > > > > credentials as well, and that would be, afaict, perfectly > valid: > > > the > > > > > > > credentials would then be valid for any prefix. Polaris, btw, > does > > > > > > > exactly that [1]. Therefore it would be more accurate, strictly > > > > > > > speaking, to distinguish two scopes: "table defaults", and > > > > > > > "prefix-specific overrides"; these would be orthogonal to the > > > > > > > credential vs non-credential distinction, as illustrated in > this > > > > > > > example: [2]. > > > > > > > > > > > > > > Thanks, > > > > > > > Alex > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/8b108d6be7222a8ed78b1b2b70816ecbeea1b327/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L876-L881 > > > > > > > [2]: > > > https://gist.github.com/adutra/1b6adb2c169c08bd91cd2a01ab4338d3 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 23, 2026 at 4:12 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > Re: config property mixing, my concern is not so much about > > > prefix > > > > > vs. > > > > > > > > non-prefix config, but more about credentials vs. > non-credentials > > > > > > > > configuration. > > > > > > > > > > > > > > > > I believe distinguishing them at the API level will help > > > > > implementations > > > > > > > > treat them correctly too. Conversely, mixing credentials with > > > other > > > > > > > config > > > > > > > > increases the chance of mis-handling them on the client side > > > (e.g. > > > > > > > logging > > > > > > > > when they are not supposed to be logged). > > > > > > > > > > > > > > > > Your point about the fact that these properties will > eventually > > > be > > > > > mixed > > > > > > > on > > > > > > > > the client side is quite valid. Therefore, I will not insist > on > > > > > > > separating > > > > > > > > credential properties from ordinary config in the API. > > > > > > > > > > > > > > > > However, if the properties are mixed we should not use the > name > > > > > > > > "StorageAccessCredential" (note the last word) for their > > > container, > > > > > > > because > > > > > > > > that (IMHO) would actually make the API spec confusing. I'd > > > suggest > > > > > > > naming > > > > > > > > the container "StorageAccessConfiguration" (other names are > > > welcome > > > > > too). > > > > > > > > > > > > > > > > Re: well-structured properties objects - after thinking > about it > > > some > > > > > > > more, > > > > > > > > I believe your points about client compatibility prevail. I'd > > > think > > > > > > > clients > > > > > > > > should be able to hangle new / unknown JSON properties, but > it > > > would > > > > > > > > certainly be an overhead in many cases. > > > > > > > > > > > > > > > > I'm fine using a generic property bag as the type at the API > > > level. > > > > > > > > > > > > > > > > I still believe we should document exact property names in > the > > > spec. > > > > > For > > > > > > > > that matter I propose moving property descriptions from Open > API > > > > > > > _comments_ > > > > > > > > to Open API _description_ fields. This way, notes about > property > > > > > meaning > > > > > > > > will not be lost after automated processing of the spec > (e.g. by > > > > > > > > SwaggerHub). WDYT? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Feb 23, 2026 at 7:27 AM Alexandre Dutra < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > I do not think it is correct to mix the former with the > > > latter. > > > > > > > > > > > > > > > > > > I don't think mixing general and prefix-specific > > > configurations is > > > > > > > > > inherently incorrect—it's a grey area. However, I agree > that > > > it is > > > > > > > > > likely better practice to separate them. I believe the > > > sensitivity > > > > > > > > > level of the configuration should not be the deciding > factor, > > > as > > > > > any > > > > > > > > > configuration in a LoadTable response should be treated as > > > > > sensitive. > > > > > > > > > > > > > > > > > > In Iceberg, the distinction in `LoadTableResult` between > > > `config` > > > > > and > > > > > > > > > `storage-credentials.config` is primarily one of scope: the > > > former > > > > > > > > > applies broadly to the entire FileIO, and the latter is > > > specific > > > > > to an > > > > > > > > > S3 client tied to a particular prefix. It's important to > note, > > > > > though, > > > > > > > > > that any table-wide properties are ultimately merged with > > > > > > > > > prefix-specific properties when the S3 client is created > [1]. > > > > > > > > > > > > > > > > > > > Open API can be leveraged to report them as > well-structured > > > > > objects > > > > > > > > > (JSON) > > > > > > > > > > > > > > > > > > The idea is interesting, especially since we control the > > > > > specification > > > > > > > > > and thus know the supported properties. > > > > > > > > > > > > > > > > > > However, this approach may complicate the evolution of the > > > > > > > > > specification. Our guidance on evolution [2] states that > new > > > > > releases > > > > > > > > > of Polaris should maintain compatibility with older > clients. > > > This > > > > > > > > > requires that any addition or deprecation of properties > must be > > > > > done > > > > > > > > > in a backward-compatible way. Removing or renaming > properties > > > would > > > > > > > > > necessitate a major version bump for the specification. I > am > > > not > > > > > > > > > convinced that the potential benefits outweigh these > > > consequences. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/iceberg/blob/fec9800bcc0c4073ca727f3b3bfdc2f34abb26a3/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L411-L415 > > > > > > > > > [2]: https://polaris.apache.org/releases/1.3.0/evolution/ > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Feb 21, 2026 at 1:12 AM Dmitri Bourlatchkov < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > I'm transferring some points from my GH comments [3826] > here, > > > > > for a > > > > > > > wider > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > 1) Apparently some of the response properties relate to > > > actual > > > > > > > > > credentials > > > > > > > > > > (key, expiry time), while others are more general > > > configuration > > > > > items > > > > > > > > > (e.g. > > > > > > > > > > the refresh endpoint). > > > > > > > > > > > > > > > > > > > > I do not think it is corrent to mix the former with the > > > latter. > > > > > > > Primarily > > > > > > > > > > because of their different leak sensitivity levels but > also > > > > > because > > > > > > > > > > `StorageAccessCredential` are provided as a list, and I > > > wonder > > > > > why we > > > > > > > > > would > > > > > > > > > > want to send multiple endpoint config entries (one for > each > > > > > location > > > > > > > > > > prefix). > > > > > > > > > > > > > > > > > > > > 2) Currently properties are defined as an unstructured > bag of > > > > > > > key/value > > > > > > > > > > pair. I think Open API can be leveraged to report them as > > > > > > > well-structured > > > > > > > > > > objects (JSON). > > > > > > > > > > > > > > > > > > > > Server code changes are required to add suport for new > > > properties > > > > > > > anyway. > > > > > > > > > > It should not be too difficult to evolve the Open API > types > > > at > > > > > the > > > > > > > > > > same time. We've done it many times already (e.g. to > support > > > > > non-AWS > > > > > > > S3 > > > > > > > > > > storage). > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > [3826] > > > > > > > > https://github.com/apache/polaris/pull/3826/changes#r2829649503 > > > > > > > > > > > > > > > > > > > > On Sat, Feb 7, 2026 at 1:07 AM Jack Ye < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > How are clients supposed to know the meaning of > > > credential > > > > > > > > > properties in > > > > > > > > > > > the "config" section? > > > > > > > > > > > > > > > > > > > > > > I think the contract is clear: since we are using the > > > Iceberg > > > > > REST > > > > > > > > > model > > > > > > > > > > > for StorageCredential, it must use the same > configurations > > > as > > > > > the > > > > > > > > > Iceberg > > > > > > > > > > > credentials vending-related ones [1]. > > > > > > > > > > > > > > > > > > > > > > Based on the current Polaris implementation [2], the > full > > > list > > > > > of > > > > > > > > > configs > > > > > > > > > > > is probably the following: > > > > > > > > > > > > > > > > > > > > > > S3: > > > > > > > > > > > Credentials: > > > > > > > > > > > - s3.access-key-id - AWS access key ID > > > > > > > > > > > - s3.secret-access-key - AWS secret access key > > > > > > > > > > > - s3.session-token - Temporary STS session token > > > > > > > > > > > - s3.session-token-expires-at-ms - Token expiration > > > timestamp > > > > > > > (ms) > > > > > > > > > > > Extra Properties: > > > > > > > > > > > - s3.endpoint - S3 endpoint URI (optional) > > > > > > > > > > > - s3.path-style-access - Path-style access flag > > > (optional) > > > > > > > > > > > - client.region - AWS region > > > > > > > > > > > - aws.refresh-credentials-endpoint - Credential > refresh > > > > > endpoint > > > > > > > > > > > (optional) > > > > > > > > > > > > > > > > > > > > > > GCS: > > > > > > > > > > > Credentials: > > > > > > > > > > > - gcs.oauth2.token - Downscoped OAuth2 access token > > > > > > > > > > > - gcs.oauth2.token-expires-at - Token expiration > > > timestamp > > > > > (ms) > > > > > > > > > > > Extra Properties: > > > > > > > > > > > - gcs.oauth2.refresh-credentials-endpoint - > Credential > > > > > refresh > > > > > > > > > endpoint > > > > > > > > > > > (optional) > > > > > > > > > > > > > > > > > > > > > > Azure: > > > > > > > > > > > Credentials: > > > > > > > > > > > - adls.sas-token.<hostname> - SAS token keyed by > storage > > > > > account > > > > > > > > > hostname > > > > > > > > > > > - adls.sas-token-expires-at-ms.<hostname> - SAS token > > > > > expiration > > > > > > > (ms) > > > > > > > > > > > Extra Properties: > > > > > > > > > > > - adls.refresh-credentials-endpoint - Credential > refresh > > > > > endpoint > > > > > > > > > > > (optional) > > > > > > > > > > > > > > > > > > > > > > It would be helpful to specify those at least in the > > > generic > > > > > tables > > > > > > > > > spec, > > > > > > > > > > > ideally also in the Iceberg REST spec, which currently > only > > > > > lists > > > > > > > S3 > > > > > > > > > > > configs and is already outdated. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Jack Ye > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/spec/iceberg-rest-catalog-open-api.yaml#L3299C1-L3306C1 > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 6, 2026 at 11:03 AM Eric Maynard < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > How are clients supposed to know the meaning of > > > credential > > > > > > > > > properties > > > > > > > > > > > in > > > > > > > > > > > > the "config" section? > > > > > > > > > > > > > > > > > > > > > > > > How are clients supposed to know the meaning of *any* > > > > > properties > > > > > > > > > written > > > > > > > > > > > to > > > > > > > > > > > > a generic table? > > > > > > > > > > > > > > > > > > > > > > > > That clients need to interpret the payload of a > generic > > > table > > > > > > > > > response is > > > > > > > > > > > > already intrinsic to the generic table design. True, > > > adding > > > > > > > > > credential > > > > > > > > > > > > vending pushes generic table support towards the > service > > > > > being > > > > > > > > > slightly > > > > > > > > > > > > more opinionated about the generic table metadata > (i.e. > > > the > > > > > > > location > > > > > > > > > is > > > > > > > > > > > now > > > > > > > > > > > > implied to be a place that may require credentials to > > > access) > > > > > > > but as > > > > > > > > > this > > > > > > > > > > > > would be an opt-in for your generic tables I don't > see > > > this > > > > > as a > > > > > > > > > blocking > > > > > > > > > > > > issue. > > > > > > > > > > > > > > > > > > > > > > > > --EM > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 6, 2026 at 10:57 AM Dmitri Bourlatchkov < > > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > > > The proposal looks reasonable to me in general > after a > > > > > quick > > > > > > > > > review of > > > > > > > > > > > > the > > > > > > > > > > > > > doc. > > > > > > > > > > > > > > > > > > > > > > > > > > I have one concern, though, which may be a whole > can of > > > > > worms, > > > > > > > I'm > > > > > > > > > > > afraid > > > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > How are clients supposed to know the meaning of > > > credential > > > > > > > > > properties > > > > > > > > > > > in > > > > > > > > > > > > > the "config" section? The doc proposes to define > it as > > > a > > > > > > > generic > > > > > > > > > > > property > > > > > > > > > > > > > bag. > > > > > > > > > > > > > > > > > > > > > > > > > > The example appears to use properties that Iceberg > > > (java?) > > > > > > > clients > > > > > > > > > > > might > > > > > > > > > > > > > use in a similar situation. However, the Generic > Tables > > > > > API is > > > > > > > not > > > > > > > > > > > > related > > > > > > > > > > > > > to Iceberg in any way (AFAIK). > > > > > > > > > > > > > > > > > > > > > > > > > > Plus, I do not think these properties are > well-defined > > > > > even in > > > > > > > > > Iceberg > > > > > > > > > > > > > specifications. > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 6, 2026 at 1:29 PM yun zou < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Generic Tables have been available since Polaris > > > 1.0.0 > > > > > and > > > > > > > have > > > > > > > > > seen > > > > > > > > > > > > > > growing interest from an increasing number of > > > customers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > However, the current Generic Table capability has > > > some > > > > > > > > > limitations. > > > > > > > > > > > One > > > > > > > > > > > > > > key gap is the lack of credential vending > support. > > > > > Without > > > > > > > > > credential > > > > > > > > > > > > > > vending, query engines must access tables using > > > > > long-lived, > > > > > > > > > static > > > > > > > > > > > > cloud > > > > > > > > > > > > > > credentials configured directly in the engine > > > runtime, > > > > > which > > > > > > > > > limits > > > > > > > > > > > > both > > > > > > > > > > > > > > usability and security. > > > > > > > > > > > > > > > > > > > > > > > > > > > > To address this, we propose adding credential > vending > > > > > > > support for > > > > > > > > > > > > Generic > > > > > > > > > > > > > > Tables. This enhancement would allow a Polaris > > > catalog to > > > > > > > > > dynamically > > > > > > > > > > > > > vend > > > > > > > > > > > > > > short-lived, scoped storage credentials to query > > > engines > > > > > at > > > > > > > > > runtime > > > > > > > > > > > > when > > > > > > > > > > > > > > accessing Generic Tables. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The goals of this proposal are to: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Enable credential vending support for Generic > > > Tables > > > > > in > > > > > > > > > Polaris > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Deliver an end-to-end experience for currently > > > > > supported > > > > > > > table > > > > > > > > > > > > > > formats, including Delta, Hudi, and Lance > > > > > > > > > > > > > > 3. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Maintain consistency with the existing Iceberg > > > > > credential > > > > > > > > > vending > > > > > > > > > > > > > model > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please find the attached short design document > with > > > > > > > additional > > > > > > > > > > > details. > > > > > > > > > > > > > We > > > > > > > > > > > > > > would appreciate your review and valuable > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > link to google doc: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1QzTx4tcS23_mF-gc77GbTqtwuRHY5f_Aa_6E4VSKFU4/edit?tab=t.0#heading=h.rpqtaz73xt4v > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
