Hi JB, You bring up good points!
We should certainly follow the S3 code patterns when adding customizable endpoints for GCS and ADLS. I'm not sure I agree with relying only on env. variables. StorageConfiguration provides a way to have named base credentials for S3 and Rishi recently proposed to extend that pattern to Azure [5952] and GCS [5951]. I think we can build on that to support catalog-level endpoint overrides plus named based credentials. However, the endpoint work can probably proceed in parallel. I hope we can avoid changes to AzureLocation code. Existing Floci-based tests are operational already, but they rely [1] [2] on the legacy property bags inside Catalog entities, IIRC. We just need the ability to configure similar values as first-class Storage Configuration properties. Re: GCP endpoints, could we override "universeDomain" in DownscopedCredentials? Thanks, Dmitri. [1] https://github.com/apache/polaris/blob/97e02ab062b3c96ef58aa2da7599611ac36908bf/tools/testcontainers/floci-az/src/main/java/org/apache/polaris/test/floci/az/FlociAzureContainer.java#L149 [2] https://github.com/apache/polaris/blob/97e02ab062b3c96ef58aa2da7599611ac36908bf/tools/testcontainers/floci-gcp/src/main/java/org/apache/polaris/test/floci/gcp/FlociGcpContainer.java#L133 [5951] https://github.com/apache/polaris/pull/5951 [5952] https://github.com/apache/polaris/pull/5952 On Mon, Nov 9, 2026 at 2:07 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi Vignesh, > > I agree that we have a difference here. It makes sense to align Azure and > GCS with S3. > > I have a few comments: > - For S3, we distinguish endpoint (what clients see) from endpointInternal > (what Polaris uses), plus stsEndpoint. In private link or Kubernetes > deployments, the host Polaris reaches is often not the one clients use. I > think the Azure/GCS fields should follow the same logic, with the same > naming, so users won't be surprised. > - The security gate is the main point for me. For Azure and GCS, Polaris > sends bearer token to these hosts (the Entra token for the user delegation > key request, the source GCP token for IAM/STS). Unlike SigV4 on S3, a > bearer token can be replayed. As storage config can be set by catalog > admins, the new fields should be guarded by a realm feature flag, similar > to ALLOW_SETTING_S3_ENDPOINTS. Given the bearer token aspect, I would go > with disabled by default (but happy to discuss that :)). I think it's also > aligned with our thread model: configured endpoints must not redirect > credentials outside the intended trust relationship. > - The Entra authority and the GCP STS/IAM hosts related to the server > identity, not to a specific catalog. I would prefer them as server level > configuration rather than catalog storage config fields. For Entra, I think > DefaultAzureCredential already honors the AZURE_AUTHORITY_HOST env > variable, so it's worth to check whether that's enough before adding a new > Polaris config. > - In the Google Auth version we use (1.55.0), I checked and > DownscopedCredentials buids the STS URL from the domain, and the builder > has no endpoint setter. So a custom STS endpoint can't just be passed > through. We would need either a domain approach or our own token exchange > behind GcpCredentialOps. The IAM credentials side is easier, as > IamCredentialsSettings supports setEndpoint(). > - AzureLocation currently rejects any host other than blob/ > dfs.core.windows.net. My preference would be to keep locations unchanged > and have the endpoint override only change the network target. The way, > location validation and allowed locations keep working as they do today. On > the client side, we could use Iceberg adls.connection-string-xxx. > > As this changes the Management API, I suggest to split the work into two > PRs: one for the spec, and the follow up. > > Happy to review the PRs :) > > Regards > JB > > On Sat, Nov 7, 2026 at 4:46 PM vignesh a <[email protected]> wrote: > > > Hi all, > > > > Polaris can already vend S3 credentials against a non-public > > endpoint. Catalog storage configuration includes endpoint and > > stsEndpoint, and the Floci S3 integration tests run > > vended-credentials on that path. > > > > Azure and GCS have no equivalent. Polaris mints Azure > > user-delegation SAS against > > https://{account}.dfs.core.windows.net (or blob), derived from > > the table location. GCS downscoping always uses the public IAM > > and STS hosts. Private-link and emulator deployments cannot > > direct that minting at a local or private endpoint. > > > > The in-tree Floci Azure and GCS integration tests show the > > gap. Iceberg clients receive static shared-key or OAuth tokens > > plus a client host extra, because Polaris cannot vend through > > Floci. > > > > Therefore I propose optional catalog endpoint fields for Azure > > and GCS, analogous to S3 endpoint and stsEndpoint. Polaris > > would use them when minting Azure SAS and GCS downscoped > > tokens, and would pass the storage endpoint to Iceberg clients > > as FileIO extras. Azure token fetch uses the default credential > > chain against public Entra, so an emulator that also issues > > tokens may need an authority override as well. > > > > The Floci Azure and GCS integration tests could then use > > vended-credentials instead of static tokens. > > > > If this direction looks reasonable, I'll put up a PR & > > adjust the shape based on feedback here.. > > > > Thanks, > > Vignesh > > >
