Hi all,
+1 on aligning Azure and GCS with the S3 storage-config
pattern.
On private links, if the setup keeps the public storage
hostname, it's really a DNS and routing concern, and
Polaris already mints against {account}.dfs.core.windows.net
(or blob) and the public GCS IAM and STS hosts. The
endpoint fields only come into play when Polaris and
clients reach storage through different hosts, or when the
host isn't the public cloud name at all. I think that's
the split for the private-link use case Yufei mentioned.
For naming, I'll follow JB's suggestion and use the S3
names: endpoint for clients and endpointInternal for
Polaris. Clients would get the storage endpoint as FileIO
extras (adls.connection-string for Azure, gcs.service.host
for GCS), and AzureLocation and the allowed-location checks
stay untouched. The new fields go behind a realm flag, off
by default. I'll keep stsEndpoint out of this spec for both
Azure and GCS, since the Entra authority and GCP STS/IAM
hosts belong with server identity, and DefaultAzureCredential
should already pick up AZURE_AUTHORITY_HOST.
As Dmitri mentioned, these can be first-class fields on the
Azure and GCS Storage Configuration, with the named
credentials work [5951] [5952] going on in parallel. On
universeDomain, DownscopedCredentials builds the STS URL as
https://sts.{universe}/v1/token, so a local or custom STS
URL would still need handling in GcpCredentialOps.
I'll start with the Management API spec PR and follow up
with the implementation..
Thanks,
Vignesh
[5951] https://github.com/apache/polaris/pull/5951
[5952] https://github.com/apache/polaris/pull/5952