Thanks for the feedback, Adnan! I was initially thinking of deferring separating STS and S3 endpoints to a later PR, but given interest, I'll add it to the current PR to reduce the number of API changes over time.
Cheers, Dmitri. On Wed, Jun 25, 2025 at 1:28 PM Adnan Hemani <adnan.hem...@snowflake.com.invalid> wrote: > +1 on overall idea > > Comment however on keeping the same endpoint for STS and S3 - there may be > a customer use case (I have seen this before for similar use cases) where > the S3 bucket is not in the same region as the STS endpoint that has been > exposed to the Polaris server. In that case, you really need two separate > endpoints (example below). But I don't think this necessarily blocks this > idea from being merged today and then enriched later. > > Example: > * Polaris is deployed in us-west-1 in a EC2 instance that is within a > private subnet, which is given access only to STS' us-west-1 regional > endpoint through AWS configurations. > * User has S3 data in both us-west-1 and us-east-1. > * Under this proposal, user can now access data in us-west-1, but when > trying to access data in us-east-1, STS will not respond due to no network > routing tables for the us-east-1 regional endpoint. > * Important thing to remember is that STS token generated at any regional > endpoint can be used at any other regional endpoint. So, in theory, user > could still use us-west-1 STS to generate credentials to use for S3 data in > us-east-1 and it would've been ok - but this proposal does not allow for > this. > > Best, > Adnan Hemani > > On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org> > wrote: > > > Hi All, > > > > I propose to add an `endpoint` optional parameter to AwsStorageConfigInfo > > in PR [1913]. > > > > The main idea is to support non-AWS S3 implementations for [1530]. > > > > Existing classes related to supporting S3 Polaris are coded to the AWS > SDK, > > which supports setting STS endpoints. Therefore, it seems natural to > allow > > users to optionally define a specific endpoint for their catalogs. > > > > This change is backward-compatible with existing clients and deployed > > catalogs. > > > > When an endpoint is defined it will be used for both STS and S3 requests > > inside Polaris and will be used to populate the "s3.endpoint" properties > in > > REST Catalog clients. > > > > Thoughts? > > > > [1530] https://github.com/apache/polaris/issues/1530 > > [1913] https://github.com/apache/polaris/pull/1913 > > > > Thanks, > > Dmitri. > > >
