> It is only relevant for on-prem S3-compatible storage. I imagine, "endpointInternal" will never be needed for AWS storage.
That's not true, is it? If we are imagining scenarios where the client and the server are on totally different networks, the regular endpoint could indeed need to be addressed differently on the two networks. On Sat, Aug 2, 2025 at 8:17 AM Dmitri Bourlatchkov <di...@apache.org> wrote: > Hi Yufei, > > > I think Polaris server will only need the internal endpoint in that case, > > while engines could use the public endpoint. Do we need to configure both > > for the Polaris server > > Polaris puts "s3.endpoint" into loadTable responses when credential vending > is enabled. > So, yes, both settings are needed, but only in complex deployment cases. > > If the same endpoint works for both the engine and the Polaris Server, it > is only necessary > to set one "endpoint" parameter. > > Cheers, > Dmitri. > > > > On Fri, Aug 1, 2025 at 1:33 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > > > 1: I do not really know. This is a question about a specific deployment > > > environment. > > > > > If the endpoint used by engines could be also used by the Polaris server, > > we should just use it, instead of configuring another one. > > > > > > > 2: I'm not sure I understand your question. Two endpoints are necessary > > in > > > cases when the server's view of the network is different from the > > engine's > > > view. > > > > > I think Polaris server will only need the internal endpoint in that case, > > while engines could use the public endpoint. Do we need to configure both > > for the Polaris server? > > > > > > > Cheers, > > > Dmitri. > > > > > > > > > > > > On Thu, Jul 31, 2025 at 5:59 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > > > Thanks for the explanation. Two questions: > > > > 1. Should the public endpoint used by engines still work with Polaris > > > even > > > > if it co-locates with MinIO server? > > > > 2. Can we set Polaris endpoint directly to the internal address in > that > > > > case? Another way to ask this question is that why do we need to keep > > > both > > > > endpoints in the Polaris server? > > > > > > > > Yufei > > > > > > > > > > > > On Thu, Jul 31, 2025 at 11:06 AM Dmitri Bourlatchkov < > di...@apache.org > > > > > > > wrote: > > > > > > > > > Hi Yufei, > > > > > > > > > > The "how" in your question depends on the deployment environment, I > > > > guess. > > > > > There are a lot of variants. > > > > > > > > > > If you wonder whether such a situation is possible in practice, I > > > > > believe it is. An example would be self-hosting non-AWS S3 storage > > and > > > > > Polaris in a way that Polaris connections go through a certain > > internal > > > > > network, while connections from query engines running outside of > that > > > > > deployment environment go through a different network. This is very > > > > > high-level, of course, since the deployment choices are largely > > driven > > > by > > > > > specific users' needs. The proposed "endpointInternal" config entry > > > > merely > > > > > expands deployment options that users can choose from. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Thu, Jul 31, 2025 at 1:07 PM Yufei Gu <flyrain...@gmail.com> > > wrote: > > > > > > > > > > > Hi Dimtri, > > > > > > > > > > > > That generally makes sense to me. For awareness, could you > > elaborate > > > a > > > > > bit > > > > > > on how the Polaris server and query engines (like Spark, Trino, > > etc.) > > > > > might > > > > > > access the same object storage (e.g., MinIO) via different DNS > > > > endpoints? > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 4:36 AM Alexandre Dutra < > adu...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > I think your suggestion makes sense. We added something similar > > in > > > > > > > Nessie long ago, and it is definitely useful. > > > > > > > > > > > > > > I left some comments in the PR. > > > > > > > > > > > > > > Thanks, > > > > > > > Alex > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 4:12 AM Dmitri Bourlatchkov < > > > > di...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > I propose to add an `endpointInternal` optional parameter to > > > > > > > > AwsStorageConfigInfo > > > > > > > > in PR [2213]. > > > > > > > > > > > > > > > > The main idea is to support deployment edge cases where > Polaris > > > > > Servers > > > > > > > may > > > > > > > > 'see' storage under a different DNS name than query engines. > > This > > > > use > > > > > > > case > > > > > > > > applies mostly to non-AWS S3 storage (e.g. MinIO). > > > > > > > > > > > > > > > > This change is backward-compatible with existing clients and > > > > deployed > > > > > > > > catalogs. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > [2213] https://github.com/apache/polaris/pull/2213 > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
