> Two endpoints are necessary in cases when the server's view of the network is different from the engine's view.
That could be true -- but at present it seems like a bit of a contrived scenario. It's also not exclusive to any particular cloud right? On Fri, Aug 1, 2025 at 7:47 AM Dmitri Bourlatchkov <dmitri.bourlatch...@dremio.com.invalid> wrote: > Hi Yufei, > > 1: I do not really know. This is a question about a specific deployment > environment. > > 2: I'm not sure I understand your question. Two endpoints are necessary in > cases when the server's view of the network is different from the engine's > view. > > Cheers, > Dmitri. > > > > On Thu, Jul 31, 2025 at 5:59 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > Thanks for the explanation. Two questions: > > 1. Should the public endpoint used by engines still work with Polaris > even > > if it co-locates with MinIO server? > > 2. Can we set Polaris endpoint directly to the internal address in that > > case? Another way to ask this question is that why do we need to keep > both > > endpoints in the Polaris server? > > > > Yufei > > > > > > On Thu, Jul 31, 2025 at 11:06 AM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > > > > > Hi Yufei, > > > > > > The "how" in your question depends on the deployment environment, I > > guess. > > > There are a lot of variants. > > > > > > If you wonder whether such a situation is possible in practice, I > > > believe it is. An example would be self-hosting non-AWS S3 storage and > > > Polaris in a way that Polaris connections go through a certain internal > > > network, while connections from query engines running outside of that > > > deployment environment go through a different network. This is very > > > high-level, of course, since the deployment choices are largely driven > by > > > specific users' needs. The proposed "endpointInternal" config entry > > merely > > > expands deployment options that users can choose from. > > > > > > Cheers, > > > Dmitri. > > > > > > On Thu, Jul 31, 2025 at 1:07 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > > > Hi Dimtri, > > > > > > > > That generally makes sense to me. For awareness, could you elaborate > a > > > bit > > > > on how the Polaris server and query engines (like Spark, Trino, etc.) > > > might > > > > access the same object storage (e.g., MinIO) via different DNS > > endpoints? > > > > > > > > Yufei > > > > > > > > > > > > On Thu, Jul 31, 2025 at 4:36 AM Alexandre Dutra <adu...@apache.org> > > > wrote: > > > > > > > > > Hi Dmitri, > > > > > > > > > > I think your suggestion makes sense. We added something similar in > > > > > Nessie long ago, and it is definitely useful. > > > > > > > > > > I left some comments in the PR. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > On Thu, Jul 31, 2025 at 4:12 AM Dmitri Bourlatchkov < > > di...@apache.org> > > > > > wrote: > > > > > > > > > > > > Hi All, > > > > > > > > > > > > I propose to add an `endpointInternal` optional parameter to > > > > > > AwsStorageConfigInfo > > > > > > in PR [2213]. > > > > > > > > > > > > The main idea is to support deployment edge cases where Polaris > > > Servers > > > > > may > > > > > > 'see' storage under a different DNS name than query engines. This > > use > > > > > case > > > > > > applies mostly to non-AWS S3 storage (e.g. MinIO). > > > > > > > > > > > > This change is backward-compatible with existing clients and > > deployed > > > > > > catalogs. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > [2213] https://github.com/apache/polaris/pull/2213 > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > > > >
