I'm adding the feature based on some prior private experience, so from my POV it is not contrived, although I cannot really go into the details of that experience :)
It is only relevant for on-prem S3-compatible storage. I imagine, "endpointInternal" will never be needed for AWS storage. Cheers, Dmitri. On Fri, Aug 1, 2025 at 1:22 PM Eric Maynard <eric.w.mayn...@gmail.com> wrote: > > Two endpoints are necessary in cases when the server's view of the > network is different from the engine's view. > > That could be true -- but at present it seems like a bit of a contrived > scenario. It's also not exclusive to any particular cloud right? > > On Fri, Aug 1, 2025 at 7:47 AM Dmitri Bourlatchkov > <dmitri.bourlatch...@dremio.com.invalid> wrote: > > > Hi Yufei, > > > > 1: I do not really know. This is a question about a specific deployment > > environment. > > > > 2: I'm not sure I understand your question. Two endpoints are necessary > in > > cases when the server's view of the network is different from the > engine's > > view. > > > > Cheers, > > Dmitri. > > > > > > > > On Thu, Jul 31, 2025 at 5:59 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > Thanks for the explanation. Two questions: > > > 1. Should the public endpoint used by engines still work with Polaris > > even > > > if it co-locates with MinIO server? > > > 2. Can we set Polaris endpoint directly to the internal address in that > > > case? Another way to ask this question is that why do we need to keep > > both > > > endpoints in the Polaris server? > > > > > > Yufei > > > > > > > > > On Thu, Jul 31, 2025 at 11:06 AM Dmitri Bourlatchkov <di...@apache.org > > > > > wrote: > > > > > > > Hi Yufei, > > > > > > > > The "how" in your question depends on the deployment environment, I > > > guess. > > > > There are a lot of variants. > > > > > > > > If you wonder whether such a situation is possible in practice, I > > > > believe it is. An example would be self-hosting non-AWS S3 storage > and > > > > Polaris in a way that Polaris connections go through a certain > internal > > > > network, while connections from query engines running outside of that > > > > deployment environment go through a different network. This is very > > > > high-level, of course, since the deployment choices are largely > driven > > by > > > > specific users' needs. The proposed "endpointInternal" config entry > > > merely > > > > expands deployment options that users can choose from. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Thu, Jul 31, 2025 at 1:07 PM Yufei Gu <flyrain...@gmail.com> > wrote: > > > > > > > > > Hi Dimtri, > > > > > > > > > > That generally makes sense to me. For awareness, could you > elaborate > > a > > > > bit > > > > > on how the Polaris server and query engines (like Spark, Trino, > etc.) > > > > might > > > > > access the same object storage (e.g., MinIO) via different DNS > > > endpoints? > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 4:36 AM Alexandre Dutra <adu...@apache.org > > > > > > wrote: > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > I think your suggestion makes sense. We added something similar > in > > > > > > Nessie long ago, and it is definitely useful. > > > > > > > > > > > > I left some comments in the PR. > > > > > > > > > > > > Thanks, > > > > > > Alex > > > > > > > > > > > > On Thu, Jul 31, 2025 at 4:12 AM Dmitri Bourlatchkov < > > > di...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > I propose to add an `endpointInternal` optional parameter to > > > > > > > AwsStorageConfigInfo > > > > > > > in PR [2213]. > > > > > > > > > > > > > > The main idea is to support deployment edge cases where Polaris > > > > Servers > > > > > > may > > > > > > > 'see' storage under a different DNS name than query engines. > This > > > use > > > > > > case > > > > > > > applies mostly to non-AWS S3 storage (e.g. MinIO). > > > > > > > > > > > > > > This change is backward-compatible with existing clients and > > > deployed > > > > > > > catalogs. > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > [2213] https://github.com/apache/polaris/pull/2213 > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > >
