Thanks for the explanation. Two questions: 1. Should the public endpoint used by engines still work with Polaris even if it co-locates with MinIO server? 2. Can we set Polaris endpoint directly to the internal address in that case? Another way to ask this question is that why do we need to keep both endpoints in the Polaris server?
Yufei On Thu, Jul 31, 2025 at 11:06 AM Dmitri Bourlatchkov <di...@apache.org> wrote: > Hi Yufei, > > The "how" in your question depends on the deployment environment, I guess. > There are a lot of variants. > > If you wonder whether such a situation is possible in practice, I > believe it is. An example would be self-hosting non-AWS S3 storage and > Polaris in a way that Polaris connections go through a certain internal > network, while connections from query engines running outside of that > deployment environment go through a different network. This is very > high-level, of course, since the deployment choices are largely driven by > specific users' needs. The proposed "endpointInternal" config entry merely > expands deployment options that users can choose from. > > Cheers, > Dmitri. > > On Thu, Jul 31, 2025 at 1:07 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > Hi Dimtri, > > > > That generally makes sense to me. For awareness, could you elaborate a > bit > > on how the Polaris server and query engines (like Spark, Trino, etc.) > might > > access the same object storage (e.g., MinIO) via different DNS endpoints? > > > > Yufei > > > > > > On Thu, Jul 31, 2025 at 4:36 AM Alexandre Dutra <adu...@apache.org> > wrote: > > > > > Hi Dmitri, > > > > > > I think your suggestion makes sense. We added something similar in > > > Nessie long ago, and it is definitely useful. > > > > > > I left some comments in the PR. > > > > > > Thanks, > > > Alex > > > > > > On Thu, Jul 31, 2025 at 4:12 AM Dmitri Bourlatchkov <di...@apache.org> > > > wrote: > > > > > > > > Hi All, > > > > > > > > I propose to add an `endpointInternal` optional parameter to > > > > AwsStorageConfigInfo > > > > in PR [2213]. > > > > > > > > The main idea is to support deployment edge cases where Polaris > Servers > > > may > > > > 'see' storage under a different DNS name than query engines. This use > > > case > > > > applies mostly to non-AWS S3 storage (e.g. MinIO). > > > > > > > > This change is backward-compatible with existing clients and deployed > > > > catalogs. > > > > > > > > Thoughts? > > > > > > > > [2213] https://github.com/apache/polaris/pull/2213 > > > > > > > > Thanks, > > > > Dmitri. > > > > > >
