Hello all, I have opened a PR for BigQuery Metastore support, https://github.com/apache/polaris/pull/4050 It also adds a BIGQUERY connection type to the REST API. Looking forward to your feedback.
Regards, Joy On Fri, Mar 20, 2026 at 8:35 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Joy, > > Thanks - your plan looks good to me! Looking forward to a PR :) > > Cheers, > Dmitri. > > On Fri, Mar 20, 2026 at 6:58 AM Joy <[email protected]> wrote: > > > Hello all, > > > > Thank you so much for all the feedback and guidance. > > > > I am trying to summarize what I understood from the thread and some Slack > > conversations with JB and Prashant. > > > > *On how to structure the code:* > > > > JB suggested separate modules per catalog, like > > *extensions/federation/bqms*, > > *extensions/federation/glue*. Each module has its own factory class and > > manages its own dependencies, similar to how Hive federation works today > if > > I understand correctly. > > > > Prashant suggested a hybrid, one generic factory that works with > Iceberg's > > *BaseMetastoreCatalog* interface. Separate modules would only bring in > > dependencies, for example, iceberg-bigquery, no custom factory code would > > be needed per catalog. > > > > *On pluggability:* > > > > Dmitri prefers keeping things simple with current Quarkus module > selection. > > > > Romain raised a good point about making it easier for end users who > consume > > Polaris as a Docker image. > > > > But my understanding is limited here. > > > > *My plan:* > > > > I'd like to start with a simple approach since most people seem okay with > > that. I can try an initial implementation following JB's pattern i.e. > > separate module for BQMS or Prashant's pattern i.e. generic factory. > > > > I'd be okay if the first attempt doesn't get accepted, it may help move > the > > discussion forward with some code to look at. > > > > Please let me know if I've misunderstood anything or if there's a > preferred > > direction. > > > > Regards, > > Joy > > > > On Thu, 19 Mar, 2026, 4:52 am Romain Manni-Bucau, <[email protected] > > > > wrote: > > > > > Hi, > > > > > > As an end users I have a small feedback on that topic: > > > > > > 1. it is unlikely we rebuild polaris to get federation (we'd federate > > > poalris to make it clear) - with hive catalog for now > > > 2. hive integration with polaris is not fully aligned with spark which > > has > > > shihms so there is already this classloader or alternative need > > > > > > > > > what can be envisionned if you do not want to go with classloaders is > > real > > > pluggability at *runtime* (like any real CDI application) or worse case > > > using a ServiceLoader (even if weird in a CDI stack since the SPI is > CDI > > > impl itself). > > > > > > so from my small window a valid option is to make polaris runtime > > friendly > > > (vs build friendly which is not a solution for a product) and > optionally > > > enable classloading (just needs to be a tree like > > > > > > > > > https://github.com/Talend/component-runtime/blob/master/container/container-core/src/main/java/org/talend/sdk/component/container/ContainerManager.java > > > ). > > > the classloader support can even be made in a runtime extension which > > does > > > load a configuration to load other extension so you get the best of > both > > > worlds, but point is you cant request users to *build* for a webapp > > > ultimately delivered as a docker image - I know apache is just about > > > sources but nobody consumes it this way (except vendors when they fork > > but > > > hopefully it is a few end users). > > > > > > hope it makes sense > > > > > > Romain Manni-Bucau > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > > <https://dotnetbirdie.github.io/> | Blog < > https://rmannibucau.github.io/ > > > > > > | Old > > > Blog <http://rmannibucau.wordpress.com> | Github > > > <https://github.com/rmannibucau> | LinkedIn > > > <https://www.linkedin.com/in/rmannibucau> | Book > > > < > > > > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > Le mer. 18 mars 2026 à 23:18, Dmitri Bourlatchkov <[email protected]> a > > > écrit : > > > > > > > Hi All, > > > > > > > > Good point about modeling federation implementations similarly to > > > pluggable > > > > persistence implementations. > > > > > > > > I'm not sure about meddling with ClassLoaders in Polaris. I'd prefer > to > > > > leverage the existing Quarkus-based integration of multiple modules > > into > > > a > > > > single runtime env. (current state). > > > > > > > > Downstream project will have the ability to elect which modules to > use > > > > at build time. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Wed, Mar 18, 2026 at 2:42 AM Jean-Baptiste Onofré < > [email protected]> > > > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > "Classloader" is a fair and valid point. I have shared this > > perspective > > > > in > > > > > the past and discussed it with Nandor as well: I believe federation > > > > should > > > > > utilize a "plugin" mechanism, similar to how persistence is > handled. > > > > > > > > > > By adopting this approach, each plugin (such as HMS) is in its own > > > > module, > > > > > and would be responsible for managing its own dependencies and > > shading. > > > > > > > > > > Regards, > > > > > JB > > > > > > > > > > On Wed, Mar 18, 2026 at 12:37 AM Yufei Gu <[email protected]> > > > wrote: > > > > > > > > > > > Federation seems like the strongest candidate given the > dependency > > > > > surface. > > > > > > For example, Trino-style classloader isolation could help when > > > > conflicts > > > > > > arise (e.g., Hadoop or Guava across different federation > > > > > implementations). > > > > > > That said, I’d suggest introducing this only once we actually hit > > > such > > > > > > issues. Until then, keeping things simple with proper > > modularization > > > > may > > > > > be > > > > > > the better tradeoff. WDTY? > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2026 at 2:00 PM Madhan Neethiraj < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > we need to be careful about is proper modularization to > control > > > > > > > dependency scope growth. > > > > > > > > > > > > > > I share this concern; it is applicable for any extension like > > > > > > authorizers. > > > > > > > Having a built-in support for isolation of extension specific > > > > > > dependencies > > > > > > > will significantly reduce/eliminate complexities in handling > > > > potential > > > > > > > library conflicts and avoid impact to Polaris core due to > > > > dependencies > > > > > > > dragged by extensions. Trino's plugin model seems to be a good > > > > > reference > > > > > > - > > > > > > > 1) and 2). > > > > > > > > > > > > > > Thanks, > > > > > > > Madhan > > > > > > > > > > > > > > 1) https://trino.io/docs/current/installation/plugins.html > > > > > > > 3) https://trino.io/docs/current/develop/spi-overview.html > > > > > > > > > > > > > > > > > > > > > On 3/17/26, 1:30 PM, "Prashant Singh via dev" < > > > > [email protected] > > > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > +1 to this. I would also recommend thinking about this from an > > > > > interface > > > > > > > POV to have BaseMetastoreCatalog (from Apache Iceberg) as a way > > to > > > > > > > integrate, with a defined connection object that can feed the > > > > > connection > > > > > > to > > > > > > > BQ or Glue (perhaps a standard key-value prep) > > > > > > > and the dependencies can be added accordingly. > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Prashant Singh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2026 at 12:08 PM Dmitri Bourlatchkov < > > > > [email protected] > > > > > > > <mailto:[email protected]>> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Joy, > > > > > > > > > > > > > > > > I suppose Polaris will be the REST API + AuthZ layer on top > of > > > > > > > > BigQueryMetastoreCatalog, right? How do you envision this? > > > > > > > > > > > > > > > > In general, I think it's going to be a useful contribution. > > > > > > > > > > > > > > > > From my POV, the only technical issue we need to be careful > > about > > > > is > > > > > > > > proper modularization to control dependency scope growth. I > > > imagine > > > > > > some > > > > > > > > downstream projects may want to opt in/out of > > > > > BigQueryMetastoreCatalog > > > > > > > and > > > > > > > > its transitive dependencies at their own discretion. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On 2026/03/17 18:19:34 Joy wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I'm Joy. We use BigQueryMetastoreCatalog for Iceberg on GCP > > and > > > > > want > > > > > > to > > > > > > > > use > > > > > > > > > Polaris as a catalog gateway. I see federation currently > > > supports > > > > > > REST > > > > > > > > and > > > > > > > > > HMS. > > > > > > > > > > > > > > > > > > Would there be interest in supporting other native Iceberg > > > > catalogs > > > > > > > like > > > > > > > > > BQMS or Glue? I've been working on a prototype for BQMS > > > following > > > > > the > > > > > > > HMS > > > > > > > > > pattern and would be happy to contribute it if I'm > > successful. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Joy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
