A lot of the concerns seem to arise from the fact that Polaris essentially has two distinct deployment modes:
1. Provide a highly isolated multi-tenant (multi-realm) Iceberg REST service with a tightly curated set of secure integrations to external dependencies 2. Provide a turnkey Iceberg REST service that is highly configurable as a common tool in the long tail of different users' journey from legacy Iceberg deployments to a more modern Iceberg REST architecture Notably, even if "multi-tenancy/multi-realm" is one feature for Polaris, and "HadoopCatalog federation" is another feature, the two features don't *necessarily* need to simultaneously be enabled in a single deployment. At a high level, Polaris has always historically intended precisely to address the overall problem of allowing all Iceberg users to modernize/migrate/consolidate with a common implementation of Iceberg REST, and this may inherently mean providing a spectrum of deployment modes than then ultimately work cleanly against each other. let Polaris federate to another IRC that is tailor made for HMS On this topic, there hasn't yet been a single standard go-to "IRC adapter" layer within the ecosystem for users to slap in front of HMS or a legacy HadoopCatalog-based setup, and even before we formally started on Polaris Federation support, one Polaris user adapted Polaris specifically to be the convenient IRC wrapper layer in front of their HMS: https://docs.google.com/document/d/17GPe_qawoVlEJe3qwGKgMJlx2CYU97-Jz_npTy_b8_k/edit?tab=t.0#heading=h.nuhm7bltkglt Pooja makes a good point that there's a lot of shared functionality that Polaris provides which users seem to want to apply on top of their HMS. Over time, the winds seem to be blowing towards even more heavyweight functionality that a "wrapper" IRC would need to provide, which also happens to align well with portraying such a wrapper as federation in itself. Overall, the responsibility of the Catalog Federation layer *or* whatever we call a layer that wraps an HMS to provide IRC APIs goes beyond a simple "passthrough and provide a REST response". For example, this doc I just sent out about next-steps for credential-vending/table-level-RBAC features in Polaris Federation carve out the most formal structure of Polaris serving as a high-level delegator+decorator on top of the remote catalog: https://docs.google.com/document/d/19Vm0Uy-EyEYtgd2-bZpPICODOB3rYQJvHeqL1ODOOLc/edit?tab=t.0 Delegation and decoration features (including not-yet-implemented features) may include: - Polaris RBAC - Credential-vending - Standardized retrieval of TableMetadata from ObjectStore if the remote catalog only provides a metadata filename - Freshness-aware tableLoad - Trimming "snapshots=refs" - TableMetadata caching - Access auditing - Request throttling To this end, it's highly desirable to be able to reuse a core Polaris implementation of such features to provide *standard* functionality for the "local IRC wrapper" use case, while also ensuring that we can depend on Polaris <-> Polaris REST-based federation to then pull that wrapper catalog into any other consolidated Polaris catalog, including managed-service versions of Polaris. Maybe we can divide the overall debate into two distinct topics: 1. Does Apache Polaris as a *project* intend to provide a solution to the use case of needing a "local IRC wrapper" to slap in front of HMS with all the listed features? 1. If "no" then what would be the guidance to users? Should we nominate a canonical wrapper impl, such as Apache Gravitino and then add compatibility functionality for Polaris features to the other project and/or provide another adapter layer? 2. Alternatively, do we view this use case as being an "extensions" type of project where we might have another repo under the umbrella of Apache Polaris, but just living in a standalone codebase? 2. On a *technical* level, what are the pros/cons of portraying such functionality as a subset of "federation", and are there concerns about cross-contamination of dependency packages, code maintenance, etc? 1. If we're just concerned with mutual incompatibility between these "IMPLICIT_AUTH" federation types against "multi-tenant" Polaris deployments, can we solve this concern just by structuring feature configuration settings to portray this mutual exclusion cleanly? 2. If we just don't want to make HMS and HadoopCatalog be "first-class" enumerated federation ConnectionConfig types, we could in theory relegate these to being runtime classloaded impls, and we'd just need a single catch-all ConnectionConfig type DYNAMIC_CLASSLOADED or something similar 3. We could provide a separate compilation unit for the right wrappers/factories to inject the HMS federation functionality via dynamic classloading at the service-configuration level (thus making it distinct from the "multi-tenant Polaris" scenario) 4. This would even work if we really wanted to separate out HMS federation to even another github repo -- we'd just provide the DYNAMIC_CLASSLOADED hook in core Polaris, and then the long-tail of catalog impls can be injected without changes in the core Polaris project 5. Is our complaint specific to HMS/Hadoop catalogs, or would we always veto *all* non-REST catalog implementations for consideration as first-class federation types? 1. For example, the native AWS GlueCatalog (without using Glue IRC) is also a version of a "metadata-filename-only" loadTable API, which we might also decide to relegate to DYNAMIC_CLASSLOADED injection, but the GlueCatalog client class is also *capable* of being isolated well enough for a multi-tenancy deployment, so could be worth allowing as a first-class federation type On Tue, Jul 15, 2025 at 3:28 PM Pooja Nilangekar <po...@apache.org> wrote: > Sure, I will add it to the community sync agrenda. > > My concern here is that the other IRC will still need to re-implement > significant portions of Polaris logic, some important blocks of the > top of the list: Auth token management, entity persistence, principals > management, entity access API; which in itself is a non-trivial effort. > > I'm suggesting adding "excludes" while including the iceberg hive client > package, if that helps alleviate your concerns. Secondly, nowhere does > the v1 (or any future version) suggest that we add Kerberos to Polaris. > We are simply providing an experimental pathway to allow accessing > Kerberized HMS instances via the "IMPLICIT" auth support. > > Yes, I agree that it is a complex effort and we can discuss implementation > tradeoffs. I am hoping we are aligned to the premise of supporting > existing HMS users and providing them a pathway to migrate to Polaris > in the long run. > > Thanks, > Pooja > > On 2025/07/15 09:44:10 Robert Stupp wrote: > > I think this deserves a discussion in the next community sync, feel > > free to add it to the agenda. > > > > From what I read in this thread: nobody is against the idea of being > > able to federate to HMS. > > > > However, technical concerns should really be considered, and I think > > that it is a much safer option for the evolution and maintainability > > of Polaris to not add anything HMS or even Kerberos to Polaris itself, > > but let Polaris federate to another IRC that is tailor made for HMS. I > > am not convinced that the challenges that I mentioned can "just be > > worked around". What's your proposal for this? > > > > It's overall a complex effort, so I think this deserves exploring all > > possible options considering the pros and cons of each. > > > > On Mon, Jul 14, 2025 at 8:42 PM Pooja Nilangekar <po...@apache.org> > wrote: > > > > > > Hi everyone, > > > > > > I believe Hive Metastore (HMS) federation is a critical feature for > the success and widespread > > > adoption of the project. The reality of the current data lake > ecosystem is that a vast number of > > > organizations have significant footprint in HMS. For Polaris to be the > Iceberg Rest Catalog (IRC) > > > of choice, it needs to meet users where they are. Ignoring the > existing HMS landscape creates a > > > significant barrier to entry for a large portion of our potential user > base. We have already seen > > > requests for this functionality from the community. For instance, > there has been discussion in the > > > Polaris Slack ( > https://apache-polaris.slack.com/archives/C084QSKD6S2/p1744998865663079) > > > about the need for HMS integration. > > > > > > Furthermore, other open-source catalog projects are already moving in > this direction. Apache > > > Gravitino has this capability, and Unity Catalog OSS is expected to > support HMS federation in > > > its upcoming 0.4 release. The transition from a traditional HMS to a > modern IRC is rarely an > > > overnight flip. Users will likely operate in a hybrid mode for a > considerable period, with both > > > HMS and an IRC coexisting. Thus, a one-time migration tool is not > sufficient. By supporting > > > federation, we will provide a practical path for users to gradually > migrate to Polaris. > > > > > > I understand there may be concerns about pulling in extensive > dependencies from the HMS > > > client. However, these challenges can be managed. We can be selective > about the > > > dependencies we include and explore ways to minimize the impact on the > core Polaris project. > > > The strategic benefit of enabling a smooth migration path for a huge > segment far outweighs the > > > technical challenge of managing these dependencies. > > > > > > With regards to the concern about a single HMS support, we can add > LDAP support in the next iteration, which will enable Polaris to federate > to multiple HMS instances. Essentially, Polaris will > > > federate to multiple HMS instances, as long as the installations use > LDAP authentication. The > > > existing IMPLICIT auth will enable (experimental/limited) Kerberos > support without pulling in any > > > Kerberos dependencies into the Polaris project. > > > > > > I welcome a discussion on the best way to implement this. I strongly > believe that deciding > > > against HMS support will prevent a large subset data lake users from > adopting Polaris. I'm > > > hoping we can find a better path forward together. > > > > > > Thanks, > > > Pooja > > > > > > On 2025/07/10 23:47:21 Yufei Gu wrote: > > > > It’s a valid point that Polaris needs to support multi-tenancy, or > even > > > > across different external catalogs (such as remote HMS) within a > single > > > > realm. > > > > > > > > Unfortunately, Kerberos isn’t compatible with this model, as it > requires > > > > global configuration per JVM, making it inherently single-tenant. So > I’d > > > > suggest we rule out the Kerberos option and explore more flexible > > > > authentication schemes. > > > > > > > > Here’s a quick summary of viable alternatives: > > > > > > > > - Implicit Authentication: This can not support multi-tenancy when > > > > environment variables are leveraged per instance. We could > further extend > > > > this in the future by integrating a more sophisticated secrets > manager to > > > > improve tenant isolation and credential handling. > > > > > > > > > > > > - LDAP: A well-established solution that naturally supports > multiple > > > > credentials and user contexts. It aligns well with multi-tenant > needs. > > > > > > > > Given that, I think we have a clear path forward for enabling > multi-tenancy > > > > with HMS federation. Introducing implicit authentication as a > starting > > > > point seems reasonable. It can be disabled by default, with Polaris > admins > > > > choosing to enable it based on their environment. Even in single-HMS > > > > deployments, this option brings real value without adding > complexity, esp. > > > > A lot of organizations only have one HMS instance. > > > > > > > > Yufei > > > > > > > > > > > > On Wed, Jul 9, 2025 at 7:44 AM Robert Stupp <sn...@snazy.de> wrote: > > > > > > > > > Following up on my email: > > > > > > > > > > Polaris would really benefit from supporting HMS and other catalog > > > > > types. And the way I see to get there is to have a "HMS only" IRC > > > > > service, which can be legibly built on Java 11, use Kerberos, etc. > > > > > Polaris can then federate to that HMS catalog. > > > > > AFAIU clients can authenticate to k8s and get OAuth tokens. Those > can > > > > > be used to talk to Polaris, which can in turn talk to the HMS > service. > > > > > > > > > > What I do object though is making Polaris effectively a single > realm + > > > > > single catalog service and add new dependencies to Hadoop + Hive + > > > > > Kerberos to Polaris. > > > > > > > > > > On Wed, Jul 9, 2025 at 12:17 PM Robert Stupp <sn...@snazy.de> > wrote: > > > > > > > > > > > > Let's recap what Polaris offers: > > > > > > 1. Multi tenancy via realms > > > > > > 2. Multiple catalogs per realm > > > > > > 3. OAuth/OIDC > > > > > > > > > > > > Adding Kerberos is global per JVM, making #1 impossible and > likely > > > > > > also not suitable for #2, plus adding another complicated and > complex > > > > > > auth mechanism. > > > > > > If Kerberos is a strong concern, I propose to contribute > necessary > > > > > > changes to the "Iceberg auth manager project" [1] to let clients > use > > > > > > krb and receive OAuth tokens for it.It is also worth mentioning > that > > > > > > testing all that (development and CI including unit and > especially > > > > > > integration tests) is a huge effort in itself. > > > > > > > > > > > > Again, federating to another "single tenant / single catalog HMS > krb" > > > > > > Iceberg REST service behind Polaris is fine. Krb clients can > authorize > > > > > > against Polaris via OAuth, and likely can Polaris itself > authorize > > > > > > itself using OAuth. > > > > > > > > > > > > I strongly object to depending even more on Hadoop for the > reasons > > > > > > outlined earlier. I also strongly object to adding Kerberos to > > > > > > Polaris. > > > > > > > > > > > > BTW: Hadoop is not necessary for Iceberg to work, it is rather > an "opt > > > > > > in" (ex: org.apache.iceberg.hadoop.Configurable#setConf). > > > > > > > > > > > > [1] https://github.com/dremio/iceberg-auth-manager > > > > > > > > > > > > On Tue, Jul 8, 2025 at 6:25 PM Yufei Gu <flyrain...@gmail.com> > wrote: > > > > > > > > > > > > > > HMS integration is a key step toward one of Polaris’s critical > > > > > missions: > > > > > > > helping users move off HMS. It brings clear value by aligning > with our > > > > > > > long-term direction. > > > > > > > > > > > > > > I’m not too concerned about hive.xml, most of its > configurations can be > > > > > > > dynamically injected at runtime. The real challenge lies in > Kerberos > > > > > > > integration. Since krb5.conf and the keytab are globally > configured per > > > > > > > JVM, a single JVM instance cannot support true multi-tenancy. > As far > > > > > as I > > > > > > > know, there isn’t a clean solution to this limitation. > > > > > > > > > > > > > > If that's indeed the case, Option 2a becomes far less > appealing to me. > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 7, 2025 at 11:18 AM Russell Spitzer < > > > > > russell.spit...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > I think having some integration with HMS is definitely a > good idea. > > > > > We've > > > > > > > > already seen > > > > > > > > users build this in the wild on top of Polaris showing that > there is > > > > > > > > definitely a demand. > > > > > > > > I'm still a strong believer that we should be helping users > get to > > > > > Polaris > > > > > > > > from whatever systems > > > > > > > > they are currently using to Polaris. > > > > > > > > > > > > > > > > On Mon, Jul 7, 2025 at 12:59 PM Eric Maynard < > > > > > eric.w.mayn...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > 1. We (Polaris) can provide end users a way to migrate off > of these > > > > > > > > > catalogs that the Iceberg project no longer wants to > invest into. > > > > > > > > > Implementing HMS federation is in service to the goal of > removing > > > > > > > > > non-Iceberg catalogs, not in contradiction to it. > > > > > > > > > > > > > > > > > > 2. This does not seem like a user-centered concern, but > I'm also > > > > > not > > > > > > > > sure I > > > > > > > > > understand exactly what is being expressed here. Are you > saying > > > > > that the > > > > > > > > > current HADOOP federation does not work somehow? > > > > > > > > > > > > > > > > > > 3. Yes, please see the other thread about the IMPLICIT > > > > > authentication > > > > > > > > type > > > > > > > > > for discussion of this topic. Note, however, that HMS > federation > > > > > may > > > > > > > > > support authentication types other than IMPLICIT. > > > > > > > > > > > > > > > > > > 4. That depends on what you mean by "depends on" -- it > could also > > > > > be said > > > > > > > > > that Iceberg itself depends on Hadoop. > > > > > > > > > > > > > > > > > > 5. This not only also applies to HADOOP federation, which > already > > > > > exists, > > > > > > > > > but also does *not* apply to HMS federation when using an > > > > > authentication > > > > > > > > > mechanism other than IMPLICIT -- again, please see the > other > > > > > thread for > > > > > > > > > more discussion of this topic. > > > > > > > > > > > > > > > > > > On Fri, Jul 4, 2025 at 3:52 AM Robert Stupp < > sn...@snazy.de> > > > > > wrote: > > > > > > > > > > > > > > > > > > > I'd really prefer to not add "anything Hive" to Polaris > itself, > > > > > and I'd > > > > > > > > > > really like to see Hadoop being removed entirely from the > > > > > Polaris code > > > > > > > > > > base. > > > > > > > > > > > > > > > > > > > > There are multiple reasons for this: > > > > > > > > > > > > > > > > > > > > 1. The Iceberg project would rather like to remove all > catalogs > > > > > except > > > > > > > > > > the REST catalog. (That's at least what I understood from > > > > > discussions > > > > > > > > > > quite a while ago.) > > > > > > > > > > > > > > > > > > > > 2. Hadoop is quite behind supporting recent Java > versions. It is > > > > > > > > already > > > > > > > > > > impossible to run "anything Hadoop" with Java 24. > Considering > > > > > how long > > > > > > > > > > it took Hadoop to even support Java 11, it will take a > long time > > > > > until > > > > > > > > > > Hadoop is ready for Java 24+, especially since Hadoop > has to > > > > > refactor a > > > > > > > > > > lot of things. Polaris requires Java 21 and we know it > works in > > > > > CI with > > > > > > > > > > Java 22+23 (both are EOL). Hadoop does only support Java > 11, not > > > > > 17, > > > > > > > > not > > > > > > > > > > 21. > > > > > > > > > > > > > > > > > > > > 3. Hadoop (HDFS) is as a very different security model, > which is > > > > > the > > > > > > > > > > reason why HDFS is not suitable for Polaris production > > > > > configuration, > > > > > > > > > > guarded by explicit configuration options. > > > > > > > > > > > > > > > > > > > > 4. Hive depends on Hadoop, so all concerns about Hadoop > also > > > > > apply to > > > > > > > > > Hive. > > > > > > > > > > > > > > > > > > > > 5. Polaris is multi-tenant (realms). A _single_ instance > of Hive > > > > > > > > > > contradicts this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My vote would be on *not* adding Hive and also on > removing Hadoop > > > > > > > > > entirely. > > > > > > > > > > > > > > > > > > > > If someone comes up with an Iceberg REST catalog for > Hive or > > > > > HDFS and > > > > > > > > > > Polaris can connect to it, that's fine for me, because > it's > > > > > outside of > > > > > > > > > > Polaris. But I strongly object having Hadoop or even > Hive in > > > > > Polaris. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 7/1/25 20:48, Pooja Nilangekar wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I wanted to start a discussion around the support for > Hive > > > > > Catalog > > > > > > > > > > > federation in Polaris. In particular, there are two > primary > > > > > ways we > > > > > > > > can > > > > > > > > > > add > > > > > > > > > > > support for Hive federation: > > > > > > > > > > > > > > > > > > > > > > *1. Support a single Hive instance per Polaris > deployment* The > > > > > Hive > > > > > > > > > > > workflow would be identical to the Hadoop catalog > workflow. > > > > > Polaris > > > > > > > > > > > would invoke the Iceberg connection library, that > would try to > > > > > find > > > > > > > > the > > > > > > > > > > > hive-site.xml file in (1) the CLASSPATH and (2) the > default > > > > > Hadoop > > > > > > > > > > > locations: HADOOP_PATH and HADOOP_CONF_DIR. Polaris > would then > > > > > > > > > initialize > > > > > > > > > > > the Hive connection using the configurations it found > at these > > > > > > > > > locations. > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Drawbacks: *The primary drawback of this approach > is that > > > > > if > > > > > > > > > Polaris > > > > > > > > > > > finds multiple hive-site.xml files, it would merge > their > > > > > > > > > > configurations, > > > > > > > > > > > which could lead to potentially inconsistent > connection > > > > > state. > > > > > > > > > > > Furthermore, there is no clear documentation of > the order > > > > > in > > > > > > > > which > > > > > > > > > > the > > > > > > > > > > > configuration would be applied. While this is often > > > > > predictable > > > > > > > > on > > > > > > > > > a > > > > > > > > > > given > > > > > > > > > > > OS, it is not guaranteed across environments. The > other key > > > > > > > > > drawback > > > > > > > > > > is > > > > > > > > > > > that if a Polaris user wants to federate to > multiple Hive > > > > > > > > catalogs, > > > > > > > > > > their > > > > > > > > > > > only option is to deploy a separate Polaris > instance for > > > > > each > > > > > > > > Hive > > > > > > > > > > > instance. > > > > > > > > > > > > > > > > > > > > > > *2. Support multiple Hive instances per Polaris > deployment* The > > > > > > > > > alternate > > > > > > > > > > > (and in my view, ideal) solution is to allow Polaris to > > > > > federate with > > > > > > > > > > > multiple Hive catalogs. To support multiple catalogs, > Polaris > > > > > would > > > > > > > > > > > explicitly disallow the connection library from reading > > > > > hive-site.xml > > > > > > > > > > files > > > > > > > > > > > in the default paths. To pass in the configurations, > Polaris > > > > > can > > > > > > > > adopt > > > > > > > > > > one > > > > > > > > > > > of two options: > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Option 2a: Accept a canonical path to the target > > > > > hive-site.xml.* > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Advantages:* This guarantees that the > connection > > > > > > > > configurations > > > > > > > > > > are > > > > > > > > > > > derived from a single source. It also allows > Polaris to > > > > > rely > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > > NONE/ENVIRONMENT/PROVIDER/UNMANAGED mechanism, > making it > > > > > > > > > > especially > > > > > > > > > > > useful in case the Hive instance relies on > Kerberos or > > > > > custom > > > > > > > > > > > authentication that Polaris does not natively > > > > > support/manage. > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Drawbacks:* The user needs to have access (or > some > > > > > mechanism > > > > > > > > to > > > > > > > > > > > upload files) to the Polaris server's file > system. > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Option 2b: Accept all the connection-specific > parameters > > > > > as a > > > > > > > > part > > > > > > > > > > of > > > > > > > > > > > the create-catalog request.* > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Advantage:* Polaris can directly accept and > store the > > > > > > > > > > configurations > > > > > > > > > > > in a DPO instead of relying on the user having > access > > > > > to the > > > > > > > > > > > server's file > > > > > > > > > > > system (to create/update hive-site.xml). > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > *Drawback:* Polaris would need to manage the > secrets. > > > > > This is > > > > > > > > > > easy to > > > > > > > > > > > support for certain authentication types > (LDAP/Simple), > > > > > > > > However, > > > > > > > > > > > it would > > > > > > > > > > > preclude the support for other authentication > > > > > mechanisms, such > > > > > > > > > > > as Kerberos > > > > > > > > > > > or Custom. > > > > > > > > > > > > > > > > > > > > > > I prefer option 2a primarily because it provides the > > > > > flexibility of > > > > > > > > > > > supporting multiple federated Hive catalogs while > allowing > > > > > Polaris to > > > > > > > > > > > support authentication that it does not natively > manage. > > > > > Please let > > > > > > > > me > > > > > > > > > > know > > > > > > > > > > > if you have any thoughts or feedback. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Pooja > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Robert Stupp > > > > > > > > > > @snazy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
