Hi all, I have recently submitted a PR to enable support for external principals, that is, principals that are not backed by an entity in the Polaris metastore:
https://github.com/apache/polaris/pull/5119 This implementation finalizes the concepts originally introduced in this design document [1], which we have previously discussed during our Polaris sync meetings. The key benefit of this change is the elimination of an expensive database call that is currently triggered for every request, which is especially impactful since not all metastores use a caching layer. It also brings another considerable benefit: not having to synchronize principals externally anymore. Please take a look. Essentially, when a realm uses external auth and is further configured in "external principal" mode, the system simply bypasses the metastore lookup for the principal entity. You might wonder why we don't eliminate this lookup entirely, considering that the Resolver is its only true consumer and ends up re-resolving the entity anyway. However, removing it completely would fundamentally reshape our internal authentication flow. I believe it is safer to keep the internal authentication flow intact while offering the flexibility to skip this costly overhead when external IDPs are used. You may also wonder why not conflate external auth with external principals and eliminate the metastore lookup whenever external auth is used: that would be a clean approach, but would also introduce a big behavioral change. I think it is safer to let users decide if they want to keep the current behavior (lookup on) or the new one (lookup off). In summary: 1) for realms using internal auth: this change doesn't apply at all (nothing changes) 2) for realms using external auth: nothing changes by default, but the new mode allows skipping metastore lookup and makes principal synchronization unnecessary. 3) for realms using mixed auth: - If the token is issued by Polaris (internal auth), see #1 - Otherwise, see #2 I look forward to hearing your thoughts and feedback on this approach. Thanks, Alex [1]: https://docs.google.com/document/d/1VSoN1-QsAJGaM40oWTxeLIoYTp-XlfQM9nG-jqfb_-E/edit
