Hi Alex, thank you again for driving this proposal. We have a real use case for "Detached" principals, so I took the liberty to write up this RFC[1] with a proposal to introduce an "External" Principal type, with a draft implementation plan based on your idea.
Here's the draft PR based on this RFC[2] I'd appreciate a review from anyone who's interested in this topic! Cheers, Sung [1] https://docs.google.com/document/d/1VSoN1-QsAJGaM40oWTxeLIoYTp-XlfQM9nG-jqfb_-E/edit?usp=sharing [2] https://github.com/apache/polaris/pull/3250 On 2025/11/13 13:00:12 Alexandre Dutra wrote: > Hi all, > > Thanks for the excellent discussion so far! I added this topic to the > agenda for today's meeting, in order to get some alignment before > diving into a design doc or formal proposal. I hope that's OK for > everybody. > > Thanks, > Alex > > On Sun, Nov 9, 2025 at 6:54 AM Yufei Gu <[email protected]> wrote: > > > > Thanks for the great discussion. +1 on clarifying the boundary and mutually > > exclusive modes, so that there is no confusion and chaos. > > > > Can we have a short design doc to facilitate further discussion? > > Particularly. it'd be nice to clarify authN and authZ modes Polaris > > supports like the following table: > > > > Mode Principal/Role persistence Privilege enforcement Who decides? > > *Native* Persisted Polaris RBAC Polaris > > *Federated* Persisted/synced Polaris RBAC Polaris > > *Detached* Ephemeral External PDP (e.g. OPA, Ranger) External > > > > Once we clarified different modes, we could start to figure out the > > implications and correspondent changes. Things we might consider: > > 1. Do we allow mixed mode, e.g., some operations are with detached mode, > > others are with native mode? I guess this is pretty complex, we may not do > > that if there are no strong use cases. > > 2. How do we handle the audit in case of detached mode? Can we still get > > the consistent principal names? > > > > I'm happy to jump on a dedicated call for this. > > > > Yufei > > > > > > On Sat, Nov 8, 2025 at 7:20 PM Sung Yun <[email protected]> wrote: > > > > > Thank you for suggesting this Alex, this is a great discussion. > > > > > > I see the value of persisted federated roles when Polaris is the primary > > > place where catalog privileges are managed and external IdPs just supply > > > identity and groups. In that model, stable role entities in Polaris are > > > useful and provide a clear governance boundary. > > > > > > For the OPA-backed PolarisAuthorizer use case, though, we are moving > > > toward a setup where: > > > - Principals are not persisted in Polaris. > > > - Role and attribute evaluation happen in an external Policy Decision > > > Point such as OPA. > > > - Polaris simply enforces authorization decisions on its REST resources. > > > > > > In this pattern, persisted federated roles become less compelling and can > > > even overlap with external policy configuration. So this “detached > > > principal” concept resonates with our use case… it makes the externalized > > > responsibility model explicit instead of overloading federated roles to > > > serve both the Polaris-managed RBAC and externally-managed Policy Decision > > > Point use cases. > > > > > > So IMHO, it might be worth clarifying the model boundaries so both modes > > > are first-class: > > > - “Federated” for external identity with locally managed privileges in > > > Polaris. > > > - “Detached” for external identity with externally managed authorization > > > and no local persistence. > > > > > > That distinction could help keep the semantics clean while still > > > minimizing conceptual sprawl. > > > > > > If it would be helpful, would it make sense to organize a call to discuss > > > these concepts for those that are interested? (The usual Polaris Sync time > > > slot doesn’t work for me unfortunately, and I’d love to make a case for > > > this principal type to be supported) > > > > > > Cheers, > > > Sung > > > > > > On 2025/11/08 01:09:34 Michael Collado wrote: > > > > I see the distinction you're raising. It's a good point, though I'm > > > > hesitant to introduce a third principal type to the code base. The most > > > > notable thing about federated principals and roles is that role grants > > > are > > > > never managed through the Polaris API. This would be true even for > > > > "detached" principals/roles. Federated principals (in the current code > > > > base) also can't be persisted (except possibly through some alternative > > > > process, such as SCIM). The main difference is that federated roles are > > > > expected to be persisted, but "detached" roles are not. Can we unify > > > these > > > > concepts so that we can support both ephemeral roles and persisted roles > > > > without introducing a third type of role? > > > > > > > > Mike > > > > > > > > On Wed, Nov 5, 2025 at 8:35 AM Alexandre Dutra <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > > The code change in #1353 prohibited federated principals from being > > > > > persisted at all. > > > > > > > > > > Indeed! But I think it's important to distinguish between "federated" > > > > > and "detached". Here's how I see the difference: > > > > > > > > > > 1) Federated Principals and Roles > > > > > > > > > > This approach involves pre-synchronizing principals and roles with an > > > > > external system before they can access Polaris. This concept was > > > > > outlined in an older design document [1] but never fully implemented. > > > > > Typically, this would be achieved using SCIM (eventually) or a > > > > > periodic synchronization script run by an administrator. > > > > > > > > > > Re: federated principals persistence. PR 1353 [2] assumes > > > > > just-in-time (JIT) creation of federated principals, meaning the > > > > > principal would be materialized only upon first access. That's not the > > > > > case today though. I personally wonder if it would be simpler to also > > > > > allow for principal synchronization via the management API. (I'm happy > > > > > to resume this conversation later, but that's not the topic here.) > > > > > > > > > > 2) Detached Principals and Roles > > > > > > > > > > This scenario implies that no identity management information > > > > > whatsoever is stored in Polaris. This fits use cases where the > > > > > authorizer does not rely on Polaris's default privilege model: the OPA > > > > > authorizer is one example of that. This is the specific use case > > > > > currently under discussion. > > > > > > > > > > Both flavors of principals are important for Polaris, and I would > > > > > gladly resume the work on federated ones. But, it seems to me that > > > > > enabling detached principals is even more urgent today. WDYT? > > > > > > > > > > > I'd like this lookup to be covered by a feature flag [...] it would > > > be > > > > > nice to allow skipping the unnecessary database hit. WDYT? > > > > > > > > > > Yes, that is a very good suggestion! > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > [1]: > > > > > > > > https://docs.google.com/document/d/15_3ZiRB6Lhzw0nxij341QUdxEIyFGTrI9_18bFIyJVo/edit?tab=t.0#heading=h.cu1a1acu4lc5 > > > > > [2]: https://github.com/apache/polaris/pull/1353 > > > > > > > > > > On Tue, Nov 4, 2025 at 10:05 PM Michael Collado < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > > The code change in #1353 prohibited federated principals from being > > > > > > persisted at all. Federated Principal Roles are persisted, but at > > > this > > > > > time > > > > > > (AFAIK), we can't persist federated principals. > > > > > > > > > > > > Mike > > > > > > > > > > > > On Tue, Nov 4, 2025 at 9:28 AM Dmitri Bourlatchkov <[email protected] > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > This proposal sounds very reasonable to me. > > > > > > > > > > > > > > I believe we had some recent user interest (in slack) for running > > > > > Polaris > > > > > > > with Keycloak. From my POV, implementing your proposal should > > > > > > > significantly simplify that user's deployment. > > > > > > > > > > > > > > As for item (3b), I'd like this lookup to be covered by a feature > > > flag. > > > > > > > Admin users who do not import / sync Principals know that this > > > lookup > > > > > will > > > > > > > always fail, so it would be nice to allow skipping the unnecessary > > > > > database > > > > > > > hit. WDYT? > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Tue, Nov 4, 2025 at 11:53 AM Alexandre Dutra <[email protected] > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > I'd like to propose an enhancement to Polaris's authentication > > > > > > > > process: the introduction of "detached" (non-persisted) > > > principals. > > > > > > > > > > > > > > > > A detached principal is different from a federated principal. > > > > > > > > The > > > > > > > > former isn't persisted at all, whereas the latter is persisted > > > (with > > > > > > > > federated flag = true), and possibly synchronized with some > > > external > > > > > > > > system (e.g. via SCIM). > > > > > > > > > > > > > > > > Motivation: > > > > > > > > > > > > > > > > Currently, Polaris requires principals to be pre-existing for > > > > > > > > successful authentication – even those marked as "federated". > > > > > > > > The > > > > > > > > default authenticator always performs a lookup by ID or name, > > > > > > > > and > > > > > > > > fails if the principal isn't found. > > > > > > > > > > > > > > > > This presents a challenge for users who utilize external > > > > > > > > authentication and authorization systems, such as Keycloak + > > > > > > > > OPA, > > > > > > > > where Polaris acts as a passthrough between authn and authz. > > > > > > > > > > > > > > > > In such scenarios, principals still need to be created in > > > > > > > > advance > > > > > > > > within Polaris, which doesn't make much sense. > > > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > 1) Introduce a new property: PolarisCredential.isExternal(). > > > > > > > > > > > > > > > > 2) During authentication, set isExternal=false on the > > > > > > > > PolarisCredential if Polaris is the IDP (internal auth); > > > otherwise, > > > > > > > > set isExternal=true. > > > > > > > > > > > > > > > > 3) In DefaultAuthenticator: > > > > > > > > > > > > > > > > a) If PolarisCredential.isExternal() is false, continue with > > > the > > > > > > > > existing logic (lookups by ID and name). > > > > > > > > > > > > > > > > b) If PolarisCredential.isExternal() is true, perform the > > > usual > > > > > > > > lookups. If they succeed, continue with the existing logic. If > > > they > > > > > > > > fail though, don't fail the authentication. Instead, create the > > > > > > > > principal and assign roles directly from the token. > > > > > > > > > > > > > > > > 4) In the Resolver, trust the PolarisPrincipal and avoid > > > re-resolving > > > > > > > > the principal and its roles (as this would fail for detached > > > > > > > > principals). > > > > > > > > > > > > > > > > I believe this change will not impact internal authentication > > > with > > > > > the > > > > > > > > default authorizer. It also won't impact people using external > > > IDPs > > > > > > > > with pre-synchronized federated principals: the default > > > authorizer > > > > > > > > would work just fine. > > > > > > > > > > > > > > > > However, for external IDPs combined with custom authorizers, > > > > > > > > this > > > > > > > > change would significantly improve the user experience by > > > eliminating > > > > > > > > the need to persist the principal and its roles within Polaris. > > > > > > > > > > > > > > > > What do you all think? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > >
