Hi Prashant, Here's a google copy of the proposal: https://docs.google.com/document/d/1B4K7iQvnRYHBnOrP6pXnw48keflZcBkM5l1-ooAOCZg/edit?tab=t.0. Feel free to comment on it :) The GH issue will remain the canonical version, but I will do my best to keep them in sync.
Regarding cross catalog view reference, i would say its known to have this > limitation and the uber reason is there is no catalog as an entity in IRC > .. in engines such as spark catalog names are aliases, having stuff like > uuid is not suff > (even polaris doesn't guarantee that, uuid pk etc), In the spark > implementation i will fail close, As if the query fails, in trino we fail > only when its definer (i am still working in that), the catalog will always > see the *complete chain* is what we can work as the working assumption > from the catalog POV. > One limitation of the spec is that nowhere does it say if a view is an invoker or a definer view. And I quite agree with this approach: IRC clients/engines should not deal with policy management in general. Polaris itself should probably stay out of it and delegate to the authorizer SPI as well because there are as many models as providers. Being prescriptive about the exact definition of an invoker/definer view means coercing those authorizers to follow those models (vs the IRC spec being flexible to adapt to their existing models). While IRC tries to support views across catalogs via id and mapping, treating it as an engine problem, I see this as something better solved at the catalog level if the catalog could associate a namespace with a remote/federated catalog. This would solve the view name resolution problem, simplify engine configuration (removing the need to configure a full set of catalogs and ensure they are assigned to the correct names for each query engine), and would likely resolve this specific issue, as the authorizer could request an object be resolved under a new security context (with some changes to the SPI). What it would prevent though is circular dependencies between catalogs via view definition, but it's probably a good thing? On Thu, Nov 5, 2026 at 9:48 AM Prashant Singh <[email protected]> wrote: > This is really exciting everyone, thanks Laurent for picking this up ! > In Fact very timely yesterday I merged my final PR to make the > referenced-by send with Apache Iceberg Spark connector [1] > Have been working with the Trino community so far to let them send the > referenced-by to [2] over ~6 months now ! > I feel the client ecosystem has finally caught up and it's the right time > to start discussing the server side implementation, as we await clients to > be released, presently AFAIK there was no existing client which was sending > referenced-by to Polaris ! > > Looking forward to partnering / collaborating in this effort ! > > Laurent, Is it possible to post your issue (I see a lot of context there > and they are very thorough !) to a google doc or a pr, whatever you prefer > so that I can add my feedback / questions inlines ! > > Regarding requirements for a trusted engine, it's required for this to > work E2E and reliably (please check this my comment on the spec PR [3]) and > totally makes sense to align on a common base here which we can use for our > efforts for fine grained access control as well [4] ! > > Regarding cross catalog view reference, i would say its known to have this > limitation and the uber reason is there is no catalog as an entity in IRC > .. in engines such as spark catalog names are aliases, having stuff like > uuid is not suff > (even polaris doesn't guarantee that, uuid pk etc), In the spark > implementation i will fail close, As if the query fails, in trino we fail > only when its definer (i am still working in that), the catalog will always > see the *complete chain* is what we can work as the working assumption > from the catalog POV. > > I would say let's start on these working assumptions ! > > > [1] https://github.com/apache/iceberg/pull/13979/changes > [2] https://github.com/trinodb/trino/pull/31542 > [3] https://github.com/apache/iceberg/pull/13810#issuecomment-3816022525 > [4] https://lists.apache.org/thread.html/hg0sqfpx38yn7s5m1xqq7mk08kkjcnkd > <https://lists.apache.org/thread.html/hg0sqfpx38yn7s5m1xqq7mk08kkjcnkd> > > > Best, > Prashant > > On Thu, Nov 5, 2026 at 8:58 AM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi Laurent, >> >> Good idea about the feature flag! >> I would make it overridable at catalog level (like the other >> polaris.config.* flags), so an operator can disable it for a specific >> catalog. >> When disabled, Polaris should behave like today: no parsing, no >> validation, >> no chain in the events, and an empty referencedBy(). >> I would keep it enabled by default (with strict validation), as soon as >> it's in CHANGELOG/documentation. >> >> About the engine's responsibility (ensuring that all entries in the chain >> pertain to this catalog), I agree: Polaris can't verify anything about >> other catalogs. That's also what the Trino PR does afair: only Iceberg >> views of the same catalog are sent, and a foreign DEFINER view in the >> middle of the chain fails the load by default. It's also what makes the >> duplicate check correct: with entries from several catalogs, two different >> views with the same name would be rejected as a cycle. >> >> My point was only about the javadoc :) the referencedBy() entries are >> names >> asserted by the client, interpreted in the target catalog, and not >> resolved >> by Polaris. An authorizer that needs to check that they exist (or that >> they >> are actually view) has to resolve them in resolveAuthorizationInputs, as >> you described. I just wanted to propose a couple of sentences in the >> javadoc to prevent authorizers from treating the chain as verified >> entities >> ;) >> >> About the cross catalog definer case (viewB definer in catalog 1, viewC in >> catalog 2): I agree it's a limitation if referenced-by itself, not >> something Polaris can solve on its own. Evaluating viewC as the query user >> is the right default imho: worst case is an access denied, not a data >> leak. >> I would document it as a known limitation (we can bring the discussion on >> the Iceberg dev mailing list). >> >> Thanks again for the proposal, and, as I said in my previous email, I >> think >> we can focus on part 1 only for now. >> >> Regards >> JB >> >> On Thu, Nov 5, 2026 at 4:48 PM Laurent Goujon via dev < >> [email protected]> wrote: >> >> > Hello JB, >> > >> > - Today Polaris ignores the parameter, even if the Iceberg client sends >> it >> > > since 1.12. With the PR, a request with a malformed chain in the >> > parameter, >> > > a repeated entry or more than 10 entries gets a 400 (even if the >> > authorizer >> > > ignores the chain). I'm completely ok with "strict" validation but it >> > means >> > > users "might" see it. >> > >> > >> > We could add a configuration flag to ignore the referenced-by chain. >> When >> > the flag is on, the referenced-by chain will not propagate to the SPI >> > though. >> > >> > - The spec identifiers have no catalog part, so the PR maps each entry >> to a >> > > securable in the target catalog. A entry may not exist, or may match >> an >> > > unrelated view with the same name if an engine includes a view from >> > another >> > > catalog. I would state this clearly in the referencedBy() javadoc, so >> > > authorizers don't treat the chain as a list of verified entities. >> > >> > >> > IMHO, it is the engine's responsibility to ensure that all entries in >> the >> > chain pertain to this catalog since we don't have any visibility on >> other >> > catalogs. We are also touching on the limit of referenced-by which does >> not >> > work cross catalog. Let's say the chain is viewA -> view B -> view C -> >> > table X where viewA, viewB and table X belong to catalog 1, and view C >> to >> > catalog 2. If viewB is a definer view, there's no way to express that >> the >> > security evaluation context has changed (for example a different >> user/role) >> > and should be used to check access on view C (assuming that they share >> the >> > same auth infra). The fallback would be to query view C as the original >> > query user (which would be safe/no data leakage) and only send complete >> > chains relative to a single catalog but this may cause access denied >> > scenarios. This situation has been flagged by both Spark [1] and Trino >> [2] >> > communities as well. >> > >> > [1] https://github.com/apache/iceberg/pull/15895 >> > [2] https://github.com/trinodb/trino/pull/31542 >> > >> > On Wed, Nov 4, 2026 at 11:58 PM Jean-Baptiste Onofré <[email protected]> >> > wrote: >> > >> > > Hi Laurent, >> > > >> > > Thanks for the proposal! >> > > >> > > I'm +1 on part 1. Polaris accepts referenced-by today and drops it as >> you >> > > said, so giving authorizers access to the chain makes sense to me. >> > > Regarding the SPI, Polaris is in charge of parsing, validating and >> > > forwarding the chain to the authorizers. Each authorizer decides what >> to >> > do >> > > with it, possibly ignoring it. >> > > >> > > I propose to focus on part 1 for now, I believe it will be easy to get >> > > consensus on that. >> > > >> > > I have 3 comments on part 1: >> > > - Today Polaris ignores the parameter, even if the Iceberg client >> sends >> > it >> > > since 1.12. With the PR, a request with a malformed chain in the >> > parameter, >> > > a repeated entry or more than 10 entries gets a 400 (even if the >> > authorizer >> > > ignores the chain). I'm completely ok with "strict" validation but it >> > means >> > > users "might" see it. >> > > - The spec identifiers have no catalog part, so the PR maps each >> entry to >> > > a securable in the target catalog. A entry may not exist, or may >> match an >> > > unrelated view with the same name if an engine includes a view from >> > another >> > > catalog. I would state this clearly in the referencedBy() javadoc, so >> > > authorizers don't treat the chain as a list of verified entities. >> > > - I think OPA is a good first consumer candidate. We can add the >> chain to >> > > the OPA input (in the resource, with targets and secondaries). It's >> just >> > > additive, needs no new privilege or persistence change. It's a good >> > "PoC", >> > > also helping to shape the SPI with a real use case. That's a good >> first >> > > step to part 2. >> > > >> > > Regards >> > > JB >> > > >> > > On Wed, Nov 4, 2026 at 9:19 PM Laurent Goujon via dev < >> > > [email protected]> wrote: >> > > >> > >> Hi all, >> > >> >> > >> The Iceberg REST spec has an optional `referenced-by` query >> parameter on >> > >> loadTable, loadView and loadCredentials (apache/iceberg#13810). When >> an >> > >> engine loads a table or view while evaluating a view, it can pass the >> > >> chain >> > >> of views it went through, outermost first. It was added to make >> definer >> > >> views possible. Polaris accepts the parameter but drops it today, so >> no >> > >> authorizer can act on it. >> > >> >> > >> I wrote a proposal and opened a draft PR for the first part: >> > >> >> > >> - Proposal: https://github.com/apache/polaris/issues/5979 >> > >> - Draft PR (part 1): https://github.com/apache/polaris/pull/5980 >> > >> >> > >> The proposal has two parts: >> > >> >> > >> 1. SPI. Pass the parsed chain to authorizers through a new >> > >> `SingleTargetAuthorizationIntent#referencedBy()` accessor. A chain >> that >> > >> cannot be parsed, is deeper than a configurable limit (10 by >> default) or >> > >> repeats an entry is rejected with a 400. The chain is added to the >> load >> > >> events. Existing authorizers (built-in, OPA, Ranger) are unchanged, >> and >> > >> the >> > >> PR does not change any decision. >> > >> 2. Built-in authorizer. An administrator binds a view to a catalog >> role >> > (a >> > >> new `CATALOG_ROLE_EXECUTE_AS` privilege, with the view as the >> grantee), >> > >> and >> > >> a new catalog privilege marks the callers that may assert a chain. >> The >> > >> built-in authorizer then evaluates the chain, switching to the bound >> > role >> > >> at each bound view. This part is not in the PR. >> > >> >> > >> Since part 1 changes an extension point, I would like to agree on it >> > here >> > >> before taking the PR out of draft. >> > >> >> > >> Re part 2: the proposal includes it so that part 1 comes with a >> proof of >> > >> concept for how an authorizer would use the chain. I can also work >> on a >> > PR >> > >> for it, or we could focus on a different authorizer instead (I'm >> open to >> > >> that). >> > >> >> > >> Thanks, >> > >> Laurent >> > >> >> > > >> > >> >
