I'd like to clarify the point on backward compatibility claims here first as it indicates a dangerous misunderstanding of compatibility in the REST protocol. The statement that "Forbidding is reversible; precedence isn't. Relaxing a MUST NOT later is backward-compatible" is incorrect, and we need to be very careful regarding backward compatibility. It also creates a false sense of security by deferring this, as it implies the issue can be addressed later without consequence.
While changing this requirement in the future is structurally compatible, it is not behaviorally compatible. Given the current spec wording, a responsible client should validate that they do not receive a invalid set of redactions from the catalog as this is prohibited by the spec and there's no guidance on how the client should apply them. Changing that in the future would break older clients that validate and result in an unpredictable changes in behior of clients that assume correctness from the catalog. *This is neither safe nor backward compatible.* I think addressing this now is well within scope and will prevent unnecessary future protocol evolution. Regardless of what type of precedence we define, we already know that some policies (e.g. Redshift) that will express more than we will likely include in the protocol. It's also likely that different policy systems will have conflicting views on this matter. Since we're already taking an opinionated stance and levying requirements on the catalog implementation to align their policy with the protocol, this isn't any more complicated than prohibiting conflicting policy. Read restrictions have a number of unexpected cliffs regarding how policy changes affect what clients can consume (e.g. ABAC policies that update from simple redaction to more contextual redaction) and this is just one more scenario that limits interoperability. The backward compatibility issue further highlights why we need a solid reference implementation as I think it would make the problem clear. I don't want to repeat the challenges we had with scan/plan where we released the protocol with issues that we, fortuntately, were able to address as the reference implementation was built. -Dan On Thu, Aug 6, 2026 at 7:56 AM Sung Yun <[email protected]> wrote: > Thanks Prashant for doing such extensive research on this topic. > > I believe the industry divergence on how nested type policies are handled > is a good indicator that we should hold off on forcing a standard for > nested conflict resolution right now. Defining a standard is a highly > sensitive design decision that deserves its own focused, long-term > community discussion rather than being rushed. > > During our syncs, we’ve discussed several potential evaluation models, > each with distinct trade-offs: > - Outermost wins (single policy is enforced) > - Innermost wins (single policy is enforced) > - Innermost evaluated first (pipelined and layered sequentially) > - Outermost evaluated first (pipelined and layered sequentially) > > These models carry different security implications, and hence I'm in favor > of choosing option A (forbidding overlapping nested policies and > failing-closed on the client) to not rush this discussion and unblock > ReadRestrictions today. > > If the community later decides to define a precedence standard, relaxing > this rule is entirely backward-compatible. On the client/reader side, we > would simply be introducing a new handling capability for what used to be > an invalid response. On the server/catalog side, the timeline remains > entirely in the catalog administrator's control because the new behavior > only triggers once they choose to update their catalog implementations to > emit multiple nested policies. > > One final observation on the industry examples: > Even in Redshift, which is the only example where multiple policies can > technically be registered on a nested path, the engine treats overlapping > policies as a conflict. It resolves this conflict by enforcing only the > single, highest-priority policy at query time. In practice, this > "highest-priority wins" model is highly consistent with Option A. It > already assumes that the system must ultimately resolve the conflict down > to a single, clean policy rather than attempting to chain and layer > multiple policies sequentially. I believe that this can be the > responsibility of the policy engine, rather than becoming a preemptive > standard in the ReadRestrictions spec. > > All in all, I strongly support keeping the current restrictive instruction > in the spec (Option A) for the initial release and updating the words with > normative RFC 2119 terminology (using MUST NOT and MUST fail) to clearly > define the behavior on the client and server side. [1] > > Sung > > [1] https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 > > On 2026/08/04 23:25:40 Prashant Singh wrote: > > Hi all, > > > > One open thread on the Read Restrictions spec PR [1] is the last item I'd > > like to settle before calling a vote [2]. The last sync leaned towards > the > > restrictive option, but there was an ask by Dan that the alternative be > > explored properly first, so I'm bringing it here. > > > > The question > > > > required-column-projections binds an action to a field-id, and a nested > > type has a field-id for the container and separate ones beneath it > (struct > > subfields; list element; map key and value). > > > > Concretely, with 2: address struct<3: street string, 4: city string>: > > > > - field-id: 2 action: mask-to-fixed-value > > > > - field-id: 4 action: replace-with-null > > > > mask-to-fixed-value overwrites everything beneath the container, so it > > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore > > returns {"street": > > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": null, > > "city": "XXXXXXXX"}, though the catalog asked for the whole struct to be > > masked. > > > > if a catalog returns two projections (one on the outer container and one > on > > a field-id below it), how should clients handle it? > > > > (A) forbid the overlap; the existing fail-closed rule then makes a > reader > > > > that receives one fail the query. The spec PR currently says: > > > > If a projection targets a nested-typed field (struct, list, or map), > other > > projections in the same ReadRestrictions must not target any nested > > field-id (struct subfields, list elements, or > > > > map keys/values) at any depth. This specification does not define how > such > > actions combine. > > > > A reader that receives such a response must fail the query. > > > > (B) define precedence (e.g. outer-most wins) and allow it > > > > Why current spec PR choose option A > > > > No semantics to borrow. There are different behaviors in the current > > industry practice > > > > - > > > > BigQuery: > > - > > > > "Policy tags can't be applied to columns that use the STRUCT data > > type" [3]; > > - > > > > Redshift both experience based on data-types: > > - > > > > "You can only apply masking policies to scalar values on the SUPER > > path" [4] rejects the overlap. > > - > > > > Redshift is the only one that lets both be expressed, and it calls > > the pair a conflict, resolved by an admin-supplied priority [5] > > - > > > > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, Impala > and > > Ranger define no parent-vs-nested resolution at all, and where the > question > > has been raised it has stayed open: RANGER-3525 since 2021 [6], > HIVE-22823 > > [7], trino#16964 [8]. > > - > > > > SQL:2016 defines nothing - <privilege column list> is a flat column > name > > list. > > > > > > > > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT later is > > backward-compatible: invalid responses become valid, and no reader > changes > > what it returns for a response that was already legal. If we define > > outer-most-wins now and later want the inner action to contribute, we've > > silently changed the values a compliant reader returns for the same > > response. > > > > On the future-actions concern: I don't think the prohibition limits any > > action. A future action can target a container, or a nested field - just > > not both in one response, which is the only combination with no defined > > meaning today. > > > > Asks > > > > 1. Opinions on option A (forbid) vs B (define precedence) > > > > 2. If (B), is outer-most-wins the rule — and are we comfortable that > > > > nothing in the current action set can show it’s the right choice? > > > > I am looking forward to hear what other people think and if they have > > objection to the current proposed spec > > > > We time-boxed this to the next sync (Aug 18) in the last one. If there's > no > > further input by then, we will keep the current restrictive wording and > say > > so in this thread rather than treat silence as agreement - the precedence > > question should be closed explicitly, not by default. And if a use case > > turns up after we ship, Option A lets us relax the rule compatibility, > > which is most of why the PR takes it. Notes from sync [10]. > > > > Thanks, > > > > Prashant > > > > References > > > > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 > > > > [2] Review thread: > > https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 > > > > [3] BigQuery, column data masking: > > https://cloud.google.com/bigquery/docs/column-data-masking-intro > > > > [4] Redshift, DDM with SUPER paths: > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html > > > > [5] Redshift, DDM policy hierarchy: > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html > > > > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 > > > > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 > > > > [8] trinodb/trino#16964: https://github.com/trinodb/trino/issues/16964 > > > > [9] IMPALA-15130: https://issues.apache.org/jira/browse/IMPALA-15130 > > > > [10] Note from ReadRestrictions syncs: > > > https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx > > >
