Thanks Prashant for doing such extensive research on this topic. I believe the industry divergence on how nested type policies are handled is a good indicator that we should hold off on forcing a standard for nested conflict resolution right now. Defining a standard is a highly sensitive design decision that deserves its own focused, long-term community discussion rather than being rushed.
During our syncs, we’ve discussed several potential evaluation models, each with distinct trade-offs: - Outermost wins (single policy is enforced) - Innermost wins (single policy is enforced) - Innermost evaluated first (pipelined and layered sequentially) - Outermost evaluated first (pipelined and layered sequentially) These models carry different security implications, and hence I'm in favor of choosing option A (forbidding overlapping nested policies and failing-closed on the client) to not rush this discussion and unblock ReadRestrictions today. If the community later decides to define a precedence standard, relaxing this rule is entirely backward-compatible. On the client/reader side, we would simply be introducing a new handling capability for what used to be an invalid response. On the server/catalog side, the timeline remains entirely in the catalog administrator's control because the new behavior only triggers once they choose to update their catalog implementations to emit multiple nested policies. One final observation on the industry examples: Even in Redshift, which is the only example where multiple policies can technically be registered on a nested path, the engine treats overlapping policies as a conflict. It resolves this conflict by enforcing only the single, highest-priority policy at query time. In practice, this "highest-priority wins" model is highly consistent with Option A. It already assumes that the system must ultimately resolve the conflict down to a single, clean policy rather than attempting to chain and layer multiple policies sequentially. I believe that this can be the responsibility of the policy engine, rather than becoming a preemptive standard in the ReadRestrictions spec. All in all, I strongly support keeping the current restrictive instruction in the spec (Option A) for the initial release and updating the words with normative RFC 2119 terminology (using MUST NOT and MUST fail) to clearly define the behavior on the client and server side. [1] Sung [1] https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 On 2026/08/04 23:25:40 Prashant Singh wrote: > Hi all, > > One open thread on the Read Restrictions spec PR [1] is the last item I'd > like to settle before calling a vote [2]. The last sync leaned towards the > restrictive option, but there was an ask by Dan that the alternative be > explored properly first, so I'm bringing it here. > > The question > > required-column-projections binds an action to a field-id, and a nested > type has a field-id for the container and separate ones beneath it (struct > subfields; list element; map key and value). > > Concretely, with 2: address struct<3: street string, 4: city string>: > > - field-id: 2 action: mask-to-fixed-value > > - field-id: 4 action: replace-with-null > > mask-to-fixed-value overwrites everything beneath the container, so it > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore > returns {"street": > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": null, > "city": "XXXXXXXX"}, though the catalog asked for the whole struct to be > masked. > > if a catalog returns two projections (one on the outer container and one on > a field-id below it), how should clients handle it? > > (A) forbid the overlap; the existing fail-closed rule then makes a reader > > that receives one fail the query. The spec PR currently says: > > If a projection targets a nested-typed field (struct, list, or map), other > projections in the same ReadRestrictions must not target any nested > field-id (struct subfields, list elements, or > > map keys/values) at any depth. This specification does not define how such > actions combine. > > A reader that receives such a response must fail the query. > > (B) define precedence (e.g. outer-most wins) and allow it > > Why current spec PR choose option A > > No semantics to borrow. There are different behaviors in the current > industry practice > > - > > BigQuery: > - > > "Policy tags can't be applied to columns that use the STRUCT data > type" [3]; > - > > Redshift both experience based on data-types: > - > > "You can only apply masking policies to scalar values on the SUPER > path" [4] rejects the overlap. > - > > Redshift is the only one that lets both be expressed, and it calls > the pair a conflict, resolved by an admin-supplied priority [5] > - > > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, Impala and > Ranger define no parent-vs-nested resolution at all, and where the question > has been raised it has stayed open: RANGER-3525 since 2021 [6], HIVE-22823 > [7], trino#16964 [8]. > - > > SQL:2016 defines nothing - <privilege column list> is a flat column name > list. > > > > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT later is > backward-compatible: invalid responses become valid, and no reader changes > what it returns for a response that was already legal. If we define > outer-most-wins now and later want the inner action to contribute, we've > silently changed the values a compliant reader returns for the same > response. > > On the future-actions concern: I don't think the prohibition limits any > action. A future action can target a container, or a nested field - just > not both in one response, which is the only combination with no defined > meaning today. > > Asks > > 1. Opinions on option A (forbid) vs B (define precedence) > > 2. If (B), is outer-most-wins the rule — and are we comfortable that > > nothing in the current action set can show it’s the right choice? > > I am looking forward to hear what other people think and if they have > objection to the current proposed spec > > We time-boxed this to the next sync (Aug 18) in the last one. If there's no > further input by then, we will keep the current restrictive wording and say > so in this thread rather than treat silence as agreement - the precedence > question should be closed explicitly, not by default. And if a use case > turns up after we ship, Option A lets us relax the rule compatibility, > which is most of why the PR takes it. Notes from sync [10]. > > Thanks, > > Prashant > > References > > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 > > [2] Review thread: > https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 > > [3] BigQuery, column data masking: > https://cloud.google.com/bigquery/docs/column-data-masking-intro > > [4] Redshift, DDM with SUPER paths: > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html > > [5] Redshift, DDM policy hierarchy: > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html > > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 > > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 > > [8] trinodb/trino#16964: https://github.com/trinodb/trino/issues/16964 > > [9] IMPALA-15130: https://issues.apache.org/jira/browse/IMPALA-15130 > > [10] Note from ReadRestrictions syncs: > https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx >
