Hi all, Thanks everyone for the thoughtful discussion. I wanted to offer a slightly different framing that may help us align on the decision we're making.
I see two dimensions: - Catalog: MUST NOT send overlaps vs. MAY send overlaps - Reader: fail closed vs. apply defined semantics That gives us four possible contracts: (1) MUST NOT + fail closed — strict contract (2) MUST NOT + defined semantics — defensive fallback (3) MAY + fail closed — partial interoperability (4) MAY + defined semantics — supported semantics My read is that the current proposal is (1). I would place Dan's position as a version of (4), with the concrete semantics still to be defined. At this level, I think this is partly a values/tradeoff decision rather than a question with one technically correct answer. We can value simplicity and fail-closed behavior, or place more value on expressiveness and interoperability. Until we align on those priorities, or find a compromise, I think we may continue disagreeing on the lower-level mechanics. There are of course finer questions around precedence/composition, security, compatibility, implementation complexity, and extensibility. I think choosing the contract at this level first could help clarify those discussions. Another useful lens may be the critical user journey: when overlapping projections occur, who bears the cost? With (1), the query fails safely and the operator/end user has to resolve the configuration. With (4), the query can continue, but Iceberg readers take on the complexity of correctly interpreting the overlap. My preference is (1). Catalogs should produce unambiguous restrictions, and readers should fail closed with a clear error when they don't. Catalogs will inevitably have bugs or misconfigurations; I prefer a loud, safe, diagnosable failure over requiring every reader to interpret ambiguous policy. This also gives us a safe failure mode if we expand the semantics later: older implementations may fail, but they fail closed. That said, I'm comfortable with another coherent contract if the community prefers its tradeoffs and we specify and implement it well. I hope this framing is useful. Please let me know if I've misrepresented the design space or anyone's position. Nevin On Thu, 6 Aug 2026 16:17:39 -0700, Steven Wu <[email protected]> wrote: > > This is true regardless. The client is not involved in resolving anything. > > I am only discussing main question of this thread: failing the query vs > resolving precedence rules at client/reader-side for nested fields with > conflicting policies? > > If a projection targets a nested-typed field (struct, list, or map), other > projections in the same ReadRestrictions must not target any nested field-id > (struct subfields, list elements, or > > map keys/values) at any depth. This specification does not define how such > actions combine. > > A reader that receives such a response must fail the query. > > > Relaxing the requirement is not backward compatible for clients. > > If we relax this in the future, here is how we can do so compatibly. > > A reader that receives such a response must fail the query. > > --> > > A reader that receives such a response may fail the query or may apply the > following precedence rule to deconflict. > > > we're taking a shortcut by not implementing full schema support for this > > feature > > What does the "full schema support" mean in this context of conflicting > policies for nested fields? > > Thanks, > Steven > > On Thu, Aug 6, 2026 at 2:43 PM Daniel Weeks <[email protected]> wrote: > > Current spec requires clients to fail if the catalog server returns > conflicting policies. If we relax this requirement in the future with > precedence rules for client-side de-conflicting, older clients would still > fail. I don't see it as "an unpredictable behavior change" for older clients. > I would also the relaxation would probably use the wording of MAY (not MUST) > applying these precedence rules to deconflict. > > Relaxing the requirement is not backward compatible for clients. That would > be equivalent to saying that we can make a required field optional because > the clients need to validate required fields so it's ok if they break. > Backward compatibility is about not breaking clients, not making it ok to > break clients. > > I heard comments that this rule is very easy for the policy engine to enforce > . . . > > Handling nested fields doesn't change how hard it is for a policy engine to > enforce. They can always fallback to not allowing it (they would already be > required to detect these cases). > > If we ask clients to resolve conflicting policies > > This makes it sound like we're shifting the burden to the client, but that's > not the case. We are just defining an order in which they are applied. Policy > engines are required to resolve conflicts. The client just applies the > restrictions in a well defined order. > > I would let the policy engine either enforce that conflicting/ambiguous > policies are not accepted or perform the deconflicting/precedence rules > before returning to the client. > > This is true regardless. The client is not involved in resolving anything. > > The issue here is that we're taking a shortcut by not implementing full > schema support for this feature and presenting it as a reasonable path based > on a misconception that it's backward compatible. > > -Dan > > On Thu, Aug 6, 2026 at 11:44 AM Steven Wu <[email protected]> wrote: > > > Changing that in the future would break older clients that validate and > > result in an unpredictable changes in behior of clients that assume > > correctness from the catalog. > > Current spec requires clients to fail if the catalog server returns > conflicting policies. If we relax this requirement in the future with > precedence rules for client-side de-conflicting, older clients would still > fail. I don't see it as "an unpredictable behavior change" for older clients. > I would also the relaxation would probably use the wording of MAY (not MUST) > applying these precedence rules to deconflict. > > I agree with Prashanth and Sung that it is hard for Iceberg to force a > specific behavior considering the divergent behaviors in the industry. > > As I mentioned during the sync, I also see it the same as the other > requirement we put on the catalog server. I heard comments that this rule is > very easy for the policy engine to enforce. If we ask clients to resolve > conflicting policies for nested fields with precedence rules, I can argue the > same thing here for this rule. Can't we also ask clients to define resolution > precedence with the projection policies? > > 3. A server must not return more than one projection for the same field-id > in required-column-projections. If a duplicate field-id appears, the reader > must fail the query. > > Since IRC doesn't define policy management APIs and behaviors, I would let > the policy engine either enforce that conflicting/ambiguous policies are not > accepted or perform the deconflicting/precedence rules before returning to > the client. Hence, i agree with the current spec writing that clients must > fail in the case of conflicting policies with nested fields. > > On Thu, Aug 6, 2026 at 9:40 AM Daniel Weeks <[email protected]> wrote: > > I'd like to clarify the point on backward compatibility claims here first as > it indicates a dangerous misunderstanding of compatibility in the REST > protocol. The statement that "Forbidding is reversible; precedence isn't. > Relaxing a MUST NOT later is backward-compatible" is incorrect, and we need > to be very careful regarding backward compatibility. It also creates a false > sense of security by deferring this, as it implies the issue can be addressed > later without consequence. > > While changing this requirement in the future is structurally compatible, it > is not behaviorally compatible. Given the current spec wording, a responsible > client should validate that they do not receive a invalid set of redactions > from the catalog as this is prohibited by the spec and there's no guidance on > how the client should apply them. Changing that in the future would break > older clients that validate and result in an unpredictable changes in behior > of clients that assume correctness from the catalog. This is neither safe nor > backward compatible. > > I think addressing this now is well within scope and will prevent unnecessary > future protocol evolution. Regardless of what type of precedence we define, > we already know that some policies (e.g. Redshift) that will express more > than we will likely include in the protocol. It's also likely that different > policy systems will have conflicting views on this matter. Since we're > already taking an opinionated stance and levying requirements on the catalog > implementation to align their policy with the protocol, this isn't any more > complicated than prohibiting conflicting policy. > > Read restrictions have a number of unexpected cliffs regarding how policy > changes affect what clients can consume (e.g. ABAC policies that update from > simple redaction to more contextual redaction) and this is just one more > scenario that limits interoperability. > > The backward compatibility issue further highlights why we need a solid > reference implementation as I think it would make the problem clear. I don't > want to repeat the challenges we had with scan/plan where we released the > protocol with issues that we, fortuntately, were able to address as the > reference implementation was built. > > -Dan > > On Thu, Aug 6, 2026 at 7:56 AM Sung Yun <[email protected]> wrote: > > Thanks Prashant for doing such extensive research on this topic. > > I believe the industry divergence on how nested type policies are handled is > a good indicator that we should hold off on forcing a standard for nested > conflict resolution right now. Defining a standard is a highly sensitive > design decision that deserves its own focused, long-term community discussion > rather than being rushed. > > During our syncs, we’ve discussed several potential evaluation models, each > with distinct trade-offs: > > - Outermost wins (single policy is enforced) > > - Innermost wins (single policy is enforced) > > - Innermost evaluated first (pipelined and layered sequentially) > > - Outermost evaluated first (pipelined and layered sequentially) > > These models carry different security implications, and hence I'm in favor of > choosing option A (forbidding overlapping nested policies and failing-closed > on the client) to not rush this discussion and unblock ReadRestrictions today. > > If the community later decides to define a precedence standard, relaxing this > rule is entirely backward-compatible. On the client/reader side, we would > simply be introducing a new handling capability for what used to be an > invalid response. On the server/catalog side, the timeline remains entirely > in the catalog administrator's control because the new behavior only triggers > once they choose to update their catalog implementations to emit multiple > nested policies. > > One final observation on the industry examples: > > Even in Redshift, which is the only example where multiple policies can > technically be registered on a nested path, the engine treats overlapping > policies as a conflict. It resolves this conflict by enforcing only the > single, highest-priority policy at query time. In practice, this > "highest-priority wins" model is highly consistent with Option A. It already > assumes that the system must ultimately resolve the conflict down to a > single, clean policy rather than attempting to chain and layer multiple > policies sequentially. I believe that this can be the responsibility of the > policy engine, rather than becoming a preemptive standard in the > ReadRestrictions spec. > > All in all, I strongly support keeping the current restrictive instruction in > the spec (Option A) for the initial release and updating the words with > normative RFC 2119 terminology (using MUST NOT and MUST fail) to clearly > define the behavior on the client and server side. [1] > > Sung > > [1] https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 > > On 2026/08/04 23:25:40 Prashant Singh wrote: > > > Hi all, > > > > > > One open thread on the Read Restrictions spec PR [1] is the last item I'd > > > like to settle before calling a vote [2]. The last sync leaned towards the > > > restrictive option, but there was an ask by Dan that the alternative be > > > explored properly first, so I'm bringing it here. > > > > > > The question > > > > > > required-column-projections binds an action to a field-id, and a nested > > > type has a field-id for the container and separate ones beneath it (struct > > > subfields; list element; map key and value). > > > > > > Concretely, with 2: address struct<3: street string, 4: city string>: > > > > > > - field-id: 2 action: mask-to-fixed-value > > > > > > - field-id: 4 action: replace-with-null > > > > > > mask-to-fixed-value overwrites everything beneath the container, so it > > > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore > > > returns {"street": > > > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": null, > > > "city": "XXXXXXXX"}, though the catalog asked for the whole struct to be > > > masked. > > > > > > if a catalog returns two projections (one on the outer container and one on > > > a field-id below it), how should clients handle it? > > > > > > (A) forbid the overlap; the existing fail-closed rule then makes a reader > > > > > > that receives one fail the query. The spec PR currently says: > > > > > > If a projection targets a nested-typed field (struct, list, or map), other > > > projections in the same ReadRestrictions must not target any nested > > > field-id (struct subfields, list elements, or > > > > > > map keys/values) at any depth. This specification does not define how such > > > actions combine. > > > > > > A reader that receives such a response must fail the query. > > > > > > (B) define precedence (e.g. outer-most wins) and allow it > > > > > > Why current spec PR choose option A > > > > > > No semantics to borrow. There are different behaviors in the current > > > industry practice > > > > > > - > > > > > > BigQuery: > > > - > > > > > > "Policy tags can't be applied to columns that use the STRUCT data > > > type" [3]; > > > - > > > > > > Redshift both experience based on data-types: > > > - > > > > > > "You can only apply masking policies to scalar values on the SUPER > > > path" [4] rejects the overlap. > > > - > > > > > > Redshift is the only one that lets both be expressed, and it calls > > > the pair a conflict, resolved by an admin-supplied priority [5] > > > - > > > > > > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, Impala and > > > Ranger define no parent-vs-nested resolution at all, and where the question > > > has been raised it has stayed open: RANGER-3525 since 2021 [6], HIVE-22823 > > > [7], trino#16964 [8]. > > > - > > > > > > SQL:2016 defines nothing - <privilege column list> is a flat column name > > > list. > > > > > > > > > > > > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT later is > > > backward-compatible: invalid responses become valid, and no reader changes > > > what it returns for a response that was already legal. If we define > > > outer-most-wins now and later want the inner action to contribute, we've > > > silently changed the values a compliant reader returns for the same > > > response. > > > > > > On the future-actions concern: I don't think the prohibition limits any > > > action. A future action can target a container, or a nested field - just > > > not both in one response, which is the only combination with no defined > > > meaning today. > > > > > > Asks > > > > > > 1. Opinions on option A (forbid) vs B (define precedence) > > > > > > 2. If (B), is outer-most-wins the rule — and are we comfortable that > > > > > > nothing in the current action set can show it’s the right choice? > > > > > > I am looking forward to hear what other people think and if they have > > > objection to the current proposed spec > > > > > > We time-boxed this to the next sync (Aug 18) in the last one. If there's no > > > further input by then, we will keep the current restrictive wording and say > > > so in this thread rather than treat silence as agreement - the precedence > > > question should be closed explicitly, not by default. And if a use case > > > turns up after we ship, Option A lets us relax the rule compatibility, > > > which is most of why the PR takes it. Notes from sync [10]. > > > > > > Thanks, > > > > > > Prashant > > > > > > References > > > > > > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 > > > > > > [2] Review thread: > > > https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 > > > > > > [3] BigQuery, column data masking: > > > https://cloud.google.com/bigquery/docs/column-data-masking-intro > > > > > > [4] Redshift, DDM with SUPER paths: > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html > > > > > > [5] Redshift, DDM policy hierarchy: > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html > > > > > > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 > > > > > > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 > > > > > > [8] trinodb/trino#16964: https://github.com/trinodb/trino/issues/16964 > > > > > > [9] IMPALA-15130: https://issues.apache.org/jira/browse/IMPALA-15130 > > > > > > [10] Note from ReadRestrictions syncs: > > > https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx > > >
