Thanks Dan and Thanks for the discussion everyone ! I update the spec PR with this conclusion i.e disallow overlap both from the catalog side and client side expectation, please do take a look when you get some time !
Best, Prashant Singh On Fri, Aug 21, 2026 at 10:21 AM Daniel Weeks <[email protected]> wrote: > I had a chance to catch up on the discussion from Tuesday and I think the > justification Sung laid out was good. > > My original concern was the motivation for not addressing the conflict > which centered primarily on: > 1) We lack examples (other than Redshift), so we don't know what > position to take > 2) It's acceptable because it's backwards compatible, so we can fix it > later > > #2 isn't true, and #1 felt like we were just leaving a gap in the proposal > that could be addressed by taking a more opinionated stance. > > Based on my understanding, the expectation is that the catalog should > either deconflict (by supplying appropriate substitutions/handle the > precedence conflicts) or fail. (I would also have been fine defining an > order in which they are applied, but I understand the motivation for > keeping the client simple). > > I'm fine with putting the complexity on the catalog and disallowing the > overlap. > > -Dan > > > > On Mon, Aug 10, 2026 at 1:16 PM Russell Spitzer <[email protected]> > wrote: > >> I actually think we already have precedent for how to handle situations >> like this in the spec (TLDR; (1 - Disallow Ambiguous States)) from our >> earlier discussions with default values. >> >> For those who remember <https://github.com/apache/iceberg/pull/12841>, >> we spent a lot of time discussing how parent and child defaults should >> interact when both could supply a value for the same leaf. We discussed all >> the various options almost *identically *to what we are discussing now. >> After much back and forth we just decided to disallow the ambiguous >> configuration. Struct defaults are only null or {}, and field defaults live >> on the fields themselves. Simple, with no extra rules that a client can >> mess up. >> >> I think we are in basically the same situation here. Overlapping >> projections on a nested container and one of its descendants don't have an >> obvious resolution. Similar to the default case, we could try to establish >> client rules that every client must implement and could potentially >> implement incorrectly, or we could simply declare that state is invalid. >> >> So I am +1 on Option A / Nevin’s contract (1): catalogs MUST NOT emit >> overlapping nested projections, and readers MUST fail closed if they >> receive them. Similar to a table with multiple overlapping >> initial-defaults, the reader should simply throw an error and state that >> the input is not valid. >> >> >> On Fri, Aug 7, 2026 at 3:10 PM Sung Yun <[email protected]> wrote: >> >>> > While changing this requirement in the future is structurally >>> compatible, it is not behaviorally compatible. >>> >>> Dan, that is a fair point, and I withdraw my claim that relaxing the >>> constraint later would be entirely backward-compatible. It would not be, >>> given the data access regression you described. >>> >>> There is a real behavioral consequence whenever a new capability is >>> introduced to ReadRestrictions, and I think the community would benefit >>> from working out what the cost of introducing new capabilities is, beyond >>> the immediate scope of nested policy handling. I would rather >>> compartmentalize this discussion to its own thread than litigate the >>> question on nested policy conflict resolution based on an uncalibrated >>> weighting of that cost. I will raise it at the 8/18 sync. >>> >>> Setting that aside, Nevin is right that this is partly a question of >>> which tradeoffs we value. I think Option A holds up well on >>> interoperability with existing policy engines. I put together a survey of >>> how current systems handle nested policies to help with that analysis, and >>> I am happy to walk through it together during the 8/18 sync [1]. >>> >>> Here is a short summary. Every surveyed system resolves to a single >>> policy per field value, and no system composes more than one masking >>> transforms over one value. Most make an ancestor-descendant overlap >>> un-authorable, either by excluding the container from masking or by >>> permitting exactly one attachment point per column. Redshift is the only >>> system that permits both to be attached, and it requires >>> administrator-authored priorities and resolves to a single winner before >>> the reader is involved. >>> >>> Two additional properties seem worth weighing in the discussion. First, >>> whether moving conflict resolution from the policy engine to the client is >>> valuable to the community, and whether a defined semantic in Iceberg can do >>> that effectively. Second, whether a composition rule, where transforms >>> apply on top of each other, is a capability we want to introduce without >>> industry precedent. >>> >>> As Nevin noted, we have not dug into what those semantics could be. I >>> hoped laying them out would help move the discussion forward. >>> >>> Sung >>> >>> [1] https://s.apache.org/0w1mu >>> >>> On 2026/08/07 00:00:55 Nevin Zheng wrote: >>> > Hi all, >>> > >>> > Thanks everyone for the thoughtful discussion. I wanted to offer a >>> > slightly different framing that may help us align on the decision >>> > we're making. >>> > >>> > I see two dimensions: >>> > >>> > - Catalog: MUST NOT send overlaps vs. MAY send overlaps >>> > - Reader: fail closed vs. apply defined semantics >>> > >>> > That gives us four possible contracts: >>> > >>> > (1) MUST NOT + fail closed — strict contract >>> > (2) MUST NOT + defined semantics — defensive fallback >>> > (3) MAY + fail closed — partial interoperability >>> > (4) MAY + defined semantics — supported semantics >>> > >>> > My read is that the current proposal is (1). I would place Dan's >>> > position as a version of (4), with the concrete semantics still to be >>> > defined. >>> > >>> > At this level, I think this is partly a values/tradeoff decision >>> > rather than a question with one technically correct answer. We can >>> > value simplicity and fail-closed behavior, or place more value on >>> > expressiveness and interoperability. Until we align on those >>> > priorities, or find a compromise, I think we may continue disagreeing >>> > on the lower-level mechanics. >>> > >>> > There are of course finer questions around precedence/composition, >>> > security, compatibility, implementation complexity, and extensibility. >>> > I think choosing the contract at this level first could help clarify >>> > those discussions. >>> > >>> > Another useful lens may be the critical user journey: when overlapping >>> > projections occur, who bears the cost? With (1), the query fails >>> > safely and the operator/end user has to resolve the configuration. >>> > With (4), the query can continue, but Iceberg readers take on the >>> > complexity of correctly interpreting the overlap. >>> > >>> > My preference is (1). Catalogs should produce unambiguous >>> > restrictions, and readers should fail closed with a clear error when >>> > they don't. Catalogs will inevitably have bugs or misconfigurations; I >>> > prefer a loud, safe, diagnosable failure over requiring every reader >>> > to interpret ambiguous policy. >>> > >>> > This also gives us a safe failure mode if we expand the semantics >>> > later: older implementations may fail, but they fail closed. >>> > >>> > That said, I'm comfortable with another coherent contract if the >>> > community prefers its tradeoffs and we specify and implement it well. >>> > >>> > I hope this framing is useful. Please let me know if I've >>> > misrepresented the design space or anyone's position. >>> > >>> > Nevin >>> > >>> > On Thu, 6 Aug 2026 16:17:39 -0700, Steven Wu <[email protected]> >>> wrote: >>> > > > This is true regardless. The client is not involved in resolving >>> anything. >>> > > >>> > > I am only discussing main question of this thread: failing the query >>> vs resolving precedence rules at client/reader-side for nested fields with >>> conflicting policies? >>> > > >>> > > If a projection targets a nested-typed field (struct, list, or map), >>> other projections in the same ReadRestrictions must not target any nested >>> field-id (struct subfields, list elements, or >>> > > >>> > > map keys/values) at any depth. This specification does not define >>> how such actions combine. >>> > > >>> > > A reader that receives such a response must fail the query. >>> > > >>> > > > Relaxing the requirement is not backward compatible for clients. >>> > > >>> > > If we relax this in the future, here is how we can do so compatibly. >>> > > >>> > > A reader that receives such a response must fail the query. >>> > > >>> > > --> >>> > > >>> > > A reader that receives such a response may fail the query or may >>> apply the following precedence rule to deconflict. >>> > > >>> > > > we're taking a shortcut by not implementing full schema support >>> for this feature >>> > > >>> > > What does the "full schema support" mean in this context of >>> conflicting policies for nested fields? >>> > > >>> > > Thanks, >>> > > Steven >>> > > >>> > > On Thu, Aug 6, 2026 at 2:43 PM Daniel Weeks <[email protected]> >>> wrote: >>> > > >>> > > Current spec requires clients to fail if the catalog server returns >>> conflicting policies. If we relax this requirement in the future with >>> precedence rules for client-side de-conflicting, older clients would still >>> fail. I don't see it as "an unpredictable behavior change" for older >>> clients. I would also the relaxation would probably use the wording of MAY >>> (not MUST) applying these precedence rules to deconflict. >>> > > >>> > > Relaxing the requirement is not backward compatible for clients. >>> That would be equivalent to saying that we can make a required field >>> optional because the clients need to validate required fields so it's ok if >>> they break. Backward compatibility is about not breaking clients, not >>> making it ok to break clients. >>> > > >>> > > I heard comments that this rule is very easy for the policy engine >>> to enforce . . . >>> > > >>> > > Handling nested fields doesn't change how hard it is for a policy >>> engine to enforce. They can always fallback to not allowing it (they would >>> already be required to detect these cases). >>> > > >>> > > If we ask clients to resolve conflicting policies >>> > > >>> > > This makes it sound like we're shifting the burden to the client, >>> but that's not the case. We are just defining an order in which they are >>> applied. Policy engines are required to resolve conflicts. The client just >>> applies the restrictions in a well defined order. >>> > > >>> > > I would let the policy engine either enforce that >>> conflicting/ambiguous policies are not accepted or perform the >>> deconflicting/precedence rules before returning to the client. >>> > > >>> > > This is true regardless. The client is not involved in resolving >>> anything. >>> > > >>> > > The issue here is that we're taking a shortcut by not implementing >>> full schema support for this feature and presenting it as a reasonable path >>> based on a misconception that it's backward compatible. >>> > > >>> > > -Dan >>> > > >>> > > On Thu, Aug 6, 2026 at 11:44 AM Steven Wu <[email protected]> >>> wrote: >>> > > >>> > > > Changing that in the future would break older clients that >>> validate and result in an unpredictable changes in behior of clients that >>> assume correctness from the catalog. >>> > > >>> > > Current spec requires clients to fail if the catalog server returns >>> conflicting policies. If we relax this requirement in the future with >>> precedence rules for client-side de-conflicting, older clients would still >>> fail. I don't see it as "an unpredictable behavior change" for older >>> clients. I would also the relaxation would probably use the wording of MAY >>> (not MUST) applying these precedence rules to deconflict. >>> > > >>> > > I agree with Prashanth and Sung that it is hard for Iceberg to force >>> a specific behavior considering the divergent behaviors in the industry. >>> > > >>> > > As I mentioned during the sync, I also see it the same as the other >>> requirement we put on the catalog server. I heard comments that this rule >>> is very easy for the policy engine to enforce. If we ask clients to resolve >>> conflicting policies for nested fields with precedence rules, I can argue >>> the same thing here for this rule. Can't we also ask clients to define >>> resolution precedence with the projection policies? >>> > > >>> > > 3. A server must not return more than one projection for the same >>> field-id >>> > > in required-column-projections. If a duplicate field-id appears, the >>> reader >>> > > must fail the query. >>> > > >>> > > Since IRC doesn't define policy management APIs and behaviors, I >>> would let the policy engine either enforce that conflicting/ambiguous >>> policies are not accepted or perform the deconflicting/precedence rules >>> before returning to the client. Hence, i agree with the current spec >>> writing that clients must fail in the case of conflicting policies with >>> nested fields. >>> > > >>> > > On Thu, Aug 6, 2026 at 9:40 AM Daniel Weeks <[email protected]> >>> wrote: >>> > > >>> > > I'd like to clarify the point on backward compatibility claims here >>> first as it indicates a dangerous misunderstanding of compatibility in the >>> REST protocol. The statement that "Forbidding is reversible; precedence >>> isn't. Relaxing a MUST NOT later is backward-compatible" is incorrect, and >>> we need to be very careful regarding backward compatibility. It also >>> creates a false sense of security by deferring this, as it implies the >>> issue can be addressed later without consequence. >>> > > >>> > > While changing this requirement in the future is structurally >>> compatible, it is not behaviorally compatible. Given the current spec >>> wording, a responsible client should validate that they do not receive a >>> invalid set of redactions from the catalog as this is prohibited by the >>> spec and there's no guidance on how the client should apply them. Changing >>> that in the future would break older clients that validate and result in an >>> unpredictable changes in behior of clients that assume correctness from the >>> catalog. This is neither safe nor backward compatible. >>> > > >>> > > I think addressing this now is well within scope and will prevent >>> unnecessary future protocol evolution. Regardless of what type of >>> precedence we define, we already know that some policies (e.g. Redshift) >>> that will express more than we will likely include in the protocol. It's >>> also likely that different policy systems will have conflicting views on >>> this matter. Since we're already taking an opinionated stance and levying >>> requirements on the catalog implementation to align their policy with the >>> protocol, this isn't any more complicated than prohibiting conflicting >>> policy. >>> > > >>> > > Read restrictions have a number of unexpected cliffs regarding how >>> policy changes affect what clients can consume (e.g. ABAC policies that >>> update from simple redaction to more contextual redaction) and this is just >>> one more scenario that limits interoperability. >>> > > >>> > > The backward compatibility issue further highlights why we need a >>> solid reference implementation as I think it would make the problem clear. >>> I don't want to repeat the challenges we had with scan/plan where we >>> released the protocol with issues that we, fortuntately, were able to >>> address as the reference implementation was built. >>> > > >>> > > -Dan >>> > > >>> > > On Thu, Aug 6, 2026 at 7:56 AM Sung Yun <[email protected]> wrote: >>> > > >>> > > Thanks Prashant for doing such extensive research on this topic. >>> > > >>> > > I believe the industry divergence on how nested type policies are >>> handled is a good indicator that we should hold off on forcing a standard >>> for nested conflict resolution right now. Defining a standard is a highly >>> sensitive design decision that deserves its own focused, long-term >>> community discussion rather than being rushed. >>> > > >>> > > During our syncs, we’ve discussed several potential evaluation >>> models, each with distinct trade-offs: >>> > > >>> > > - Outermost wins (single policy is enforced) >>> > > >>> > > - Innermost wins (single policy is enforced) >>> > > >>> > > - Innermost evaluated first (pipelined and layered sequentially) >>> > > >>> > > - Outermost evaluated first (pipelined and layered sequentially) >>> > > >>> > > These models carry different security implications, and hence I'm in >>> favor of choosing option A (forbidding overlapping nested policies and >>> failing-closed on the client) to not rush this discussion and unblock >>> ReadRestrictions today. >>> > > >>> > > If the community later decides to define a precedence standard, >>> relaxing this rule is entirely backward-compatible. On the client/reader >>> side, we would simply be introducing a new handling capability for what >>> used to be an invalid response. On the server/catalog side, the timeline >>> remains entirely in the catalog administrator's control because the new >>> behavior only triggers once they choose to update their catalog >>> implementations to emit multiple nested policies. >>> > > >>> > > One final observation on the industry examples: >>> > > >>> > > Even in Redshift, which is the only example where multiple policies >>> can technically be registered on a nested path, the engine treats >>> overlapping policies as a conflict. It resolves this conflict by enforcing >>> only the single, highest-priority policy at query time. In practice, this >>> "highest-priority wins" model is highly consistent with Option A. It >>> already assumes that the system must ultimately resolve the conflict down >>> to a single, clean policy rather than attempting to chain and layer >>> multiple policies sequentially. I believe that this can be the >>> responsibility of the policy engine, rather than becoming a preemptive >>> standard in the ReadRestrictions spec. >>> > > >>> > > All in all, I strongly support keeping the current restrictive >>> instruction in the spec (Option A) for the initial release and updating the >>> words with normative RFC 2119 terminology (using MUST NOT and MUST fail) to >>> clearly define the behavior on the client and server side. [1] >>> > > >>> > > Sung >>> > > >>> > > [1] >>> https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 >>> > > >>> > > On 2026/08/04 23:25:40 Prashant Singh wrote: >>> > > >>> > > > Hi all, >>> > > >>> > > > >>> > > >>> > > > One open thread on the Read Restrictions spec PR [1] is the last >>> item I'd >>> > > >>> > > > like to settle before calling a vote [2]. The last sync leaned >>> towards the >>> > > >>> > > > restrictive option, but there was an ask by Dan that the >>> alternative be >>> > > >>> > > > explored properly first, so I'm bringing it here. >>> > > >>> > > > >>> > > >>> > > > The question >>> > > >>> > > > >>> > > >>> > > > required-column-projections binds an action to a field-id, and a >>> nested >>> > > >>> > > > type has a field-id for the container and separate ones beneath it >>> (struct >>> > > >>> > > > subfields; list element; map key and value). >>> > > >>> > > > >>> > > >>> > > > Concretely, with 2: address struct<3: street string, 4: city >>> string>: >>> > > >>> > > > >>> > > >>> > > > - field-id: 2 action: mask-to-fixed-value >>> > > >>> > > > >>> > > >>> > > > - field-id: 4 action: replace-with-null >>> > > >>> > > > >>> > > >>> > > > mask-to-fixed-value overwrites everything beneath the container, >>> so it >>> > > >>> > > > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore >>> > > >>> > > > returns {"street": >>> > > >>> > > > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns >>> {"street": null, >>> > > >>> > > > "city": "XXXXXXXX"}, though the catalog asked for the whole struct >>> to be >>> > > >>> > > > masked. >>> > > >>> > > > >>> > > >>> > > > if a catalog returns two projections (one on the outer container >>> and one on >>> > > >>> > > > a field-id below it), how should clients handle it? >>> > > >>> > > > >>> > > >>> > > > (A) forbid the overlap; the existing fail-closed rule then makes a >>> reader >>> > > >>> > > > >>> > > >>> > > > that receives one fail the query. The spec PR currently says: >>> > > >>> > > > >>> > > >>> > > > If a projection targets a nested-typed field (struct, list, or >>> map), other >>> > > >>> > > > projections in the same ReadRestrictions must not target any nested >>> > > >>> > > > field-id (struct subfields, list elements, or >>> > > >>> > > > >>> > > >>> > > > map keys/values) at any depth. This specification does not define >>> how such >>> > > >>> > > > actions combine. >>> > > >>> > > > >>> > > >>> > > > A reader that receives such a response must fail the query. >>> > > >>> > > > >>> > > >>> > > > (B) define precedence (e.g. outer-most wins) and allow it >>> > > >>> > > > >>> > > >>> > > > Why current spec PR choose option A >>> > > >>> > > > >>> > > >>> > > > No semantics to borrow. There are different behaviors in the >>> current >>> > > >>> > > > industry practice >>> > > >>> > > > >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > BigQuery: >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > "Policy tags can't be applied to columns that use the STRUCT data >>> > > >>> > > > type" [3]; >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > Redshift both experience based on data-types: >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > "You can only apply masking policies to scalar values on the SUPER >>> > > >>> > > > path" [4] rejects the overlap. >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > Redshift is the only one that lets both be expressed, and it calls >>> > > >>> > > > the pair a conflict, resolved by an admin-supplied priority [5] >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, >>> Impala and >>> > > >>> > > > Ranger define no parent-vs-nested resolution at all, and where the >>> question >>> > > >>> > > > has been raised it has stayed open: RANGER-3525 since 2021 [6], >>> HIVE-22823 >>> > > >>> > > > [7], trino#16964 [8]. >>> > > >>> > > > - >>> > > >>> > > > >>> > > >>> > > > SQL:2016 defines nothing - <privilege column list> is a flat >>> column name >>> > > >>> > > > list. >>> > > >>> > > > >>> > > >>> > > > >>> > > >>> > > > >>> > > >>> > > > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT >>> later is >>> > > >>> > > > backward-compatible: invalid responses become valid, and no reader >>> changes >>> > > >>> > > > what it returns for a response that was already legal. If we define >>> > > >>> > > > outer-most-wins now and later want the inner action to contribute, >>> we've >>> > > >>> > > > silently changed the values a compliant reader returns for the same >>> > > >>> > > > response. >>> > > >>> > > > >>> > > >>> > > > On the future-actions concern: I don't think the prohibition >>> limits any >>> > > >>> > > > action. A future action can target a container, or a nested field >>> - just >>> > > >>> > > > not both in one response, which is the only combination with no >>> defined >>> > > >>> > > > meaning today. >>> > > >>> > > > >>> > > >>> > > > Asks >>> > > >>> > > > >>> > > >>> > > > 1. Opinions on option A (forbid) vs B (define precedence) >>> > > >>> > > > >>> > > >>> > > > 2. If (B), is outer-most-wins the rule — and are we comfortable >>> that >>> > > >>> > > > >>> > > >>> > > > nothing in the current action set can show it’s the right choice? >>> > > >>> > > > >>> > > >>> > > > I am looking forward to hear what other people think and if they >>> have >>> > > >>> > > > objection to the current proposed spec >>> > > >>> > > > >>> > > >>> > > > We time-boxed this to the next sync (Aug 18) in the last one. If >>> there's no >>> > > >>> > > > further input by then, we will keep the current restrictive >>> wording and say >>> > > >>> > > > so in this thread rather than treat silence as agreement - the >>> precedence >>> > > >>> > > > question should be closed explicitly, not by default. And if a use >>> case >>> > > >>> > > > turns up after we ship, Option A lets us relax the rule >>> compatibility, >>> > > >>> > > > which is most of why the PR takes it. Notes from sync [10]. >>> > > >>> > > > >>> > > >>> > > > Thanks, >>> > > >>> > > > >>> > > >>> > > > Prashant >>> > > >>> > > > >>> > > >>> > > > References >>> > > >>> > > > >>> > > >>> > > > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 >>> > > >>> > > > >>> > > >>> > > > [2] Review thread: >>> > > >>> > > > >>> https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 >>> > > >>> > > > >>> > > >>> > > > [3] BigQuery, column data masking: >>> > > >>> > > > https://cloud.google.com/bigquery/docs/column-data-masking-intro >>> > > >>> > > > >>> > > >>> > > > [4] Redshift, DDM with SUPER paths: >>> > > >>> > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html >>> > > >>> > > > >>> > > >>> > > > [5] Redshift, DDM policy hierarchy: >>> > > >>> > > > >>> https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html >>> > > >>> > > > >>> > > >>> > > > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 >>> > > >>> > > > >>> > > >>> > > > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 >>> > > >>> > > > >>> > > >>> > > > [8] trinodb/trino#16964: >>> https://github.com/trinodb/trino/issues/16964 >>> > > >>> > > > >>> > > >>> > > > [9] IMPALA-15130: >>> https://issues.apache.org/jira/browse/IMPALA-15130 >>> > > >>> > > > >>> > > >>> > > > [10] Note from ReadRestrictions syncs: >>> > > >>> > > > >>> https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx >>> > > >>> > > > >>> > >>> >>
