> This is true regardless. The client is not involved in resolving anything.
I am only discussing main question of this thread: failing the query *vs* resolving precedence rules at *client/reader-side *for nested fields with conflicting policies? If a projection targets a nested-typed field (struct, list, or map), other projections in the same ReadRestrictions must not target any nested field-id (struct subfields, list elements, or map keys/values) at any depth. This specification does not define how such actions combine. *A reader that receives such a response must fail the query.* > Relaxing the requirement is not backward compatible for clients. If we relax this in the future, here is how we can do so compatibly. A reader that receives such a response must fail the query. --> A reader that receives such a response *may* fail the query or *may* apply the following precedence rule to deconflict. > we're taking a shortcut by not implementing full schema support for this feature What does the "full schema support" mean in this context of conflicting policies for nested fields? Thanks, Steven On Thu, Aug 6, 2026 at 2:43 PM Daniel Weeks <[email protected]> wrote: > Current spec requires clients to *fail* if the catalog server returns >> conflicting policies. If we relax this requirement in the future with >> precedence rules for client-side de-conflicting, older clients would still >> fail. I don't see it as "an unpredictable behavior change" for older >> clients. I would also the relaxation would probably use the wording of >> *MAY* (not *MUST*) applying these precedence rules to deconflict. > > > Relaxing the requirement is not backward compatible for clients. That > would be equivalent to saying that we can make a required field optional > because the clients need to validate required fields so it's ok if they > break. Backward compatibility is about not breaking clients, not making it > ok to break clients. > > I heard comments that this rule is very easy for the policy engine to >> enforce . . . > > > Handling nested fields doesn't change how hard it is for a policy engine > to enforce. They can always fallback to not allowing it (they would > already be required to detect these cases). > > If we ask clients to resolve conflicting policies > > > This makes it sound like we're shifting the burden to the client, but > that's not the case. We are just defining an order in which they are > applied. Policy engines are required to resolve conflicts. The client > just applies the restrictions in a well defined order. > > I would let the policy engine either enforce that conflicting/ambiguous >> policies are not accepted or perform the deconflicting/precedence rules >> before returning to the client. > > > This is true regardless. The client is not involved in resolving anything. > > The issue here is that we're taking a shortcut by not implementing full > schema support for this feature and presenting it as a reasonable path > based on a misconception that it's backward compatible. > > -Dan > > On Thu, Aug 6, 2026 at 11:44 AM Steven Wu <[email protected]> wrote: > >> > Changing that in the future would break older clients that validate >> and result in an unpredictable changes in behior of clients that assume >> correctness from the catalog. >> >> Current spec requires clients to *fail* if the catalog server returns >> conflicting policies. If we relax this requirement in the future with >> precedence rules for client-side de-conflicting, older clients would still >> fail. I don't see it as "an unpredictable behavior change" for older >> clients. I would also the relaxation would probably use the wording of >> *MAY* (not *MUST*) applying these precedence rules to deconflict. >> >> I agree with Prashanth and Sung that it is hard for Iceberg to force a >> specific behavior considering the divergent behaviors in the industry. >> >> As I mentioned during the sync, I also see it the same as the other >> requirement we put on the catalog server. I heard comments that this rule >> is very easy for the policy engine to enforce. If we ask clients to resolve >> conflicting policies for nested fields with precedence rules, I can argue >> the same thing here for this rule. Can't we also ask clients to define >> resolution precedence with the projection policies? >> 3. A server must not return more than one projection for the >> same field-id >> in required-column-projections. If a duplicate field-id >> appears, the reader >> must fail the query. >> >> Since IRC doesn't define policy management APIs and behaviors, I would >> let the policy engine either enforce that conflicting/ambiguous policies >> are not accepted or perform the deconflicting/precedence rules before >> returning to the client. Hence, i agree with the current spec writing that >> clients must fail in the case of conflicting policies with nested fields. >> >> On Thu, Aug 6, 2026 at 9:40 AM Daniel Weeks <[email protected]> wrote: >> >>> I'd like to clarify the point on backward compatibility claims here >>> first as it indicates a dangerous misunderstanding of compatibility in the >>> REST protocol. The statement that "Forbidding is reversible; precedence >>> isn't. Relaxing a MUST NOT later is backward-compatible" is incorrect, and >>> we need to be very careful regarding backward compatibility. It also >>> creates a false sense of security by deferring this, as it implies the >>> issue can be addressed later without consequence. >>> >>> While changing this requirement in the future is >>> structurally compatible, it is not behaviorally compatible. Given the >>> current spec wording, a responsible client should validate that they do not >>> receive a invalid set of redactions from the catalog as this is prohibited >>> by the spec and there's no guidance on how the client should apply them. >>> Changing that in the future would break older clients that validate and >>> result in an unpredictable changes in behior of clients that assume >>> correctness from the catalog. *This is neither safe nor backward >>> compatible.* >>> >>> I think addressing this now is well within scope and will prevent >>> unnecessary future protocol evolution. Regardless of what type of >>> precedence we define, we already know that some policies (e.g. Redshift) >>> that will express more than we will likely include in the protocol. It's >>> also likely that different policy systems will have conflicting views on >>> this matter. Since we're already taking an opinionated stance and levying >>> requirements on the catalog implementation to align their policy with the >>> protocol, this isn't any more complicated than prohibiting conflicting >>> policy. >>> >>> Read restrictions have a number of unexpected cliffs regarding how >>> policy changes affect what clients can consume (e.g. ABAC policies that >>> update from simple redaction to more contextual redaction) and this is just >>> one more scenario that limits interoperability. >>> >>> The backward compatibility issue further highlights why we need a solid >>> reference implementation as I think it would make the problem clear. I >>> don't want to repeat the challenges we had with scan/plan where we released >>> the protocol with issues that we, fortuntately, were able to address as the >>> reference implementation was built. >>> >>> -Dan >>> >>> On Thu, Aug 6, 2026 at 7:56 AM Sung Yun <[email protected]> wrote: >>> >>>> Thanks Prashant for doing such extensive research on this topic. >>>> >>>> I believe the industry divergence on how nested type policies are >>>> handled is a good indicator that we should hold off on forcing a standard >>>> for nested conflict resolution right now. Defining a standard is a highly >>>> sensitive design decision that deserves its own focused, long-term >>>> community discussion rather than being rushed. >>>> >>>> During our syncs, we’ve discussed several potential evaluation models, >>>> each with distinct trade-offs: >>>> - Outermost wins (single policy is enforced) >>>> - Innermost wins (single policy is enforced) >>>> - Innermost evaluated first (pipelined and layered sequentially) >>>> - Outermost evaluated first (pipelined and layered sequentially) >>>> >>>> These models carry different security implications, and hence I'm in >>>> favor of choosing option A (forbidding overlapping nested policies and >>>> failing-closed on the client) to not rush this discussion and unblock >>>> ReadRestrictions today. >>>> >>>> If the community later decides to define a precedence standard, >>>> relaxing this rule is entirely backward-compatible. On the client/reader >>>> side, we would simply be introducing a new handling capability for what >>>> used to be an invalid response. On the server/catalog side, the timeline >>>> remains entirely in the catalog administrator's control because the new >>>> behavior only triggers once they choose to update their catalog >>>> implementations to emit multiple nested policies. >>>> >>>> One final observation on the industry examples: >>>> Even in Redshift, which is the only example where multiple policies can >>>> technically be registered on a nested path, the engine treats overlapping >>>> policies as a conflict. It resolves this conflict by enforcing only the >>>> single, highest-priority policy at query time. In practice, this >>>> "highest-priority wins" model is highly consistent with Option A. It >>>> already assumes that the system must ultimately resolve the conflict down >>>> to a single, clean policy rather than attempting to chain and layer >>>> multiple policies sequentially. I believe that this can be the >>>> responsibility of the policy engine, rather than becoming a preemptive >>>> standard in the ReadRestrictions spec. >>>> >>>> All in all, I strongly support keeping the current restrictive >>>> instruction in the spec (Option A) for the initial release and updating the >>>> words with normative RFC 2119 terminology (using MUST NOT and MUST fail) to >>>> clearly define the behavior on the client and server side. [1] >>>> >>>> Sung >>>> >>>> [1] https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 >>>> >>>> On 2026/08/04 23:25:40 Prashant Singh wrote: >>>> > Hi all, >>>> > >>>> > One open thread on the Read Restrictions spec PR [1] is the last item >>>> I'd >>>> > like to settle before calling a vote [2]. The last sync leaned >>>> towards the >>>> > restrictive option, but there was an ask by Dan that the alternative >>>> be >>>> > explored properly first, so I'm bringing it here. >>>> > >>>> > The question >>>> > >>>> > required-column-projections binds an action to a field-id, and a >>>> nested >>>> > type has a field-id for the container and separate ones beneath it >>>> (struct >>>> > subfields; list element; map key and value). >>>> > >>>> > Concretely, with 2: address struct<3: street string, 4: city string>: >>>> > >>>> > - field-id: 2 action: mask-to-fixed-value >>>> > >>>> > - field-id: 4 action: replace-with-null >>>> > >>>> > mask-to-fixed-value overwrites everything beneath the container, so it >>>> > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore >>>> > returns {"street": >>>> > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": >>>> null, >>>> > "city": "XXXXXXXX"}, though the catalog asked for the whole struct to >>>> be >>>> > masked. >>>> > >>>> > if a catalog returns two projections (one on the outer container and >>>> one on >>>> > a field-id below it), how should clients handle it? >>>> > >>>> > (A) forbid the overlap; the existing fail-closed rule then makes a >>>> reader >>>> > >>>> > that receives one fail the query. The spec PR currently says: >>>> > >>>> > If a projection targets a nested-typed field (struct, list, or map), >>>> other >>>> > projections in the same ReadRestrictions must not target any nested >>>> > field-id (struct subfields, list elements, or >>>> > >>>> > map keys/values) at any depth. This specification does not define how >>>> such >>>> > actions combine. >>>> > >>>> > A reader that receives such a response must fail the query. >>>> > >>>> > (B) define precedence (e.g. outer-most wins) and allow it >>>> > >>>> > Why current spec PR choose option A >>>> > >>>> > No semantics to borrow. There are different behaviors in the current >>>> > industry practice >>>> > >>>> > - >>>> > >>>> > BigQuery: >>>> > - >>>> > >>>> > "Policy tags can't be applied to columns that use the STRUCT >>>> data >>>> > type" [3]; >>>> > - >>>> > >>>> > Redshift both experience based on data-types: >>>> > - >>>> > >>>> > "You can only apply masking policies to scalar values on the >>>> SUPER >>>> > path" [4] rejects the overlap. >>>> > - >>>> > >>>> > Redshift is the only one that lets both be expressed, and it >>>> calls >>>> > the pair a conflict, resolved by an admin-supplied priority [5] >>>> > - >>>> > >>>> > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, >>>> Impala and >>>> > Ranger define no parent-vs-nested resolution at all, and where the >>>> question >>>> > has been raised it has stayed open: RANGER-3525 since 2021 [6], >>>> HIVE-22823 >>>> > [7], trino#16964 [8]. >>>> > - >>>> > >>>> > SQL:2016 defines nothing - <privilege column list> is a flat >>>> column name >>>> > list. >>>> > >>>> > >>>> > >>>> > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT later >>>> is >>>> > backward-compatible: invalid responses become valid, and no reader >>>> changes >>>> > what it returns for a response that was already legal. If we define >>>> > outer-most-wins now and later want the inner action to contribute, >>>> we've >>>> > silently changed the values a compliant reader returns for the same >>>> > response. >>>> > >>>> > On the future-actions concern: I don't think the prohibition limits >>>> any >>>> > action. A future action can target a container, or a nested field - >>>> just >>>> > not both in one response, which is the only combination with no >>>> defined >>>> > meaning today. >>>> > >>>> > Asks >>>> > >>>> > 1. Opinions on option A (forbid) vs B (define precedence) >>>> > >>>> > 2. If (B), is outer-most-wins the rule — and are we comfortable that >>>> > >>>> > nothing in the current action set can show it’s the right choice? >>>> > >>>> > I am looking forward to hear what other people think and if they have >>>> > objection to the current proposed spec >>>> > >>>> > We time-boxed this to the next sync (Aug 18) in the last one. If >>>> there's no >>>> > further input by then, we will keep the current restrictive wording >>>> and say >>>> > so in this thread rather than treat silence as agreement - the >>>> precedence >>>> > question should be closed explicitly, not by default. And if a use >>>> case >>>> > turns up after we ship, Option A lets us relax the rule compatibility, >>>> > which is most of why the PR takes it. Notes from sync [10]. >>>> > >>>> > Thanks, >>>> > >>>> > Prashant >>>> > >>>> > References >>>> > >>>> > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 >>>> > >>>> > [2] Review thread: >>>> > https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 >>>> > >>>> > [3] BigQuery, column data masking: >>>> > https://cloud.google.com/bigquery/docs/column-data-masking-intro >>>> > >>>> > [4] Redshift, DDM with SUPER paths: >>>> > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html >>>> > >>>> > [5] Redshift, DDM policy hierarchy: >>>> > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html >>>> > >>>> > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 >>>> > >>>> > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 >>>> > >>>> > [8] trinodb/trino#16964: >>>> https://github.com/trinodb/trino/issues/16964 >>>> > >>>> > [9] IMPALA-15130: https://issues.apache.org/jira/browse/IMPALA-15130 >>>> > >>>> > [10] Note from ReadRestrictions syncs: >>>> > >>>> https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx >>>> > >>>> >>>
