nevzheng commented on PR #13879: URL: https://github.com/apache/iceberg/pull/13879#issuecomment-5090768690
+1 Agreed on proposed wording direction. I’m glad we researched this. Excellent work! On Sun, Jul 26, 2026 at 10:31 PM Prashant Singh ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In open-api/rest-catalog-open-api.yaml > <https://github.com/apache/iceberg/pull/13879#discussion_r3654614778>: > > > + > + These restrictions apply only to the authenticated principal, user, or account > + associated with the request. They MUST NOT be interpreted as global policy and > + MUST NOT be applied beyond the entity identified by the Authentication header > + (or other applicable authentication mechanism). > + > + If both properties are absent or empty, the ReadRestrictions object imposes no > + restrictions and is equivalent to the field being absent from the response. > + A server MUST NOT return an action for a column whose type is not listed in > + that action's "Applicable to" set. > + For all actions, if the input column value is NULL, the output MUST be NULL. > + > + If a column projection targets a struct-typed field, other column projections > + in the same ReadRestrictions MUST NOT target any of that struct's subfields > + (at any depth). This avoids ambiguity about which action governs a given > + leaf value. > > I spent a fair amount of time researching how other systems handle this > before answering, and the short version is that I couldn't find an > established semantics to borrow — which is why I'd rather not define a > resolution here yet. > > Every system I looked at either makes the container un-targetable — > BigQuery: "Policy tags can't be applied to columns that use the STRUCT data > type, but they can be associated with the leaf fields of such columns" [1]; > Redshift: "You can only apply masking policies to scalar values on the > SUPER path. You can't apply masking policies to complex structures or > arrays" [2] — or it rejects the overlap outright. I also checked Snowflake, > Databricks, Oracle, Db2, Vertica, Trino, Hive, Impala and Ranger; none of > them define a parent-vs-nested-field resolution for masks. > > Redshift is the only engine I found that lets both be expressed. It > formally defines the pair as a conflict — "the SUPER paths a.b and a.b.c > conflict because they are on the same path, with a.b being the parent of > a.b.c" [2] — and resolves it with an admin-supplied integer, where "only > the highest priority attachment takes effect" [3]. Critically, it "can't > attach two different policies to the same column with equal priority", and > the default priority is 0 [4]. We have no priority concept, so Redshift's > own behaviour in our situation is to reject. > > The question is also unresolved everywhere it's been raised: RANGER-3525 > "Clarify handling of column masks on nested types" has been open since 2021 > [5], HIVE-22823 likewise [6], and Trino #16964 > <https://github.com/apache/iceberg/pull/16964> is open with a reviewer > explicitly confirming that ANSI SQL defines nothing for nested column > access control [7]. I checked the SQL:2016 Foundation grammar directly — > resolves to a flat , and MASK, MASKED, POLICY and ROW LEVEL don't appear > anywhere in it [8]. > > Given that, forbidding is the reversible choice and defining precedence is > not. If we say MUST NOT now and later find a real use case, relaxing it is > backward-compatible — responses that were invalid become valid. If we > define "outer-most wins" now and later decide the inner action should > contribute, we've silently changed the values a compliant reader returns > for the same response. > > On the future-actions concern specifically — I don't think the prohibition > limits any individual action. A future action can still target a container, > or target a nested field; it just can't do both in the same response. Only > the combination is ruled out, and that combination has no defined meaning > today regardless. > > What I do think is wrong is the current justification. It claims this > "avoids ambiguity about which action governs a given leaf value," and that > isn't accurate: with today's action set there is no ambiguity, because > replace-with-null and mask-to-fixed-value are the only actions applicable > to a nested type and both fully overwrite everything beneath > (mask-to-fixed-value on a struct sets "each field ... to its type-specific > default (applied recursively)", on a list yields [], on a map yields {}) > [9]. So I'd like the wording to say what we actually mean: > > If a projection targets a nested-typed field (struct, list, or map), other > projections in the same ReadRestrictions must not target any nested > field-id (struct subfields, list elements, or map keys/values) at any > depth. This specification does not define how such actions combine; the > restriction is reserved and may be relaxed in a future revision. A reader > that receives such a response must fail the query. > > The last sentence closes a gap I noticed while re-reading: the > duplicate-field-id rule (item 3) tells the reader tfail, but this rule only > constrains the servertion is the failure mode I'd most want ruledout — > Impala shipped exactly that for container masks and changed it to > fail-closed in 5.0.0 precisely because it "returned the column unmasked, > leaking data the" [10]. > > References > > [1] BigQuery — Introduction to column data mas > https://docs.cloud.google.com/bigquery/docs/co > [2] Amazon Redshift — Using dynamic data maskiaths. > https://docs.aws.amazon.com/redshift/latest/dg > [3] Amazon Redshift — Dynamic data masking polhttps:// > docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html > [4] Amazon Redshift — ATTACH MASKING POLICY. > https://docs.aws.amazon.com/redshift/latest/dghtml > [5] RANGER-3525 — Clarify handling of column masks on nested types (open > since 2021-11-24). https://issues.apache.org/jira/browse/RANGER-3525 > [6] HIVE-22823 — Support column masking policy. > https://issues.apache.org/jira/browse/HIVE-22823 > [7] trinodb/trino#16964 <https://github.com/trinodb/trino/issues/16964> — > Add support of access control for nested columns (open). > trinodb/trino#16964 <https://github.com/trinodb/trino/issues/16964> > [8] SQL:2016 Foundation grammar (extracted BNF), §12 . > https://jakewheat.github.io/sql-overview/sql-2016-foundation-grammar.html > [9] This PR — MaskToFixedValue and ReplaceWithnitions, > open-api/rest-catalog-open-api.yamllines 3805–3854. > [10] IMPALA-15130 — Raise error when top-levelmasking (fixed, Impala > 5.0.0).https://issues.apache.org/jira/browse/IMPALA-15130 > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/iceberg/pull/13879?email_source=notifications&email_token=ABR2SI6BP2W6HEFYMZ6SBZ35G3SJ3A5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYGM4DKOJXGY22M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#discussion_r3654614778>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABR2SI52557SWWUAESJJMZD5G3SJ3AVCNFSNUABFKJSXA33TNF2G64TZHMYTKOBSGU3DINZZHNEXG43VMU5TGMZTHE3DAOBQGQ4KC5QC> > . > You are receiving this because you commented.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
