Hi all,
One open thread on the Read Restrictions spec PR [1] is the last item I'd
like to settle before calling a vote [2]. The last sync leaned towards the
restrictive option, but there was an ask by Dan that the alternative be
explored properly first, so I'm bringing it here.
The question
required-column-projections binds an action to a field-id, and a nested
type has a field-id for the container and separate ones beneath it (struct
subfields; list element; map key and value).
Concretely, with 2: address struct<3: street string, 4: city string>:
- field-id: 2 action: mask-to-fixed-value
- field-id: 4 action: replace-with-null
mask-to-fixed-value overwrites everything beneath the container, so it
yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore
returns {"street":
"XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": null,
"city": "XXXXXXXX"}, though the catalog asked for the whole struct to be
masked.
if a catalog returns two projections (one on the outer container and one on
a field-id below it), how should clients handle it?
(A) forbid the overlap; the existing fail-closed rule then makes a reader
that receives one fail the query. The spec PR currently says:
If a projection targets a nested-typed field (struct, list, or map), other
projections in the same ReadRestrictions must not target any nested
field-id (struct subfields, list elements, or
map keys/values) at any depth. This specification does not define how such
actions combine.
A reader that receives such a response must fail the query.
(B) define precedence (e.g. outer-most wins) and allow it
Why current spec PR choose option A
No semantics to borrow. There are different behaviors in the current
industry practice
-
BigQuery:
-
"Policy tags can't be applied to columns that use the STRUCT data
type" [3];
-
Redshift both experience based on data-types:
-
"You can only apply masking policies to scalar values on the SUPER
path" [4] rejects the overlap.
-
Redshift is the only one that lets both be expressed, and it calls
the pair a conflict, resolved by an admin-supplied priority [5]
-
Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, Impala and
Ranger define no parent-vs-nested resolution at all, and where the question
has been raised it has stayed open: RANGER-3525 since 2021 [6], HIVE-22823
[7], trino#16964 [8].
-
SQL:2016 defines nothing - <privilege column list> is a flat column name
list.
Forbidding is reversible; precedence isn't. Relaxing a MUST NOT later is
backward-compatible: invalid responses become valid, and no reader changes
what it returns for a response that was already legal. If we define
outer-most-wins now and later want the inner action to contribute, we've
silently changed the values a compliant reader returns for the same
response.
On the future-actions concern: I don't think the prohibition limits any
action. A future action can target a container, or a nested field - just
not both in one response, which is the only combination with no defined
meaning today.
Asks
1. Opinions on option A (forbid) vs B (define precedence)
2. If (B), is outer-most-wins the rule — and are we comfortable that
nothing in the current action set can show it’s the right choice?
I am looking forward to hear what other people think and if they have
objection to the current proposed spec
We time-boxed this to the next sync (Aug 18) in the last one. If there's no
further input by then, we will keep the current restrictive wording and say
so in this thread rather than treat silence as agreement - the precedence
question should be closed explicitly, not by default. And if a use case
turns up after we ship, Option A lets us relax the rule compatibility,
which is most of why the PR takes it. Notes from sync [10].
Thanks,
Prashant
References
[1] Spec PR: https://github.com/apache/iceberg/pull/13879
[2] Review thread:
https://github.com/apache/iceberg/pull/13879#discussion_r3211673048
[3] BigQuery, column data masking:
https://cloud.google.com/bigquery/docs/column-data-masking-intro
[4] Redshift, DDM with SUPER paths:
https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html
[5] Redshift, DDM policy hierarchy:
https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html
[6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525
[7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823
[8] trinodb/trino#16964: https://github.com/trinodb/trino/issues/16964
[9] IMPALA-15130: https://issues.apache.org/jira/browse/IMPALA-15130
[10] Note from ReadRestrictions syncs:
https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx
