stevenzwu commented on code in PR #13879:
URL: https://github.com/apache/iceberg/pull/13879#discussion_r3906294394
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -1051,6 +1051,16 @@ paths:
table. The configuration key "token" is used to pass an access token
to be used as a bearer token
for table requests. Otherwise, a token may be passed using a RFC 8693
token type as a configuration
key. For example, "urn:ietf:params:oauth:token-type:jwt=<JWT-token>".
+
+
+ The response may include a read-restrictions field. A reader that
supports read
+ restrictions must fail any read against the loaded table that cannot
apply a returned
+ restriction in full. This includes unrecognized action or expression
types, and actions
+ whose definition cannot be loaded, parsed, or evaluated. These
restrictions apply to every
+ read performed using this response (including subsequent planTableScan
and fetchScanTasks
+ calls), but subsequent requests or requests made from a different
authentication scope
+ may have different restrictions. The response should not be cached
outside of the
+ authenticated scope.
Review Comment:
wondering if we should be more explicit about `should not be cached outside
of the authenticated scope`.
The `ETag` parameter is still described as identifying a version of table
metadata. If two principals get the same metadata (and thus the same ETag) but
different `read-restrictions. Either:
- ETags for `loadTable` MUST distinguish responses that differ in
`read-restrictions` (and storage credentials), or
- 304 MUST NOT be returned when `read-restrictions` can vary across
authentication scopes.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3839,6 +3849,315 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader that supports read-restrictions cannot apply any returned
+ restriction (a filter expression or an action), it must fail the
query
+ and must not silently return raw, partial, or empty results.
Review Comment:
`cannot apply any returned restriction` is easy to read as "if it can apply
none of them" rather than "if there is a restriction it cannot apply." The
loadTable wording (`cannot apply a returned restriction in full`) is
unambiguous — please use that here.
This also conflicts with required-column-projections rule 5 (unread columns
"do not apply"). If an unrecognized action arrives for a column the query does
not read, does the reader fail (this paragraph) or skip (rule 5)? Pick one and
state it in both places.
Nit: this section uses lowercase `must` while `MaskToFixedValue` uses `MUST
NOT`. Please stick to RFC 2119 keywords throughout.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3839,6 +3849,315 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader that supports read-restrictions cannot apply any returned
+ restriction (a filter expression or an action), it must fail the
query
+ and must not silently return raw, partial, or empty results.
+
+ A server must not return projections on both a nested-typed field
+ (struct, list, or map) and any field-id nested within it at any
+ depth. A reader that receives such a response must fail the query.
+
+ A missing or empty ReadRestrictions object (no
required-column-projections and no
+ required-row-filter) imposes no restrictions.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ left:
+ type: reference
+ id: 14
+ right:
+ type: literal
+ value: "US"
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original
field-id. For
+ example, if the action for field-id `9` is mask-alphanum, the
reader must
+ return the masked value as field-id `9` in the query output.
+
+ 3. A server must not return more than one projection for the same
field-id
+ in required-column-projections. If a duplicate field-id appears,
the reader
+ must fail the query.
+
+ 4. A projection must not target a map's key field-id. Applying an
action
+ to keys can produce duplicate or null keys, which readers
silently
+ coalesce or reject, causing data loss.
+
+ 5. A reader must enforce projections on the columns it is actually
reading.
+ Projections referencing columns that are not being read do not
apply.
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that limits which rows the reader may return.
+
+ 1. The expression must evaluate to a boolean (TRUE or FALSE;
Iceberg predicates
+ never produce NULL). A reader must discard any row for which the
filter
+ evaluates to FALSE, and no information derived from discarded
rows may be
+ included in the query result.
+
+ 2. If this property is absent, null, or always true then no
mandatory filtering is required.
+
+ 3. Column references within the expression must use field IDs
(IdReference),
+ not column names. This ensures the filter remains valid across
column renames,
+ consistent with required-column-projections which also reference
columns by field-id.
+ allOf:
+ - $ref: '#/components/schemas/Predicate'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: Field ID of the column being projected.
+
+ MaskAlphanum:
+ description: >
+ Redacts the column value using the following rules to transform
Unicode code points:
+
+ - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+ - The following punctuation characters are kept as-is:
+ U+0028 '(' LEFT PARENTHESIS
+ U+0029 ')' RIGHT PARENTHESIS
+ U+002C ',' COMMA
+ U+002E '.' FULL STOP
+ U+002D '-' HYPHEN-MINUS
+ U+0040 '@' COMMERCIAL AT
+ - All other Unicode characters (including letters, whitespace, and any
punctuation
+ not listed above) are replaced with 'x'
+
+ For example: "[email protected]" ->
"[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-alphanum"
+
+ MaskToFixedValue:
+ description: >
+ Replaces the column value with a type-specific fixed value.
+ Readers must use exactly the values listed below to ensure consistency
+ across implementations.
+
+ Fixed values by type:
+ - boolean: false
+ - int: 0
+ - long: 0
+ - float: 0.0
+ - double: 0.0
+ - decimal(p, s): 0 (the unscaled value is 0)
+ - string: "XXXXXXXX"
+ - date: 1970-01-01
+ - time: 00:00:00
+ - timestamp: 1970-01-01T00:00:00
+ - timestamptz: 1970-01-01T00:00:00+00:00
+ - timestamp_ns: 1970-01-01T00:00:00.000000000
+ - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00
+ - uuid: 00000000-0000-0000-0000-000000000000
+ - fixed(n): n zero bytes
+ - binary: empty byte sequence
+ - variant: {}
Review Comment:
`{}` is JSON object syntax. Iceberg `variant` is a Parquet binary
(`metadata` + `value`); see
[VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md).
Iceberg also forbids a non-null variant as `initial-default` /
`write-default`, so there is no existing table-spec constant to copy.
Please name the physical value, for example an empty variant object:
- metadata: empty v1 dictionary `[0x01, 0x00, 0x00]`
(`VariantMetadata.empty()`)
- value: empty object `[0x02, 0x00, 0x00]` (`basic_type=OBJECT`,
`num_elements=0`)
Same style as `fixed(n): n zero bytes` and `binary: empty byte sequence`.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3839,6 +3849,315 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader that supports read-restrictions cannot apply any returned
+ restriction (a filter expression or an action), it must fail the
query
+ and must not silently return raw, partial, or empty results.
+
+ A server must not return projections on both a nested-typed field
+ (struct, list, or map) and any field-id nested within it at any
+ depth. A reader that receives such a response must fail the query.
+
+ A missing or empty ReadRestrictions object (no
required-column-projections and no
+ required-row-filter) imposes no restrictions.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ left:
+ type: reference
+ id: 14
+ right:
+ type: literal
+ value: "US"
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original
field-id. For
+ example, if the action for field-id `9` is mask-alphanum, the
reader must
+ return the masked value as field-id `9` in the query output.
+
+ 3. A server must not return more than one projection for the same
field-id
+ in required-column-projections. If a duplicate field-id appears,
the reader
+ must fail the query.
+
+ 4. A projection must not target a map's key field-id. Applying an
action
+ to keys can produce duplicate or null keys, which readers
silently
+ coalesce or reject, causing data loss.
Review Comment:
Pin the actor, matching the nested-overlap rule and rule 3: a server MUST
NOT return a projection that targets a map key field-id; a reader that receives
one MUST fail the query.
"A projection must not" leaves it unclear whether a non-conforming catalog
response is something the reader may ignore.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3839,6 +3849,315 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader that supports read-restrictions cannot apply any returned
+ restriction (a filter expression or an action), it must fail the
query
+ and must not silently return raw, partial, or empty results.
+
+ A server must not return projections on both a nested-typed field
Review Comment:
should we move this paragraph as rule 6 in the required-column-projections
section below.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3839,6 +3849,315 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader that supports read-restrictions cannot apply any returned
+ restriction (a filter expression or an action), it must fail the
query
+ and must not silently return raw, partial, or empty results.
+
+ A server must not return projections on both a nested-typed field
+ (struct, list, or map) and any field-id nested within it at any
+ depth. A reader that receives such a response must fail the query.
+
+ A missing or empty ReadRestrictions object (no
required-column-projections and no
+ required-row-filter) imposes no restrictions.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ left:
+ type: reference
+ id: 14
+ right:
+ type: literal
+ value: "US"
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original
field-id. For
+ example, if the action for field-id `9` is mask-alphanum, the
reader must
+ return the masked value as field-id `9` in the query output.
+
+ 3. A server must not return more than one projection for the same
field-id
+ in required-column-projections. If a duplicate field-id appears,
the reader
+ must fail the query.
+
+ 4. A projection must not target a map's key field-id. Applying an
action
+ to keys can produce duplicate or null keys, which readers
silently
+ coalesce or reject, causing data loss.
+
+ 5. A reader must enforce projections on the columns it is actually
reading.
+ Projections referencing columns that are not being read do not
apply.
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that limits which rows the reader may return.
+
+ 1. The expression must evaluate to a boolean (TRUE or FALSE;
Iceberg predicates
+ never produce NULL). A reader must discard any row for which the
filter
+ evaluates to FALSE, and no information derived from discarded
rows may be
+ included in the query result.
+
+ 2. If this property is absent, null, or always true then no
mandatory filtering is required.
+
+ 3. Column references within the expression must use field IDs
(IdReference),
+ not column names. This ensures the filter remains valid across
column renames,
+ consistent with required-column-projections which also reference
columns by field-id.
+ allOf:
+ - $ref: '#/components/schemas/Predicate'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: Field ID of the column being projected.
+
+ MaskAlphanum:
+ description: >
+ Redacts the column value using the following rules to transform
Unicode code points:
+
+ - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+ - The following punctuation characters are kept as-is:
+ U+0028 '(' LEFT PARENTHESIS
+ U+0029 ')' RIGHT PARENTHESIS
+ U+002C ',' COMMA
+ U+002E '.' FULL STOP
+ U+002D '-' HYPHEN-MINUS
+ U+0040 '@' COMMERCIAL AT
+ - All other Unicode characters (including letters, whitespace, and any
punctuation
+ not listed above) are replaced with 'x'
+
+ For example: "[email protected]" ->
"[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-alphanum"
+
+ MaskToFixedValue:
+ description: >
+ Replaces the column value with a type-specific fixed value.
+ Readers must use exactly the values listed below to ensure consistency
+ across implementations.
+
+ Fixed values by type:
+ - boolean: false
+ - int: 0
+ - long: 0
+ - float: 0.0
+ - double: 0.0
+ - decimal(p, s): 0 (the unscaled value is 0)
+ - string: "XXXXXXXX"
+ - date: 1970-01-01
+ - time: 00:00:00
+ - timestamp: 1970-01-01T00:00:00
+ - timestamptz: 1970-01-01T00:00:00+00:00
+ - timestamp_ns: 1970-01-01T00:00:00.000000000
+ - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00
+ - uuid: 00000000-0000-0000-0000-000000000000
+ - fixed(n): n zero bytes
+ - binary: empty byte sequence
+ - variant: {}
+ - list: empty list []
+ - map: empty map {}
+ - struct: struct with each field set to its type-specific default
(applied recursively)
+
+ NULL input is also replaced with the type-specific fixed value; NULL
is not preserved.
+
+ Applicable to: the types with a fixed value defined above. A catalog
server
+ MUST NOT return mask-to-fixed-value for any other type.
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-to-fixed-value"
+
+ ReplaceWithNull:
+ description: >
+ Replaces the column value with NULL. NULL input is preserved (NULL ->
NULL).
+
+ Applicable to: all optional types. Applying to a required
(non-nullable) column is invalid.
Review Comment:
"is invalid" doesn't say who rejects it. Suggest the same pairing used for
nested overlap: a server MUST NOT return `replace-with-null` for a required
field; a reader that receives it MUST fail the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
