huaxingao commented on code in PR #17822: URL: https://github.com/apache/iceberg/pull/17822#discussion_r4291910175
########## format/spec.md: ########## @@ -661,6 +664,111 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a condition that a table's live rows, after applying deletes, are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the values of a set of fields must be distinct across the rows in which every field in the set is non-null; a row in which any field in the set is null is not compared, so more than one such row may exist +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +A constraint is stored separately from the table schema and must refer to fields by field ID. Constraints may evolve independently from the schema, being added, modified, or removed. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The set of field IDs that the constraint applies to. Must contain at least one field ID and no duplicates. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The `field-ids` of a `unique` or `primary-key` constraint must only reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float` or `double` must not be used. Fields of type `geometry` or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. + +The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. + +When a constraint is `enforced`, writers must ensure that written data conform to the constraint. When a constraint is not enforced, writers are not required to ensure that written data conform to the constraint. + +For constraints, a row whose values changed in a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). + +When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. Review Comment: Agreed. I rewrote it as a short requirement. ########## format/spec.md: ########## @@ -661,6 +664,111 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a condition that a table's live rows, after applying deletes, are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the values of a set of fields must be distinct across the rows in which every field in the set is non-null; a row in which any field in the set is null is not compared, so more than one such row may exist +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +A constraint is stored separately from the table schema and must refer to fields by field ID. Constraints may evolve independently from the schema, being added, modified, or removed. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The set of field IDs that the constraint applies to. Must contain at least one field ID and no duplicates. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The `field-ids` of a `unique` or `primary-key` constraint must only reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float` or `double` must not be used. Fields of type `geometry` or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. + +The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. + +When a constraint is `enforced`, writers must ensure that written data conform to the constraint. When a constraint is not enforced, writers are not required to ensure that written data conform to the constraint. + +For constraints, a row whose values changed in a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). + +When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. + +If a constraint is not enforced, readers can decide whether to trust the constraint. Review Comment: Done, thanks ########## format/spec.md: ########## @@ -661,6 +664,111 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a condition that a table's live rows, after applying deletes, are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the values of a set of fields must be distinct across the rows in which every field in the set is non-null; a row in which any field in the set is null is not compared, so more than one such row may exist +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +A constraint is stored separately from the table schema and must refer to fields by field ID. Constraints may evolve independently from the schema, being added, modified, or removed. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The set of field IDs that the constraint applies to. Must contain at least one field ID and no duplicates. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The `field-ids` of a `unique` or `primary-key` constraint must only reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float` or `double` must not be used. Fields of type `geometry` or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. + +The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. + +When a constraint is `enforced`, writers must ensure that written data conform to the constraint. When a constraint is not enforced, writers are not required to ensure that written data conform to the constraint. + +For constraints, a row whose values changed in a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). + +When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. + +If a constraint is not enforced, readers can decide whether to trust the constraint. + +A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row. Iceberg never guaranteed uniqueness for identifier fields and did not track whether it held. A `primary-key` constraint makes both expressible, through `enforced` and `constraint-statuses`. Review Comment: Done. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
