huaxingao commented on code in PR #17822:
URL: https://github.com/apache/iceberg/pull/17822#discussion_r4355138961
##########
format/spec.md:
##########
@@ -1144,13 +1253,14 @@ A snapshot consists of the following fields:
| _required_ | **`first-row-id`** | The first `_row_id` assigned
to the first row in the first data file in the first manifest, see [Row
Lineage](#row-lineage) |
| _required_ | **`added-rows`** | The upper bound of the
number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| _optional_ | **`key-id`** | ID of the encryption key
that encrypts the root manifest key metadata |
+ | _optional_ | **`constraint-statuses`** | A struct that records the
status of the table's constraints for the snapshot, see [Constraint
Validation](#constraint-validation) |
When committed while the table is at version 4 or higher, a snapshot's
`timestamp-ms` must be strictly greater than its parent snapshot's
`timestamp-ms` on the same branch. Snapshots with no parent, and snapshots that
were committed before a table was upgraded to v4 or higher, do not have this
requirement.
The snapshot summary's `operation` field is used by some operations, like
snapshot expiration, to skip processing certain snapshots. Possible `operation`
values are:
* `append` -- Only data files were added and no files were removed.
-* `replace` -- Data and delete files were added and removed without changing
table data; i.e., compaction, changing the data file format, or relocating data
files.
+* `replace` -- Metadata, data, and delete files were added and removed without
changing table data; i.e., compaction, changing the data file format, or
relocating data files.
Review Comment:
Reverted
##########
format/spec.md:
##########
@@ -693,6 +696,112 @@ Sorting floating-point numbers should produce the
following behavior: `-NaN` < `
A data or delete file is associated with a sort order by the sort order's id
within [a manifest](#manifests). Therefore, the table must declare all the sort
orders for lookup. A table could also be configured with a default sort order
id, indicating how the new data should be sorted by default. Writers should use
this default sort order to sort the data on write, but are not required to if
the default order is prohibitively expensive, as it would be for streaming
writes.
+### Constraints
+
+Constraints are added in v4 and are not supported in v3 or earlier.
+
+A **constraint** declares a condition that a table's live rows are expected to
satisfy. Its definition is stored in table metadata. Each snapshot contains the
writer-reported status of each constraint that existed when the snapshot was
created; see [Constraint Validation](#constraint-validation).
+
+Three constraint types are defined:
+
+* `check` -- every row must satisfy a predicate
+* `unique` -- the values of a set of fields must be distinct across the rows
in which every field in the set is non-null; a row in which any field in the
set is null is not compared, so more than one such row may exist
+* `primary-key` -- the values of a set of fields must be distinct across all
rows and must not be null
+
+A constraint is stored separately from the table schema and must refer to
fields by field ID. Constraints may evolve independently from the schema, being
added, modified, or removed.
+
+#### Constraint Fields
+
+A constraint consists of the following fields:
+
+| Requirement | Field name | Type | Description |
+|-------------|---------------------------|-----------|-------------|
+| _required_ | **`constraint-id`** | `int` | ID of the constraint;
unique within the table |
+| _required_ | **`type`** | `string` | The constraint type:
`check`, `unique`, or `primary-key` |
+| _required_ | **`name`** | `string` | A name for the
constraint that is unique within the table. Names are for human consumption and
must not be used to identify a constraint in metadata |
+| _required_ | **`enforced`** | `boolean` | Whether writers must
only add or modify rows so that they satisfy the constraint |
+| _required_ | **`timestamp-ms`** | `long` | Timestamp in
milliseconds from the unix epoch when the constraint was created or last
modified. The timestamp is informational and must not be used to determine
whether a constraint applies to a snapshot or whether it holds |
+| _optional_ | **`expression`** | `expression` | The predicate that
every row must satisfy, see [Check Constraint
Expressions](#check-constraint-expressions). Required for a `check` constraint
and must not be set for other types |
+| _optional_ | **`field-ids`** | `list<int>` | The set of field IDs
that the constraint applies to. Must contain at least one field ID and no
duplicates. Required for a `unique` or `primary-key` constraint and must not be
set for a `check` constraint |
+
+The `field-ids` of a `unique` or `primary-key` constraint must only reference
primitive fields and must not reference fields within a `list` or a `map`.
Fields of type `float` or `double` must not be used. Fields of type `geometry`
or `geography` must not be used because Iceberg does not define equality for
them. Fields of type `unknown` must not be used because their values are always
null. Fields of type `file` and their sub-fields must not be used.
+
+The fields of a `primary-key` constraint must be `required` and must not be
nested in an optional struct, so that the schema rules out null keys even when
the constraint is not enforced. The fields of a `unique` constraint may be
optional or nested in optional structs, because a row in which any field is
null is not compared.
+
+When a constraint is `enforced`, writers must ensure that written data conform
to the constraint. When a constraint is not enforced, writers are not required
to ensure that written data conform to the constraint.
+
+When a commit is retried against a new parent snapshot, every enforced
constraint, including any that a concurrent commit added or changed to
enforced, must hold for the rows that the commit adds or modifies in the
snapshot that results from applying the commit to the new parent.
`constraint-statuses` must be recomputed from the new parent.
+
+Readers may trust constraints even if not marked as enforced.
+
+A table may have at most one `primary-key` constraint. A `primary-key`
constraint with `enforced=false` replaces [identifier field
IDs](#identifier-field-ids), which express the same concept: a set of fields
that identifies a row, which is not enforced.
+
+When a table is upgraded to v4, the `identifier-field-ids` of the table's
current schema are rewritten as a single `primary-key` constraint that is not
enforced. The constraint is assigned a `constraint-id` from
`last-constraint-id` in the same way as any other constraint, its `name` is
`pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields
cannot be used in a `primary-key` constraint, such as a `geometry` or
`geography` field, no constraint is created and the upgrade continues without
it. Writers must not set `identifier-field-ids` in a v4 table schema, and
readers must ignore `identifier-field-ids` in a v4 table.
+
+Constraint IDs are assigned from the table's `last-constraint-id`, which is
treated as 0 when it is not present. Writers must assign a new constraint an ID
that is higher than the table's current `last-constraint-id` and must update
`last-constraint-id` to the highest assigned ID. Constraint IDs must not be
reused after the constraint that used an ID is removed, because retained
snapshots may still reference the removed ID. Readers must not assume that
every `constraint-id` referenced by a snapshot is present in `constraints`.
+
+#### Check Constraint Expressions
+
+The `expression` of a `check` constraint is a
[predicate](expressions-spec.md#predicates), not a value expression, and is
serialized as described in the [Iceberg expressions spec](expressions-spec.md).
It must use ID references so that it remains bound to the same fields when
columns are renamed.
+
+A check expression is evaluated for each row over the values of that row. An
expression may reference more than one field of the row, such as `start_date <=
end_date`. A check expression:
+
+* Must not depend on more than one row, such as aggregates over rows and
window functions
+* Must not depend on another table, such as subqueries
+* Must not reference a field within a `list` or a `map`, because such a field
has a value for each element rather than one value for the row; it may pass a
whole `list` or `map` field to a function, for example to check that a list is
not empty
+
+A check expression must be pure. Its result must depend only on the values of
the fields it references in the row, and evaluating it must have no side
effects.
+
+A [user-defined function](udf-spec.md) used in a check expression must declare
`deterministic` as true and is expected to preserve its behavior across
definition versions. Readers and writers are not required to detect behavioral
changes or recompute `constraint-statuses` if such a change occurs.
+
+#### Constraints and Schema Evolution
+
+A constraint references fields by ID, so schema changes interact with
constraints as follows. The referenced fields of a `check` constraint are the
field IDs in its `expression`; the referenced fields of a `unique` or
`primary-key` constraint are its `field-ids`.
+
+* Renaming or reordering a constraint-referenced field is allowed; the
constraint continues to apply to the same fields.
+* Every field referenced by a constraint must exist in the table's current
schema. A schema change that removes a constraint-referenced field must be
rejected unless the constraint is removed in the same change.
+* A field referenced by a `check` constraint must not change type or apply
type promotion, because the constraint's `expression` contains literals that
are bound to the field's type. A field referenced by a `unique` or
`primary-key` constraint may apply type promotions that preserve distinctness.
+
+Only `name` and `enforced` fields in a constraint definition can be updated,
and `timestamp-ms` must be updated when the constraint is updated. The
`expression` and `field-ids` fields must not change, as it would change the
constraint requirement. Such updates must be done by removing the constraint
and adding a new one with a new `constraint-id`. A new `constraint-id` ensures
that statuses recorded for the old definition are not applied to the new one.
+
+#### Constraint Validation
+
+Enforcement of a constraint is controlled by the `enforced` field within the
constraint definition. When true, writers must verify that the rows that they
add or modify follow the constraint.
+
+The validation status of each constraint is tracked within snapshot metadata
in the `constraint-statuses` field.
+
+The status of a constraint for a snapshot is one of:
+
+| Status | Description |
+|---------------|-------------|
+| `validated` | The constraint was checked and holds for all rows in the
snapshot, or the commit did not add or modify rows and the parent snapshot's
status is `validated` |
+| `valid` | The parent snapshot's status is `validated` or `valid`, and
either the writer can prove that the rows it added or modified follow the
constraint or the commit did not add or modify rows. By induction, the
constraint holds for all rows in the snapshot. |
+| `invalid` | At least one row in the snapshot is known to violate the
constraint |
+| `unvalidated` | Whether the constraint holds for all rows in the snapshot is
not known |
+
+A snapshot's `constraint-statuses` records, for each status, the IDs of the
constraints that have that status for the snapshot:
+
+| Requirement | Field name | Type | Description |
+|-------------|-------------------|-------------|-------------|
+| _optional_ | **`validated`** | `list<int>` | IDs of constraints that are
`validated` for the snapshot |
+| _optional_ | **`valid`** | `list<int>` | IDs of constraints that are
`valid` for the snapshot |
+| _optional_ | **`invalid`** | `list<int>` | IDs of constraints that are
`invalid` for the snapshot |
+| _optional_ | **`unvalidated`** | `list<int>` | IDs of constraints that are
`unvalidated` for the snapshot |
+
+Each list contains the `constraint-id` of every constraint that has that
status for the snapshot. Every constraint that exists when the snapshot is
created must be listed in exactly one of the four lists. A list with no
constraints may be omitted. A constraint whose ID is not present in any list
did not exist when the snapshot was created, so it may be treated as
`unvalidated` for the snapshot.
Review Comment:
Done. Thanks
##########
format/spec.md:
##########
@@ -693,6 +696,112 @@ Sorting floating-point numbers should produce the
following behavior: `-NaN` < `
A data or delete file is associated with a sort order by the sort order's id
within [a manifest](#manifests). Therefore, the table must declare all the sort
orders for lookup. A table could also be configured with a default sort order
id, indicating how the new data should be sorted by default. Writers should use
this default sort order to sort the data on write, but are not required to if
the default order is prohibitively expensive, as it would be for streaming
writes.
+### Constraints
+
+Constraints are added in v4 and are not supported in v3 or earlier.
+
+A **constraint** declares a condition that a table's live rows are expected to
satisfy. Its definition is stored in table metadata. Each snapshot contains the
writer-reported status of each constraint that existed when the snapshot was
created; see [Constraint Validation](#constraint-validation).
+
+Three constraint types are defined:
+
+* `check` -- every row must satisfy a predicate
+* `unique` -- the values of a set of fields must be distinct across the rows
in which every field in the set is non-null; a row in which any field in the
set is null is not compared, so more than one such row may exist
+* `primary-key` -- the values of a set of fields must be distinct across all
rows and must not be null
+
+A constraint is stored separately from the table schema and must refer to
fields by field ID. Constraints may evolve independently from the schema, being
added, modified, or removed.
+
+#### Constraint Fields
+
+A constraint consists of the following fields:
+
+| Requirement | Field name | Type | Description |
+|-------------|---------------------------|-----------|-------------|
+| _required_ | **`constraint-id`** | `int` | ID of the constraint;
unique within the table |
+| _required_ | **`type`** | `string` | The constraint type:
`check`, `unique`, or `primary-key` |
+| _required_ | **`name`** | `string` | A name for the
constraint that is unique within the table. Names are for human consumption and
must not be used to identify a constraint in metadata |
+| _required_ | **`enforced`** | `boolean` | Whether writers must
only add or modify rows so that they satisfy the constraint |
+| _required_ | **`timestamp-ms`** | `long` | Timestamp in
milliseconds from the unix epoch when the constraint was created or last
modified. The timestamp is informational and must not be used to determine
whether a constraint applies to a snapshot or whether it holds |
+| _optional_ | **`expression`** | `expression` | The predicate that
every row must satisfy, see [Check Constraint
Expressions](#check-constraint-expressions). Required for a `check` constraint
and must not be set for other types |
+| _optional_ | **`field-ids`** | `list<int>` | The set of field IDs
that the constraint applies to. Must contain at least one field ID and no
duplicates. Required for a `unique` or `primary-key` constraint and must not be
set for a `check` constraint |
+
+The `field-ids` of a `unique` or `primary-key` constraint must only reference
primitive fields and must not reference fields within a `list` or a `map`.
Fields of type `float` or `double` must not be used. Fields of type `geometry`
or `geography` must not be used because Iceberg does not define equality for
them. Fields of type `unknown` must not be used because their values are always
null. Fields of type `file` and their sub-fields must not be used.
+
+The fields of a `primary-key` constraint must be `required` and must not be
nested in an optional struct, so that the schema rules out null keys even when
the constraint is not enforced. The fields of a `unique` constraint may be
optional or nested in optional structs, because a row in which any field is
null is not compared.
+
+When a constraint is `enforced`, writers must ensure that written data conform
to the constraint. When a constraint is not enforced, writers are not required
to ensure that written data conform to the constraint.
+
+When a commit is retried against a new parent snapshot, every enforced
constraint, including any that a concurrent commit added or changed to
enforced, must hold for the rows that the commit adds or modifies in the
snapshot that results from applying the commit to the new parent.
`constraint-statuses` must be recomputed from the new parent.
+
+Readers may trust constraints even if not marked as enforced.
+
+A table may have at most one `primary-key` constraint. A `primary-key`
constraint with `enforced=false` replaces [identifier field
IDs](#identifier-field-ids), which express the same concept: a set of fields
that identifies a row, which is not enforced.
+
+When a table is upgraded to v4, the `identifier-field-ids` of the table's
current schema are rewritten as a single `primary-key` constraint that is not
enforced. The constraint is assigned a `constraint-id` from
`last-constraint-id` in the same way as any other constraint, its `name` is
`pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields
cannot be used in a `primary-key` constraint, such as a `geometry` or
`geography` field, no constraint is created and the upgrade continues without
it. Writers must not set `identifier-field-ids` in a v4 table schema, and
readers must ignore `identifier-field-ids` in a v4 table.
+
+Constraint IDs are assigned from the table's `last-constraint-id`, which is
treated as 0 when it is not present. Writers must assign a new constraint an ID
that is higher than the table's current `last-constraint-id` and must update
`last-constraint-id` to the highest assigned ID. Constraint IDs must not be
reused after the constraint that used an ID is removed, because retained
snapshots may still reference the removed ID. Readers must not assume that
every `constraint-id` referenced by a snapshot is present in `constraints`.
+
+#### Check Constraint Expressions
+
+The `expression` of a `check` constraint is a
[predicate](expressions-spec.md#predicates), not a value expression, and is
serialized as described in the [Iceberg expressions spec](expressions-spec.md).
It must use ID references so that it remains bound to the same fields when
columns are renamed.
+
+A check expression is evaluated for each row over the values of that row. An
expression may reference more than one field of the row, such as `start_date <=
end_date`. A check expression:
+
+* Must not depend on more than one row, such as aggregates over rows and
window functions
+* Must not depend on another table, such as subqueries
+* Must not reference a field within a `list` or a `map`, because such a field
has a value for each element rather than one value for the row; it may pass a
whole `list` or `map` field to a function, for example to check that a list is
not empty
+
+A check expression must be pure. Its result must depend only on the values of
the fields it references in the row, and evaluating it must have no side
effects.
+
+A [user-defined function](udf-spec.md) used in a check expression must declare
`deterministic` as true and is expected to preserve its behavior across
definition versions. Readers and writers are not required to detect behavioral
changes or recompute `constraint-statuses` if such a change occurs.
+
+#### Constraints and Schema Evolution
+
+A constraint references fields by ID, so schema changes interact with
constraints as follows. The referenced fields of a `check` constraint are the
field IDs in its `expression`; the referenced fields of a `unique` or
`primary-key` constraint are its `field-ids`.
+
+* Renaming or reordering a constraint-referenced field is allowed; the
constraint continues to apply to the same fields.
+* Every field referenced by a constraint must exist in the table's current
schema. A schema change that removes a constraint-referenced field must be
rejected unless the constraint is removed in the same change.
+* A field referenced by a `check` constraint must not change type or apply
type promotion, because the constraint's `expression` contains literals that
are bound to the field's type. A field referenced by a `unique` or
`primary-key` constraint may apply type promotions that preserve distinctness.
+
+Only `name` and `enforced` fields in a constraint definition can be updated,
and `timestamp-ms` must be updated when the constraint is updated. The
`expression` and `field-ids` fields must not change, as it would change the
constraint requirement. Such updates must be done by removing the constraint
and adding a new one with a new `constraint-id`. A new `constraint-id` ensures
that statuses recorded for the old definition are not applied to the new one.
+
+#### Constraint Validation
+
+Enforcement of a constraint is controlled by the `enforced` field within the
constraint definition. When true, writers must verify that the rows that they
add or modify follow the constraint.
+
+The validation status of each constraint is tracked within snapshot metadata
in the `constraint-statuses` field.
+
+The status of a constraint for a snapshot is one of:
+
+| Status | Description |
+|---------------|-------------|
+| `validated` | The constraint was checked and holds for all rows in the
snapshot, or the commit did not add or modify rows and the parent snapshot's
status is `validated` |
+| `valid` | The parent snapshot's status is `validated` or `valid`, and
either the writer can prove that the rows it added or modified follow the
constraint or the commit did not add or modify rows. By induction, the
constraint holds for all rows in the snapshot. |
+| `invalid` | At least one row in the snapshot is known to violate the
constraint |
+| `unvalidated` | Whether the constraint holds for all rows in the snapshot is
not known |
+
+A snapshot's `constraint-statuses` records, for each status, the IDs of the
constraints that have that status for the snapshot:
+
+| Requirement | Field name | Type | Description |
+|-------------|-------------------|-------------|-------------|
+| _optional_ | **`validated`** | `list<int>` | IDs of constraints that are
`validated` for the snapshot |
+| _optional_ | **`valid`** | `list<int>` | IDs of constraints that are
`valid` for the snapshot |
+| _optional_ | **`invalid`** | `list<int>` | IDs of constraints that are
`invalid` for the snapshot |
+| _optional_ | **`unvalidated`** | `list<int>` | IDs of constraints that are
`unvalidated` for the snapshot |
+
+Each list contains the `constraint-id` of every constraint that has that
status for the snapshot. Every constraint that exists when the snapshot is
created must be listed in exactly one of the four lists. A list with no
constraints may be omitted. A constraint whose ID is not present in any list
did not exist when the snapshot was created, so it may be treated as
`unvalidated` for the snapshot.
+
+A `constraint-id` listed in a snapshot that is not in the table's
`constraints` refers to a constraint that no longer exists and must be ignored.
Review Comment:
Done. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
