This is an automated email from the ASF dual-hosted git repository.
huaxingao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new a6461374fd Spec: v4 constraints (#17822)
a6461374fd is described below
commit a6461374fd5ebd4c074cd21d9f71cf434ca46bd1
Author: Huaxin Gao <[email protected]>
AuthorDate: Wed Oct 28 16:44:13 2026 -0700
Spec: v4 constraints (#17822)
* Spec: Add table constraints
* nit
* more changes
* fix constraint spec based on the sync
* address comments
* Spec: add constraints and schema evolution section
* clarify constraint status semantics
* clarify constraint validation status semantics
* clarify check expression scope and incremental validation
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* move constraint-statuses into v4 snapshot table
Co-authored-by: Cursor <[email protected]>
* address comments
* Remove unvalidated rule for check constraints that call a UDF
* Assume UDFs do not change in check constraints
* remove redudant sentence
* address comments
* address comments
* Remove unknown constraint type rule
* Address Ryan's comments
* Address review comments
---------
Co-authored-by: Cursor <[email protected]>
---
format/spec.md | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 118 insertions(+), 6 deletions(-)
diff --git a/format/spec.md b/format/spec.md
index 380fcac6ab..b3801b7c22 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -66,6 +66,7 @@ Version 4 of the Iceberg spec restructures metadata for
improved performance and
* Support for [relative locations](#file-locations-in-metadata) in metadata
fields
* Writing new [equality deletes](#equality-delete-files) is no longer allowed
* New data type: `file`
+* Support for table [constraints](#constraints)
The full set of changes are listed in [Appendix E](#version-4).
@@ -420,6 +421,8 @@ Type promotion is not allowed for a field that is
referenced by `source-id` or `
* `date` to `timestamp` or `timestamp_ns`
+The fields referenced by a [constraint](#constraints) are the field IDs in the
`expression` of a `check` constraint, or the `field-ids` of a `unique` or
`primary-key` constraint. Every referenced field must exist in the table's
current schema. A schema change that removes a constraint-referenced field must
be rejected unless the constraint is removed in the same change. A
constraint-referenced field may apply type promotion. For a `unique` or
`primary-key` constraint, the promotion must [...]
+
Any struct, including a top-level schema, can evolve through deleting fields,
adding new fields, renaming existing fields, reordering existing fields, or
promoting a primitive using the valid type promotions. Adding a new field
assigns a new ID for that field and for any nested fields. Renaming an existing
field must change the name, but not the field ID. Deleting a field removes it
from the current schema. Field deletion cannot be rolled back unless the field
was nullable or if the curr [...]
Grouping a subset of a struct’s fields into a nested struct is **not**
allowed, nor is moving fields from a nested struct into its immediate parent
struct (`struct<a, b, c> ↔ struct<a, struct<b, c>>`). Evolving primitive types
to structs is **not** allowed, nor is evolving a single-field struct to a
primitive (`map<string, int> ↔ map<string, struct<int>>`).
@@ -474,6 +477,8 @@ Two rows are the "same"---that is, the rows represent the
same entity---if the i
Identifier fields may be nested in structs but cannot be nested within maps or
lists. Float, double, and optional fields cannot be used as identifier fields
and a nested field cannot be used as an identifier field if it is nested in an
optional struct, to avoid null values in identifiers.
+Identifier field IDs are replaced by a [`primary-key`
constraint](#constraints) in v4 and must not be set in a v4 table schema.
+
#### Reserved Field IDs
Iceberg tables must not use field ids greater than 2147483447
(`Integer.MAX_VALUE - 200`). This id range is reserved for metadata columns
that can be used in user data schemas, like the `_file` column that holds the
file path in which a row was stored.
@@ -693,6 +698,99 @@ Sorting floating-point numbers should produce the
following behavior: `-NaN` < `
A data or delete file is associated with a sort order by the sort order's id
within [a manifest](#manifests). Therefore, the table must declare all the sort
orders for lookup. A table could also be configured with a default sort order
id, indicating how the new data should be sorted by default. Writers should use
this default sort order to sort the data on write, but are not required to if
the default order is prohibitively expensive, as it would be for streaming
writes.
+### Constraints
+
+Constraints are added in v4 and are not supported in v3 or earlier.
+
+A **constraint** declares a condition that a table's live rows are expected to
satisfy. Its definition is stored in table metadata. Each snapshot contains the
writer-reported status of each constraint that existed when the snapshot was
created; see [Constraint Status](#constraint-status).
+
+Three constraint types are defined:
+
+* `check` -- every row must satisfy a predicate
+* `unique` -- the values of a set of fields must be distinct across the rows
in which every field in the set is non-null; a row in which any field in the
set is null is not compared, so more than one such row may exist
+* `primary-key` -- the values of a set of fields must be distinct across all
rows and must not be null
+
+A constraint is stored separately from the table schema and must refer to
fields by field ID. Constraints may evolve independently from the schema, being
added, modified, or removed.
+
+#### Constraint Fields
+
+A constraint consists of the following fields:
+
+| Requirement | Field name | Type | Description |
+|-------------|---------------------------|-----------|-------------|
+| _required_ | **`constraint-id`** | `int` | ID of the constraint;
unique within the table |
+| _required_ | **`type`** | `string` | The constraint type:
`check`, `unique`, or `primary-key` |
+| _required_ | **`name`** | `string` | A name for the
constraint that is unique within the table. Names are for human consumption and
must not be used to identify a constraint in metadata |
+| _required_ | **`enforced`** | `boolean` | Whether writers must
only add or modify rows so that they satisfy the constraint |
+| _required_ | **`timestamp-ms`** | `long` | Timestamp in
milliseconds from the unix epoch when the constraint was created or last
modified. The timestamp is informational and must not be used to determine
whether a constraint applies to a snapshot or whether the constraint holds for
the snapshot |
+| _optional_ | **`expression`** | `expression` | The predicate that
every row must satisfy, using ID references, see [Check Constraint
Expressions](#check-constraint-expressions). Required for a `check` constraint
and must not be set for other types |
+| _optional_ | **`field-ids`** | `list<int>` | The set of field IDs
that the constraint applies to. Must contain at least one field ID and no
duplicates. Required for a `unique` or `primary-key` constraint and must not be
set for a `check` constraint |
+
+The `field-ids` of a `unique` or `primary-key` constraint:
+
+* Must only reference primitive fields
+* Must not reference fields within a `list` or a `map`
+* Must not reference fields of type `float` or `double`
+* Must not reference fields of type `geometry` or `geography`, because Iceberg
does not define equality for them
+* Must not reference fields of type `unknown`, because their values are always
null
+* Must not reference fields of type `file` or their sub-fields
+
+A table may have at most one `primary-key` constraint. The fields of a
`primary-key` constraint must be `required` and must not be nested in an
optional struct, so that the schema rejects null keys even when the constraint
is not enforced. The fields of a `unique` constraint may be optional or nested
in optional structs, because a row in which any field is null is not compared.
+
+When a constraint is `enforced`, writers must ensure that written data conform
to the constraint, and a write that would violate the constraint must fail.
When a constraint is not enforced, writers are not required to ensure that
written data conform to the constraint.
+
+When a table is upgraded to v4, the `identifier-field-ids` of the table's
current schema are rewritten as a single `primary-key` constraint that is not
enforced. The constraint is assigned a `constraint-id` from
`last-constraint-id` in the same way as any other constraint, and its
`timestamp-ms` is the time of the upgrade. If any of those fields cannot be
used in a `primary-key` constraint, such as a `geometry` or `geography` field,
no constraint is created and the upgrade continues with [...]
+
+Constraint IDs are assigned from the table's `last-constraint-id`, which is
treated as 0 when it is not present. Writers must assign a new constraint an ID
that is higher than the table's current `last-constraint-id` and must update
`last-constraint-id` to the highest assigned ID. Readers must not assume that
every `constraint-id` referenced by a snapshot is present in `constraints`.
+
+#### Check Constraint Expressions
+
+The `expression` of a `check` constraint is a
[predicate](expressions-spec.md#predicates), not a value expression, and is
serialized as described in the [Iceberg expressions spec](expressions-spec.md).
It must use ID references so that it remains bound to the same fields when
columns are renamed.
+
+A `check` expression is evaluated for each row and may only reference the
fields of that row by ID.
+
+A `check` expression must be pure. Its result must depend only on the values
of the fields it references in the row, and evaluating it must have no side
effects.
+
+A [user-defined function](udf-spec.md) used in a `check` expression must
declare `deterministic` as true and is expected to preserve its behavior across
definition versions. Readers and writers are not required to detect behavioral
changes or recompute `constraint-statuses` if such a change occurs.
+
+#### Constraint Evolution
+
+Once a constraint is created, its `expression` must always produce the same
validation result for table rows, and its `field-ids` must not change. A change
to the meaning of a constraint must be made by removing the constraint and
adding a new one with a new `constraint-id`, so that statuses recorded for the
old definition are not applied to the new one. `timestamp-ms` must be updated
when a constraint is updated.
+
+#### Constraint Status
+
+The validation status of each constraint is tracked within snapshot metadata
in the `constraint-statuses` field.
+
+The status of a constraint for a snapshot is one of:
+
+| Status | Description |
+|---------------|-------------|
+| `validated` | The constraint was checked and holds for all rows in the
snapshot |
+| `valid` | The writer has ensured that additions and modifications do
not violate the constraint, and the constraint was `valid` or `validated` for
the parent snapshot |
+| `invalid` | At least one row in the snapshot is known to violate the
constraint |
+| `unvalidated` | Whether the constraint holds for all rows in the snapshot is
not known |
+
+A snapshot's `constraint-statuses` records, for each status, the IDs of the
constraints that have that status for the snapshot:
+
+| Requirement | Field name | Type | Description |
+|-------------|-------------------|-------------|-------------|
+| _optional_ | **`validated`** | `list<int>` | IDs of constraints that are
`validated` for the snapshot |
+| _optional_ | **`valid`** | `list<int>` | IDs of constraints that are
`valid` for the snapshot |
+| _optional_ | **`invalid`** | `list<int>` | IDs of constraints that are
`invalid` for the snapshot |
+| _optional_ | **`unvalidated`** | `list<int>` | IDs of constraints that are
`unvalidated` for the snapshot |
+
+Each list contains the `constraint-id` of every constraint that has that
status for the snapshot. Every constraint that exists when the snapshot is
created must be listed in exactly one of the four lists. A list with no
constraints may be omitted. A constraint whose ID is not present in any list
must be treated as `unvalidated` for the snapshot.
+
+A `constraint-id` listed in a snapshot that is not in the table's
`constraints` must be ignored.
+
+A commit that does not add or modify rows cannot introduce a violation. Such a
commit, for example a delete or a compaction operation, may record `valid` for
a constraint that was `validated` or `valid` for the parent snapshot, without
checking rows again.
+
+A snapshot's `constraint-statuses` must not be modified after the snapshot is
created. A `constraint-statuses` update requires a new snapshot. A snapshot
that only changes constraint statuses must use the `replace` operation, as it
does not change table data.
+
+When a commit is retried against a new parent snapshot, every enforced
constraint, including any that a concurrent commit added or changed to
enforced, must hold for the rows that the commit adds or modifies in the
snapshot that results from applying the commit to the new parent.
`constraint-statuses` must be recomputed from the new parent.
+
+A constraint may be enforced while it is `invalid` or `unvalidated`, and
writers may commit to such a table. An enforced constraint requires that a
writer not add or modify rows so that they violate the constraint; it does not
require a writer to repair existing violations.
+
### Manifests
A table's metadata tree is composed of manifests. A manifest is an immutable
file that tracks a subset of a table metadata for a given
[snapshot](#snapshots). Leaf manifests are the lowest level of the metadata
tree and track data or delete files, along with each file's partition data,
metrics, and tracking information. The snapshot root file tracks leaf
manifests; in v4 and later the root is also a manifest and can also track data
files.
@@ -1158,6 +1256,7 @@ A snapshot consists of the following fields:
| _required_ | **`first-row-id`** | The first `_row_id` assigned
to the first row in the first data file in the first manifest, see [Row
Lineage](#row-lineage) |
| _required_ | **`added-rows`** | The upper bound of the
number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| _optional_ | **`key-id`** | ID of the encryption key
that encrypts the root manifest key metadata |
+ | _optional_ | **`constraint-statuses`** | A struct that records the
status of the table's constraints for the snapshot, see [Constraint
Status](#constraint-status) |
When committed while the table is at version 4 or higher, a snapshot's
`timestamp-ms` must be strictly greater than its parent snapshot's
`timestamp-ms` on the same branch. Snapshots with no parent, and snapshots that
were committed before a table was upgraded to v4 or higher, do not have this
requirement.
@@ -1408,6 +1507,8 @@ Table metadata consists of the following fields:
| _optional_ | **`partition-statistics`** | A list (optional) of
[partition statistics](#partition-statistics). |
| _required_ | **`next-row-id`** | A `long` higher than all
assigned row IDs; the next snapshot's `first-row-id`. See [Row
Lineage](#row-lineage). |
| _optional_ | **`encryption-keys`** | A list (optional) of
[encryption keys](#encryption-keys) used for table encryption. |
+ | _optional_ | **`last-constraint-id`** | An integer; the highest
assigned constraint ID for the table. This is used to ensure constraints are
always assigned an unused ID. See [Constraints](#constraints). |
+ | _optional_ | **`constraints`** | A list of
[constraints](#constraints) for the table. |
For serialization details, see Appendix C.
@@ -1524,10 +1625,11 @@ Notes:
When two commits happen at the same time and are based on the same version,
only one commit will succeed. In most cases, the failed commit can be applied
to the new current version of table metadata and retried. Updates verify the
conditions under which they can be applied to a new version and retry if those
conditions are met.
-* Append operations have no requirements and can always be applied.
+* Append operations have no requirements and can always be applied, other than
the constraint requirements below.
* Replace operations must verify that the files that will be deleted are still
in the table. Examples of replace operations include format changes (replace an
Avro file with a Parquet file) and compactions (several files are replaced with
a single file that contains the same rows).
* Delete operations must verify that specific files to delete are still in the
table. Delete operations based on expressions can always be applied (e.g.,
where timestamp < X).
* Table schema updates and partition spec changes must validate that the
schema has not changed between the base version and the current version.
+* For a table with constraints, every operation must also follow the retry
requirements in [Constraints](#constraints), including recomputing
`constraint-statuses` from the new version.
#### File System Tables
@@ -1892,10 +1994,12 @@ Hash results are not dependent on decimal scale, which
is part of the type, not
Schemas are serialized as a JSON object with the same fields as a struct in
the table below, and the following additional fields:
-| v1 | v2 and v3 |Field|JSON representation|Example|
-| ---------- | ---------- |--- |--- |--- |
-| _optional_ | _required_ |**`schema-id`**|`JSON int`|`0`|
-| _optional_ | _optional_ |**`identifier-field-ids`**|`JSON list of ints`|`[1,
2]`|
+| v1 | v2 and v3 | v4 |Field|JSON representation|Example|
+| ---------- | ---------- | ---------- |--- |--- |--- |
+| _optional_ | _required_ | _required_ |**`schema-id`**|`JSON int`|`0`|
+| _optional_ | _optional_ | |**`identifier-field-ids`**|`JSON list
of ints`|`[1, 2]`|
+
+In v4, [`primary-key` constraint](#constraints) replaces
`identifier-field-ids`.
Types are serialized according to this table:
@@ -2025,13 +2129,15 @@ A metadata JSON file may be compressed with
[GZIP](https://datatracker.ietf.org/
|**`last-partition-id`**|`JSON int`|`1000`|
|**`properties`**|`JSON object: {`<br /> `"<key>": "<val>",`<br
/> `...`<br />`}`|`{`<br /> `"write.format.default":
"avro",`<br /> `"commit.retry.num-retries": "4"`<br />`}`|
|**`current-snapshot-id`**|`JSON long`|`3051729675574597004`|
-|**`snapshots`**|`JSON list of objects: [ {`<br /> `"snapshot-id":
<id>,`<br /> `"timestamp-ms": <timestamp-in-ms>,`<br
/> `"summary": {`<br /> `"operation":
<operation>,`<br /> `... },`<br
/> `"manifest-list": "<location>",`<br /> `"schema-id":
"<id>"`<br /> `},`<br /> `...`<br />`]`|`[ {`<br
/> `"snapshot-id": 3051729675574597004,`<br /> `"tim [...]
+|**`snapshots`**|`JSON list of objects: [ {`<br /> `"snapshot-id":
<id>,`<br /> `"timestamp-ms": <timestamp-in-ms>,`<br
/> `"summary": {`<br /> `"operation":
<operation>,`<br /> `... },`<br
/> `"manifest-list": "<location>",`<br /> `"schema-id":
"<id>",`<br /> `"constraint-statuses": {`<br
/> `"validated": [ ... ],`<br
/> `... }`<br [...]
|**`snapshot-log`**|`JSON list of objects: [`<br /> `{`<br
/> `"snapshot-id": ,`<br /> `"timestamp-ms":`<br
/> `},`<br /> `...`<br />`]`|`[ {`<br
/> `"snapshot-id": 30517296...,`<br /> `"timestamp-ms":
1515100...`<br />`} ]`|
|**`metadata-log`**|`JSON list of objects: [`<br /> `{`<br
/> `"metadata-file": ,`<br /> `"timestamp-ms":`<br
/> `},`<br /> `...`<br />`]`|`[ {`<br
/> `"metadata-file": "s3://bucket/.../v1.json",`<br
/> `"timestamp-ms": 1515100...`<br />`} ]` |
|**`sort-orders`**|`JSON sort orders (list of sort field object)`|`See above`|
|**`default-sort-order-id`**|`JSON int`|`0`|
|**`refs`**|`JSON map with string key and object value:`<br />`{`<br
/> `"<name>": {`<br /> `"snapshot-id": <id>,`<br
/> `"type": <type>,`<br /> `"max-ref-age-ms": <long>,`<br
/> `...`<br /> `}`<br /> `...`<br />`}`|`{`<br
/> `"test": {`<br /> `"snapshot-id": 123456789000,`<br
/> `"type": "tag",`<br /> `"max-ref-age-ms": 10000000`<br
/> `}`<br />`}`|
|**`encryption-keys`**|`JSON list of encryption key objects`|`[ {"key-id":
"5f819b", "key-metadata": "aWNlYmVyZwo="} ]`|
+|**`last-constraint-id`**|`JSON int`|`2`|
+|**`constraints`**|`JSON list of objects: [ {`<br
/> `"constraint-id": <id>,`<br /> `"type": "check",`<br
/> `"name": "<name>",`<br /> `"enforced": <boolean>,`<br
/> `"timestamp-ms": <timestamp-in-ms>,`<br
/> `"expression": { ... }`<br />`}, {`<br
/> `"constraint-id": <id>,`<br /> `"type":
"primary-key",`<br /> `"name": "<name>",`<br
/> `"enforced": <boolean>,`<br /> ` [...]
### Name Mapping Serialization
@@ -2163,6 +2269,12 @@ Equality deletes are prohibited in v4.
* Upgrading a v2 or v3 table to v4 does not require rewriting data or delete
files
* Readers must continue to apply equality deletes for v2 and v3 tables and for
equality deletes carried over into upgraded v4 tables
+Constraints are added in v4:
+
+* Table metadata may contain `constraints` and `last-constraint-id`
+* Snapshots may contain `constraint-statuses`
+* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids`
as a `primary-key` constraint that is not enforced, unless a field cannot be
used in a `primary-key` constraint
+
### Version 3
Default values are added to struct fields in v3.
