This is an automated email from the ASF dual-hosted git repository.
huaxingao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new 7ba28c352c Spec: Forbid writing new equality deletes in v4 (#17783)
7ba28c352c is described below
commit 7ba28c352c0e1f9833e5abd7d0c09ad8e9969f2a
Author: Huaxin Gao <[email protected]>
AuthorDate: Mon Sep 28 18:38:27 2026 -0700
Spec: Forbid writing new equality deletes in v4 (#17783)
* Spec: Forbid writing new equality deletes in v4
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
* address comments
---
format/spec.md | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/format/spec.md b/format/spec.md
index e6f9e0425b..77c4c76824 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix
E](#version-3).
Version 4 of the Iceberg spec restructures metadata for improved performance
and new capabilities:
* Support for [relative locations](#file-locations-in-metadata) in metadata
fields
+* Writing new [equality deletes](#equality-delete-files) is no longer allowed
The full set of changes are listed in [Appendix E](#version-4).
@@ -113,7 +114,7 @@ There are two types of row-level deletes:
* **Position deletes** -- Mark a row deleted by data file path and the row
position in the data file. Position deletes are encoded in a [_position delete
file_](#position-delete-files) (V2) or [_deletion vector_](#deletion-vectors)
(V3 or above).
-* **Equality deletes** -- Mark a row deleted by one or more column values,
like id = 5. Equality deletes are encoded in [_equality delete
file_](#equality-delete-files).
+* **Equality deletes** -- Mark a row deleted by one or more column values,
like id = 5. Equality deletes are encoded in [_equality delete
file_](#equality-delete-files) (may be created in v2 and v3 tables only).
Like data files, delete files are tracked by partition. In general, a delete
file must be applied to older data files with the same partition; see [Scan
Planning](#scan-planning) for details. Column metrics can be used to determine
whether a delete file's rows overlap the contents of a data file or a scan
range.
@@ -1354,13 +1355,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files.
Row-level deletes are added by v2 and are not supported in v1. Deletion vectors
are added in v3 and are not supported in v2 or earlier. Position delete files
must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files.
Row-level deletes are added by v2 and are not supported in v1. Deletion vectors
are added in v3 and are not supported in v2 or earlier. Position delete files
must not be added to v3 tables, but existing position delete files are valid.
Equality delete files must not be added to v4 tables, but existing equality
delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data
file by position in a bitmap
-* Position delete files identify deleted rows by file location and row
position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more
columns
+* Position delete files identify deleted rows by file location and row
position (**prohibited** in v3)
+* Equality delete files identify deleted rows by the value of one or more
columns (**prohibited** in v4)
Deletion vectors are a binary representation of deletes for a single data file
that is more efficient at execution time than position delete files. Unlike
equality or position delete files, there can be at most one deletion vector for
a given data file in a snapshot. Writers must ensure that there is at most one
deletion vector per data file and must merge new deletes with existing vectors
or position delete files.
When removing a data file, writers must also remove any deletion vector that
applies to that data file from delete manifests. Writers are not required to
rewrite Puffin files that contain the removed deletion vectors.
@@ -1418,6 +1419,8 @@ The rows in the delete file must be sorted by `file_path`
then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by
one or more column values, and may optionally contain additional columns of the
deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table.
+
Equality delete files store any subset of a table's columns and use the
table's field ids. The _delete columns_ are the columns of the delete file used
to match data rows. Delete columns are identified by id in the delete file
[metadata column `equality_ids`](#manifests). The column restrictions for
columns used in equality delete files are the same as those for [identifier
fields](#identifier-field-ids) with the exception that optional columns and
columns nested under optional structs a [...]
A data row is deleted if its values are equal to all delete columns for any
row in an equality delete file that applies to the row's data file (see [`Scan
Planning`](#scan-planning)).
@@ -1915,6 +1918,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see
[Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table
Location Specification](#table-location-specification))
+Equality deletes are prohibited in v4.
+
+* Writers must not add equality delete files to v4 tables; equality deletes
cannot be added as an entry to a v4 manifest
+* Upgrading a v2 or v3 table to v4 does not require rewriting data or delete
files
+* Readers must continue to apply equality deletes for v2 and v3 tables and for
equality deletes carried over into upgraded v4 tables
+
### Version 3
Default values are added to struct fields in v3.
