gaborkaszab commented on code in PR #16025:
URL: https://github.com/apache/iceberg/pull/16025#discussion_r4025249201
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
Review Comment:
nit: "See tracking struct below." can we have a link to the struct similarly
to content_stats
This goes for other description that refer to nested structures
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
Review Comment:
nit: "Location of the file or manifest." manifest is also a file. Can't we
simply say "Location of the file." ?
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the
deletion vector was added. |
+ | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* |
Snapshot ID where the latest column file was added. |
+ | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number
of the file. Inherited when null and status is 1 (ADDED). |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | File sequence
number indicating when the file was added. Inherited when null and status is
ADDED. |
+ | 142 | **`first_row_id`** | `long` | *optional* | For a data file, the
`_row_id` for its first row. For a data manifest, the starting `_row_id` to
assign to rows added by ADDED data files. See [First Row ID
Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | Positions deleted
in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
Review Comment:
nit: "Positions deleted in the referenced leaf manifest this snapshot" Maybe
it's just me and my English, but it's not entirely unambiguously understandable
to me from this sentence that this is just the diff, the new positions being
deleted by the current snapshot. Not sure how, but might worth rephrasing this
to be more clear.
Maybe "Positions deleted from the referenced leaf manifest by the current
snapshot" ?
Same for below
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
Review Comment:
Two observation wrt the description: "Column update files associated with
this entry".
1) Instead of `entry` we could say `file`. We don't seem to call these
entries in V4.
2) "Column update" might be correct now, but looking forward column family
could also create these and those are not updates. Maybe just say `Column
files`?
##########
format/spec.md:
##########
@@ -676,13 +700,19 @@ A manifest file must store the partition spec and other
metadata as properties i
| _optional_ | _required_ | `format-version` | Table format version
number of the manifest as a string
|
| | _required_ | `content` | Type of content files
tracked by the manifest: "data" or "deletes"
|
+=== "v4"
+ | Requirement | Key | Value
|
+
|-------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | `schema-id` | ID of the schema used to write the
manifest as a string
|
+ | _optional_ | `format-version` | Table format version number of the
manifest as a string
|
+
#### Content file uniqueness
Within a snapshot, each content file must be referenced by at most one live
manifest entry across all manifests; otherwise, the snapshot has undefined
behavior. Writers should not produce multiple manifest entries for the same
content file in a snapshot (for example, both ADDED and DELETED entries for the
same file). Writers are not required to validate uniqueness at commit time.
-#### Manifest Entry Fields
+#### Entries in Manifests
-The schema of a manifest file is defined by the `manifest_entry` struct, which
consists of the following fields:
+In v1-v3, manifest entries are described by the `manifest_entry` struct. In
v4, entries are called tracked files and are described by the `tracked_file`
struct. In v4, `data_file` struct fields are flattened directly into the
tracked file, and tracking fields are grouped into a nested `tracking` struct.
Review Comment:
Just curious: Are we sure we want to expose the concept and the name of a
tracked file? I figured that's just an implementation detail and in the spec we
can still call the entries in a manifest a manifest_entry, just we have a
different schema/structure for them between pre-v4 and v4+.
##########
format/spec.md:
##########
@@ -656,15 +662,33 @@ A data or delete file is associated with a sort order by
the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files,
along with each file’s partition data tuple, metrics, and tracking information.
One or more manifest files are used to store a [snapshot](#snapshots), which
tracks all of the files in a table at some point in time. Manifests are tracked
by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along
with each file’s partition data, metrics, and tracking information. One or more
manifest files are used to store a [snapshot](#snapshots), which tracks all of
the files in a table at some point in time. Manifests are tracked by a snapshot
root for each table snapshot. In v4, the snapshot root is a root manifest that
may track data files in addition to leaf manifest files.
A manifest is a valid Iceberg data file: files must use valid Iceberg formats,
schemas, and column projection.
-A manifest may store either data files or delete files, but not both because
manifests that contain delete files are scanned first during job planning.
Whether a manifest is a data manifest or a delete manifest is stored in
manifest metadata.
+Each manifest type contains the following content:
-A manifest stores files for a single partition spec. When a table’s partition
spec changes, old files remain in the older manifest and newer files are
written to a new manifest. This is required because a manifest file’s schema is
based on its partition spec (see below). The partition spec of each manifest is
also used to transform predicates on the table's data rows into predicates on
partition values that are used during job planning to select files from a
manifest.
+| Manifest type | Contents |
+|----------------|----------|
+| v1-v3 data manifest | Data files |
+| v2-v3 delete manifest | Delete files |
+| v4 root manifest (snapshot root) | Data files, data manifests, delete
manifests |
+| v4 data manifest | Data files and their colocated deletion vectors |
Review Comment:
There are v4 delete manifests non being added to this table. Is that
intentional?
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the
deletion vector was added. |
+ | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* |
Snapshot ID where the latest column file was added. |
+ | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number
of the file. Inherited when null and status is 1 (ADDED). |
Review Comment:
This can be null (and inherited) also for MODIFIED, when adding a new column
file. Should we also cover that now, or maybe in a column file follow-up. This
is easy to miss later, so I lean towards including now.
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
Review Comment:
I see that pre-v4 has a remark about deletes are not used in scans and we
are following that pattern here. However, I don't think it belongs here. Seems
more scan planning related.
+1 for @stevenzwu 's suggestion to drop that sentence.
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the
deletion vector was added. |
+ | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* |
Snapshot ID where the latest column file was added. |
+ | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number
of the file. Inherited when null and status is 1 (ADDED). |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | File sequence
number indicating when the file was added. Inherited when null and status is
ADDED. |
+ | 142 | **`first_row_id`** | `long` | *optional* | For a data file, the
`_row_id` for its first row. For a data manifest, the starting `_row_id` to
assign to rows added by ADDED data files. See [First Row ID
Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | Positions deleted
in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | Positions
replaced in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 155 | **`location`** | `string` | *required* | Location of the Puffin
file. |
+ | 144 | **`offset`** | `long` | *required* | Offset in the file where the
content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | Length of the
referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | Cardinality of the
deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 504 | **`added_files_count`** | `int` | *required* | Count of entries
with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `int` | *required* | Count of entries
with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `int` | *required* | Count of entries
with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `int` | *required* | Count of entries
with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `int` | *required* | Count of entries
with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | Total number of
rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | Total number of
rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | Total number of
rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | Total number of
rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | Total number of
rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | Minimum data
sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | Positions in the referenced
leaf manifest that are not live. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | Cardinality of the
manifest deletion vector. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 161 | **`format_version`** | `int` | *required* | Format version of this
column file. |
+ | 162 | **`field_ids`** | `list<163: int>` | *required* | Live field IDs
stored in this column file. |
+ | 164 | **`location`** | `string` | *required* | Location of the column
file. |
+ | 165 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, or `parquet`. |
+ | 166 | **`file_size_in_bytes`** | `long` | *required* | Total column file
size in bytes. |
+ | 167 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 168 | **`split_offsets`** | `list<169: long>` | *optional* | Split
offsets for the column file. Must be sorted ascending. |
+
+ **Tracked File Requirements**
+
+ - `content_type` must not be 1 (POSITION_DELETES) or 2 (EQUALITY DELETES).
+ - `deletion_vector.offset` and `deletion_vector.size_in_bytes` must
exactly match the `offset` and `length` stored in the Puffin footer for the
deletion vector blob.
+ - A leaf manifest may only contain data files.
+ - A root manifest may reference v1-v3 manifests; a referenced v1-v3 leaf
manifest must have `format_version` PRE-V4.
+ - Other v4 tracked files must have `format_version` V4.
+ - `manifest_info` must be set if and only if the tracked file is a
manifest.
+ - `deletion_vector` may only be set if the tracked file is a data file.
+ - `column_files` may only be set if the tracked file is a data file or a
data manifest.
+ - `tracking.deleted_positions` and `tracking.replaced_positions` may only
be set if the tracked file is a manifest.
+ - `tracking.snapshot_id` and `tracking.sequence_number` are required for
the tracked file in the root manifest.
+ - For manifests, `tracking.sequence_number` must equal
`tracking.file_sequence_number`.
+ - `tracking.dv_snapshot_id` may only be set if `deletion_vector` or
`manifest_info.dv` is set.
+ - `tracking.latest_column_file_snapshot_id` may only be set if
`column_files` is set.
+ - `manifest_info.dv_cardinality` must be set if and only if
`manifest_info.dv` is non-null.
+
+ When a file is added to the dataset, its tracked file must set status to
ADDED and store the snapshot ID in which the file was added.
+
+ When a data file's deletion vector or column files are updated, the writer
records a MODIFIED entry for the live version and marks the prior version as
replaced, either with a REPLACED entry or in a [manifest deletion
vector](#manifest-deletion-vectors). The resulting entries' `dv_snapshot_id` or
`latest_column_file_snapshot_id` must record the snapshot in which the deletion
vector or column files, respectively, last changed. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed.
+
+ When a file is deleted from the dataset, its tracked file must set status
to DELETED and store the snapshot ID in which the file was deleted. Writers
must include DELETED entries in the manifest for the snapshot that deletes the
file. The next manifest written for those entries must omit the DELETED entries.
Review Comment:
Maybe I confuse this, but what if we delete an entry from a leaf manifest by
setting its position in the referencing root manifest entry's manifest DV? We
won't set the deleted entry in the left manifest to DELETED, that goes against
this sentence.
##########
format/spec.md:
##########
@@ -691,21 +707,118 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
*required* | Writer format version. V4 writers must produce `format_version` 4.
|
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data or equality delete file. Must be sorted ascending.
|
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
*optional* | Column update files associated with this entry. |
+
+ Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce
`content_type` 1.
+
+ V4 leaf data manifests must only contain entries with `content_type` 0
(DATA); V4 leaf delete manifests must only contain entries with `content_type`
2 (EQUALITY DELETES). A root manifest may reference V1-V3 manifests; V1-V3 leaf
manifest references must have `format_version` set to 0.
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null
otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null
otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null
otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is 3 or 4 (manifests).
+ - `split_offsets` must be null when `content_type` is 3 or 4 (manifests).
+ - `tracking.deleted_positions` and `tracking.replaced_positions` must be
null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | *required* | Used to track additions,
deletions, replacements, and modifications. When a data file's
`deletion_vector` or `column_files` change, REPLACED marks the prior version of
the entry and MODIFIED marks the new, live version. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed. Deletes are not used in
scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID
where the file was added or deleted. Inherited when null. Optional for leaf
manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot
ID where the deletion vector was added. Inherited when null. Must be null when
`deletion_vector` is null. |
+ | 3 | **`sequence_number`** | `long` | *optional* | *optional* | Data
sequence number of the file. Inherited when null and status is 1 (ADDED). Must
equal `file_sequence_number` if `content_type` is 3 or 4. Optional for leaf
manifests, required for root. |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | *optional* | File
sequence number indicating when the file was added. Inherited when null and
status is ADDED. Must equal `sequence_number` if `content_type` is 3 or 4. |
+ | 142 | **`first_row_id`** | `long` | *optional* | *optional* | The
`_row_id` for the first row in the data file if `content_type` is 0. If
`content_type` is 3, this is the starting `_row_id` to assign to rows added by
ADDED data files. See [First Row ID Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were deleted in this snapshot. |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were replaced in this snapshot. |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 155 | **`location`** | `string` | *required* | *required* | Location of
the Puffin file. |
+ | 144 | **`offset`** | `long` | *required* | *required* | Offset in the
file where the content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | *required* | Length of
the referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | *required* | Cardinality
of the deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 504 | **`added_files_count`** | `long` | *required* | *required* | Count
of entries with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `long` | *required* | *required* |
Count of entries with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `long` | *required* | *required* |
Count of entries with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `long` | *required* | *required* |
Count of entries with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `long` | *required* | *required* |
Count of entries with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | *required* | Total
number of rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | *required* |
Total number of rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | *required* |
Total number of rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | *required* |
Total number of rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | *required* |
Total number of rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | *required* |
Minimum data sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | *optional* | Roaring bitmap of
entry positions in the manifest that are not live in the current snapshot. |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | *optional* |
Cardinality of the manifest deletion vector. Must be set when `dv` is non-null;
must be null otherwise. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
Review Comment:
At this point, this is no longer here just to see how it fits into the big
picture, but intentionally part of this PR, right? I think we should add
`column_files` with all the others together, not in a separate PR. The exact
details and constraints could come as a follow-up
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the
deletion vector was added. |
+ | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* |
Snapshot ID where the latest column file was added. |
+ | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number
of the file. Inherited when null and status is 1 (ADDED). |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | File sequence
number indicating when the file was added. Inherited when null and status is
ADDED. |
+ | 142 | **`first_row_id`** | `long` | *optional* | For a data file, the
`_row_id` for its first row. For a data manifest, the starting `_row_id` to
assign to rows added by ADDED data files. See [First Row ID
Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | Positions deleted
in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | Positions
replaced in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 155 | **`location`** | `string` | *required* | Location of the Puffin
file. |
+ | 144 | **`offset`** | `long` | *required* | Offset in the file where the
content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | Length of the
referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | Cardinality of the
deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 504 | **`added_files_count`** | `int` | *required* | Count of entries
with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `int` | *required* | Count of entries
with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `int` | *required* | Count of entries
with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `int` | *required* | Count of entries
with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `int` | *required* | Count of entries
with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | Total number of
rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | Total number of
rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | Total number of
rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | Total number of
rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | Total number of
rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | Minimum data
sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | Positions in the referenced
leaf manifest that are not live. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | Cardinality of the
manifest deletion vector. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 161 | **`format_version`** | `int` | *required* | Format version of this
column file. |
+ | 162 | **`field_ids`** | `list<163: int>` | *required* | Live field IDs
stored in this column file. |
+ | 164 | **`location`** | `string` | *required* | Location of the column
file. |
+ | 165 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, or `parquet`. |
+ | 166 | **`file_size_in_bytes`** | `long` | *required* | Total column file
size in bytes. |
+ | 167 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 168 | **`split_offsets`** | `list<169: long>` | *optional* | Split
offsets for the column file. Must be sorted ascending. |
+
+ **Tracked File Requirements**
+
+ - `content_type` must not be 1 (POSITION_DELETES) or 2 (EQUALITY DELETES).
+ - `deletion_vector.offset` and `deletion_vector.size_in_bytes` must
exactly match the `offset` and `length` stored in the Puffin footer for the
deletion vector blob.
+ - A leaf manifest may only contain data files.
+ - A root manifest may reference v1-v3 manifests; a referenced v1-v3 leaf
manifest must have `format_version` PRE-V4.
+ - Other v4 tracked files must have `format_version` V4.
+ - `manifest_info` must be set if and only if the tracked file is a
manifest.
+ - `deletion_vector` may only be set if the tracked file is a data file.
+ - `column_files` may only be set if the tracked file is a data file or a
data manifest.
+ - `tracking.deleted_positions` and `tracking.replaced_positions` may only
be set if the tracked file is a manifest.
+ - `tracking.snapshot_id` and `tracking.sequence_number` are required for
the tracked file in the root manifest.
+ - For manifests, `tracking.sequence_number` must equal
`tracking.file_sequence_number`.
+ - `tracking.dv_snapshot_id` may only be set if `deletion_vector` or
`manifest_info.dv` is set.
+ - `tracking.latest_column_file_snapshot_id` may only be set if
`column_files` is set.
+ - `manifest_info.dv_cardinality` must be set if and only if
`manifest_info.dv` is non-null.
+
+ When a file is added to the dataset, its tracked file must set status to
ADDED and store the snapshot ID in which the file was added.
+
+ When a data file's deletion vector or column files are updated, the writer
records a MODIFIED entry for the live version and marks the prior version as
replaced, either with a REPLACED entry or in a [manifest deletion
vector](#manifest-deletion-vectors). The resulting entries' `dv_snapshot_id` or
`latest_column_file_snapshot_id` must record the snapshot in which the deletion
vector or column files, respectively, last changed. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed.
+
+ When a file is deleted from the dataset, its tracked file must set status
to DELETED and store the snapshot ID in which the file was deleted. Writers
must include DELETED entries in the manifest for the snapshot that deletes the
file. The next manifest written for those entries must omit the DELETED entries.
+
+The file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+
+Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
snapshot root.
+The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset, except during the addition of a
column file. The data sequence number represents a relative age of the file
content and should be used for planning which delete files apply to a data file.
+The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
Review Comment:
For MODIFIED it's also inherited, isn't it? E.g. when adding a column file
aren't we expected to reset sequence_number to null to allow the value being
inherited?
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
Review Comment:
Just FYI, in the implementation we say `Snapshot ID where the file was
added, deleted, or replaced`. I'm confused now what we agreed on, but either
this or the code has to change :)
##########
format/spec.md:
##########
@@ -1051,7 +1193,9 @@ A simple and valid approach is to estimate the number of
rows in data files that
### Scan Planning
-Scans are planned by reading the manifest files for the current snapshot.
Deleted entries in data and delete manifests (those marked with status
"DELETED") are not used in a scan.
+Scans are planned by reading the manifests referenced by the snapshot root for
the current snapshot; starting in v4, the snapshot root may also contain data
files.
+
+Deleted entries in data and delete manifests (those marked with status
"DELETED") are not used in a scan; starting in v4, an entry is also not live if
its status is REPLACED or, for a leaf-manifest entry, if its position is set in
the referencing root manifest entry's `manifest_info.dv` (see [Manifest
Deletion Vectors](#manifest-deletion-vectors)).
Review Comment:
+1
I'd say non-live entries are not used in scan. And then we can describe in
one sentence what non-live means. Not sure if we do that anywhere else in the
spec, or should we describe that here.
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the
deletion vector was added. |
+ | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* |
Snapshot ID where the latest column file was added. |
+ | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number
of the file. Inherited when null and status is 1 (ADDED). |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | File sequence
number indicating when the file was added. Inherited when null and status is
ADDED. |
+ | 142 | **`first_row_id`** | `long` | *optional* | For a data file, the
`_row_id` for its first row. For a data manifest, the starting `_row_id` to
assign to rows added by ADDED data files. See [First Row ID
Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | Positions deleted
in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | Positions
replaced in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 155 | **`location`** | `string` | *required* | Location of the Puffin
file. |
+ | 144 | **`offset`** | `long` | *required* | Offset in the file where the
content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | Length of the
referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | Cardinality of the
deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 504 | **`added_files_count`** | `int` | *required* | Count of entries
with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `int` | *required* | Count of entries
with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `int` | *required* | Count of entries
with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `int` | *required* | Count of entries
with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `int` | *required* | Count of entries
with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | Total number of
rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | Total number of
rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | Total number of
rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | Total number of
rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | Total number of
rows in MODIFIED entries. |
Review Comment:
FYI, I don't see this and `modified_files_count` in the implementation. Is
this part of some of the ongoing PRs or should we open a new one to cover?
##########
format/spec.md:
##########
@@ -742,18 +758,126 @@ The `data_file` struct consists of the following fields:
| | | _optional_ | **`144 content_offset`**
| `long` |
The offset in the file where the content starts [5] |
| | | _optional_ | **`145 content_size_in_bytes`**
| `long` |
The length of a referenced content stored in the file; required if
`content_offset` is present [5] |
-The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file.
Its type is derived from the partition fields of the partition spec used to
write the manifest file. In v2, the partition struct's field ids must match the
ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the
metrics maps, the type of `content_stats` is based on table metadata, like
schema. Similar to the `partition` struct, the same type is used for all files
tracked in a manifest.
+ Notes:
+
+ 1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
+ 2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in
the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
+ 3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+ 4. Position delete metadata can use `referenced_data_file` when all
deletes tracked by the entry are in a single data file. Setting the referenced
file is required for deletion vectors.
+ 5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
+ 6. The following field ids are reserved on `data_file`: 141.
+
+=== "v4"
+ **Tracked Files**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4:
DELETE_MANIFEST) | *required* | Type of content stored in the entry. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
Writer format version. |
+ | 100 | **`location`** | `string` | *required* | Location of the file or
manifest. |
+ | 101 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | Groups status,
snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used
to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort
order for this file. If missing or unknown, the order is assumed to be
unsorted. |
+ | 103 | **`record_count`** | `long` | *required* | Number of records in
this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size
in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column
stats. See [Content Stats](#content-stats). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See
manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split
offsets for the data file. Must be sorted ascending. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
Column update files associated with this entry. |
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions,
replacements, and modifications. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file
was added or deleted. Inherited when null. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the
deletion vector was added. |
+ | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* |
Snapshot ID where the latest column file was added. |
+ | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number
of the file. Inherited when null and status is 1 (ADDED). |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | File sequence
number indicating when the file was added. Inherited when null and status is
ADDED. |
+ | 142 | **`first_row_id`** | `long` | *optional* | For a data file, the
`_row_id` for its first row. For a data manifest, the starting `_row_id` to
assign to rows added by ADDED data files. See [First Row ID
Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | Positions deleted
in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | Positions
replaced in the referenced leaf manifest this snapshot. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 155 | **`location`** | `string` | *required* | Location of the Puffin
file. |
+ | 144 | **`offset`** | `long` | *required* | Offset in the file where the
content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | Length of the
referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | Cardinality of the
deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 504 | **`added_files_count`** | `int` | *required* | Count of entries
with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `int` | *required* | Count of entries
with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `int` | *required* | Count of entries
with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `int` | *required* | Count of entries
with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `int` | *required* | Count of entries
with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | Total number of
rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | Total number of
rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | Total number of
rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | Total number of
rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | Total number of
rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | Minimum data
sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | Positions in the referenced
leaf manifest that are not live. See [Manifest Deletion
Vectors](#manifest-deletion-vectors). |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | Cardinality of the
manifest deletion vector. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
+
+ | Field id | Name | Type | Required | Description |
+ |----------|------|------|----------|-------------|
+ | 161 | **`format_version`** | `int` | *required* | Format version of this
column file. |
+ | 162 | **`field_ids`** | `list<163: int>` | *required* | Live field IDs
stored in this column file. |
+ | 164 | **`location`** | `string` | *required* | Location of the column
file. |
+ | 165 | **`file_format`** | `string` | *required* | String file format
name: `avro`, `orc`, or `parquet`. |
+ | 166 | **`file_size_in_bytes`** | `long` | *required* | Total column file
size in bytes. |
+ | 167 | **`key_metadata`** | `binary` | *optional* |
Implementation-specific key metadata for encryption. |
+ | 168 | **`split_offsets`** | `list<169: long>` | *optional* | Split
offsets for the column file. Must be sorted ascending. |
+
+ **Tracked File Requirements**
+
+ - `content_type` must not be 1 (POSITION_DELETES) or 2 (EQUALITY DELETES).
+ - `deletion_vector.offset` and `deletion_vector.size_in_bytes` must
exactly match the `offset` and `length` stored in the Puffin footer for the
deletion vector blob.
+ - A leaf manifest may only contain data files.
+ - A root manifest may reference v1-v3 manifests; a referenced v1-v3 leaf
manifest must have `format_version` PRE-V4.
+ - Other v4 tracked files must have `format_version` V4.
+ - `manifest_info` must be set if and only if the tracked file is a
manifest.
+ - `deletion_vector` may only be set if the tracked file is a data file.
+ - `column_files` may only be set if the tracked file is a data file or a
data manifest.
+ - `tracking.deleted_positions` and `tracking.replaced_positions` may only
be set if the tracked file is a manifest.
+ - `tracking.snapshot_id` and `tracking.sequence_number` are required for
the tracked file in the root manifest.
+ - For manifests, `tracking.sequence_number` must equal
`tracking.file_sequence_number`.
+ - `tracking.dv_snapshot_id` may only be set if `deletion_vector` or
`manifest_info.dv` is set.
+ - `tracking.latest_column_file_snapshot_id` may only be set if
`column_files` is set.
+ - `manifest_info.dv_cardinality` must be set if and only if
`manifest_info.dv` is non-null.
+
+ When a file is added to the dataset, its tracked file must set status to
ADDED and store the snapshot ID in which the file was added.
+
+ When a data file's deletion vector or column files are updated, the writer
records a MODIFIED entry for the live version and marks the prior version as
replaced, either with a REPLACED entry or in a [manifest deletion
vector](#manifest-deletion-vectors). The resulting entries' `dv_snapshot_id` or
`latest_column_file_snapshot_id` must record the snapshot in which the deletion
vector or column files, respectively, last changed. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed.
+
+ When a file is deleted from the dataset, its tracked file must set status
to DELETED and store the snapshot ID in which the file was deleted. Writers
must include DELETED entries in the manifest for the snapshot that deletes the
file. The next manifest written for those entries must omit the DELETED entries.
+
+The file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+
+Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
snapshot root.
+The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset, except during the addition of a
column file. The data sequence number represents a relative age of the file
content and should be used for planning which delete files apply to a data file.
Review Comment:
Should we have a separate section for column file stuff in a follow-up? This
part can stay because it's relevant for sequence numbers, but what @stevenzwu
mention about rewriting eq-deletes is a requirement and should be covered
somewhere.
##########
format/spec.md:
##########
@@ -656,15 +662,33 @@ A data or delete file is associated with a sort order by
the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files,
along with each file’s partition data tuple, metrics, and tracking information.
One or more manifest files are used to store a [snapshot](#snapshots), which
tracks all of the files in a table at some point in time. Manifests are tracked
by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along
with each file’s partition data, metrics, and tracking information. One or more
manifest files are used to store a [snapshot](#snapshots), which tracks all of
the files in a table at some point in time. Manifests are tracked by a snapshot
root for each table snapshot. In v4, the snapshot root is a root manifest that
may track data files in addition to leaf manifest files.
Review Comment:
At L150 we already cover that. I think even the current sentence is
redundant because we already explain what a snapshot root is. I'd prefer to
remove the addition of the V4 explanation here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
