amogh-jahagirdar commented on code in PR #16025:
URL: https://github.com/apache/iceberg/pull/16025#discussion_r4266499457
##########
format/spec.md:
##########
@@ -656,15 +661,27 @@ A data or delete file is associated with a sort order by
the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files,
along with each file’s partition data tuple, metrics, and tracking information.
One or more manifest files are used to store a [snapshot](#snapshots), which
tracks all of the files in a table at some point in time. Manifests are tracked
by a [manifest list](#manifest-lists) for each table snapshot.
+A table's metadata tree is composed of manifests. A manifest is an immutable
file that tracks a subset of a table metadata for a given
[snapshot](#snapshots). Leaf manifests are the lowest level of the metadata
tree and track data or delete files, along with each file's partition data,
metrics, and tracking information. The snapshot root file tracks leaf
manifests; in v4 and later the root is also a manifest and can also track data
files.
Review Comment:
The manifest type table right below covers this; v4 leaf manifests only
contain data files, while v2-v3 delete manifests can still be tracked by a v4
root. So I left this as is.
##########
format/spec.md:
##########
@@ -676,15 +693,26 @@ A manifest file must store the partition spec and other
metadata as properties i
| _optional_ | _required_ | `format-version` | Table format version
number of the manifest as a string
|
| | _required_ | `content` | Type of content files
tracked by the manifest: "data" or "deletes"
|
+=== "v4"
+ | Requirement | Key | Value |
+ |-------------|---------------------|-------|
+ | _required_ | `schema-id` | ID of the schema used to write the
manifest as a string |
+ | _required_ | `format-version` | Table format version number of the
manifest as a string |
+
#### Content file uniqueness
Within a snapshot, each content file must be referenced by at most one live
manifest entry across all manifests; otherwise, the snapshot has undefined
behavior. Writers should not produce multiple manifest entries for the same
content file in a snapshot (for example, both ADDED and DELETED entries for the
same file). Writers are not required to validate uniqueness at commit time.
-#### Manifest Entry Fields
+#### Manifest Schema
+
+In v4, manifests store `tracked_file` records that describe data or manifest
files and contain `tracking` metadata, like `status`. The relationship between
a file and its tracking metadata is inverted in v1-v3 manifests, which store
`manifest_entry` records that contain tracking information and a `data_file`
struct. The v4 manifest schema is not compatible with the v1-v3 manifest schema.
Review Comment:
Done!
##########
format/spec.md:
##########
@@ -676,15 +693,26 @@ A manifest file must store the partition spec and other
metadata as properties i
| _optional_ | _required_ | `format-version` | Table format version
number of the manifest as a string
|
| | _required_ | `content` | Type of content files
tracked by the manifest: "data" or "deletes"
|
+=== "v4"
+ | Requirement | Key | Value |
+ |-------------|---------------------|-------|
+ | _required_ | `schema-id` | ID of the schema used to write the
manifest as a string |
+ | _required_ | `format-version` | Table format version number of the
manifest as a string |
+
#### Content file uniqueness
Within a snapshot, each content file must be referenced by at most one live
manifest entry across all manifests; otherwise, the snapshot has undefined
behavior. Writers should not produce multiple manifest entries for the same
content file in a snapshot (for example, both ADDED and DELETED entries for the
same file). Writers are not required to validate uniqueness at commit time.
-#### Manifest Entry Fields
+#### Manifest Schema
+
+In v4, manifests store `tracked_file` records that describe data or manifest
files and contain `tracking` metadata, like `status`. The relationship between
a file and its tracking metadata is inverted in v1-v3 manifests, which store
`manifest_entry` records that contain tracking information and a `data_file`
struct. The v4 manifest schema is not compatible with the v1-v3 manifest schema.
-The schema of a manifest file is defined by the `manifest_entry` struct, which
consists of the following fields:
+The contents of a file are part of a snapshot if its tracking `status` is
**live**: ADDED, EXISTING, or MODIFIED. Changes in a snapshot are tracked using
the additional DELETED and REPLACED statuses.
Review Comment:
REPLACED isn't a removal though; it's the previous version of a file that's
still live, paired with a MODIFIED entry. So I'd prefer to keep the more
general "changes" wording here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
