manuzhang opened a new issue, #1107: URL: https://github.com/apache/iceberg-cpp/issues/1107
This is the **parent tracking issue** for implementing the Iceberg **format version 4** spec in iceberg-cpp. > [!NOTE] > v4 is still under active development and has not been formally adopted (see [Format Versioning](https://iceberg.apache.org/spec/#format-versioning)). The list below reflects the spec on `apache/iceberg` `main` as of 2026-11-10 and will be updated as the spec evolves. Until v4 is adopted, v4 support here should be treated as experimental and should not change the default format version. iceberg-cpp currently supports up to v3 (`TableMetadata::kSupportedTableFormatVersion = 3`). Remaining v3 work is tracked separately in #959. If you are interested in working on any of the items below, please **open a dedicated sub-issue** (or link an existing one) and reference this issue so progress is easy to follow. Where the Java implementation already exists, please keep behavior consistent with it; the relevant Java PRs are linked as references. ## 1. Foundation - [ ] Accept `format-version: 4` in table metadata parsing, creation and upgrade (bump `kSupportedTableFormatVersion`, add `kMinFormatVersion*` gates for v4-only features) - [ ] v3 → v4 upgrade path and validation in `TableMetadataBuilder::UpgradeFormatVersion` - [ ] Test fixtures: v4 table metadata and manifests produced by Java for interop tests ## 2. Relative paths Spec: [Paths in Metadata](https://iceberg.apache.org/spec/#paths-in-metadata) (apache/iceberg#13141, apache/iceberg#15630) - [ ] Path resolution and relativization utilities (Java: apache/iceberg#16174) - [ ] Make table metadata `location` optional in v4; must be absolute when present, otherwise supplied by the catalog (Java: apache/iceberg#16572) - [ ] Resolve relative locations when reading v4 metadata (manifests, deletion vectors, column files, statistics files, metadata log) (Java: apache/iceberg#17434, apache/iceberg#17807) - [ ] Write relative locations by default for files under the table location - [ ] Prepend a scheme to scheme-less locations from v1–v3 metadata for consistency with v4 absolute paths - [ ] Catalog integration: pass the table location when loading metadata without `location` (REST: apache/iceberg#17642) ## 3. Adaptive metadata tree Spec: [Manifests](https://iceberg.apache.org/spec/#manifests) (apache/iceberg#16025) ### 3.1 Data model - [ ] `tracked_file`, `tracking`, `manifest_info`, `deletion_vector` and `column_file` structs and their schemas (Java: apache/iceberg#15049, apache/iceberg#15854, apache/iceberg#16092) - [ ] New entry statuses `REPLACED` and `MODIFIED`, and content types `DATA_MANIFEST` / `DELETE_MANIFEST` - [ ] Snapshot `root-manifest` field (replaces `manifest-list` in v4) in snapshot parsing and serialization ### 3.2 Read path - [ ] Parquet manifest reader (v4 manifests are Parquet) (Java: apache/iceberg#16958) - [ ] Read root manifests: inline data files, v4 leaf manifests and pre-v4 data/delete manifests - [ ] Apply manifest deletion vectors (`manifest_info.dv`) and `tracking.deleted_positions` / `replaced_positions` - [ ] Sequence number and `first_row_id` inheritance for v4 tracked files - [ ] Adapters exposing v4 tracked files through the existing `ManifestFile` / `DataFile` / `ManifestEntry` abstractions (Java: apache/iceberg#17932, apache/iceberg#18021, apache/iceberg#17928) - [ ] Scan planning with colocated deletion vectors (single-phase planning; no separate delete manifest pass for v4 leaf manifests) - [ ] Continue to apply position and equality deletes from pre-v4 delete manifests in upgraded tables ### 3.3 Write path - [ ] Parquet manifest writer for leaf and root manifests (Java: apache/iceberg#15634, apache/iceberg#18853) - [ ] `SnapshotProducer` / `FastAppend` / `MergeAppend` producing root manifests, including small commits that inline data files in the root manifest - [ ] Manifest filter/merge managers for v4 (`ManifestFilterManager`, `ManifestMergeManager`, `RewriteManifests`) - [ ] Row-level deletes written as deletion vectors colocated with the data file entry (`MODIFIED` + `REPLACED` entries) - [ ] Reject new equality delete files and delete manifests in v4 tables - [ ] Snapshot expiration and orphan file cleanup aware of root manifests, column files and colocated DVs - [ ] Root manifest encryption key handling (pending apache/iceberg#18503; depends on #988) ## 4. Content stats Spec: [Content Stats](https://iceberg.apache.org/spec/#content-stats) (apache/iceberg#13153, apache/iceberg#14234) - [ ] `content_stats` struct type derived from the table schema (stats field ID assignment: `10_000 + 200 * field-id`, reserved ranges for `_row_id` and `_last_updated_sequence_number`) - [ ] Per-field stats: `lower_bound`, `upper_bound`, `tight_bounds`, `value_count`, `null_value_count`, `nan_value_count`, `avg_value_size_in_bytes` - [ ] `geo_lower` / `geo_upper` bounds for `geometry` and `geography` - [ ] Convert between content stats and legacy metrics maps (Java: apache/iceberg#15251, apache/iceberg#17322) - [ ] Collect content stats in Parquet/Avro writers, including avg value sizes (Java: apache/iceberg#17451) - [ ] Evaluate content stats in `InclusiveMetricsEvaluator` / `StrictMetricsEvaluator` and manifest pruning (Java: apache/iceberg#17413) - [ ] File-level `tight_stats` (pending apache/iceberg#18598) - [ ] Stats-only fields (`stats-fields` in table metadata), e.g. `partition-value` stats ## 5. Column updates Spec: [Column File](https://iceberg.apache.org/spec/#column-file-fields) (apache/iceberg#15146, apache/iceberg#16425) - [ ] Track `column_files` on tracked files (metadata only) - [ ] Read path: stitch column files with the base data file by row position - [ ] Write path: commit column files as `MODIFIED` / `REPLACED` entries with correct sequence number handling ## 6. Schema and types - [ ] `file` type: schema, JSON serde, Parquet/Avro/Arrow projection (apache/iceberg#17919) - [ ] Default value expressions for `write-default` (apache/iceberg#17616, apache/iceberg#16777; depends on the [expressions spec](https://iceberg.apache.org/expressions-spec/)) - [ ] Reject `identifier-field-ids` in v4 schemas (replaced by `primary-key` constraints) ## 7. Constraints Spec: [Constraints](https://iceberg.apache.org/spec/#constraints) (apache/iceberg#17822) - [ ] `constraints` and `last-constraint-id` in table metadata; `constraint-statuses` in snapshots - [ ] Rewrite `identifier-field-ids` as an unenforced `primary-key` constraint when upgrading to v4 - [ ] Table update API and REST update/requirement types for constraints ## 8. Snapshots and commits - [ ] Enforce strictly increasing `timestamp-ms` relative to the parent snapshot for commits to v4 tables - [ ] Remove/disable the deprecated file-system (Hadoop-style) metadata commit scheme for v4 tables, if applicable ## Still under discussion upstream (not yet in the spec) These are tracked in the [Iceberg V4 Spec milestone](https://github.com/apache/iceberg/milestone/58) and will be added above once merged: - `vector` type (apache/iceberg#18297, apache/iceberg#18416) - `_last_updated_timestamp` metadata column (apache/iceberg#17619, apache/iceberg#18526) - Collation support for string types (apache/iceberg#17620) - Derived columns (apache/iceberg#15923) - V4 changelog planning (apache/iceberg#18852) ## Suggested order 1. Foundation + relative paths (small, self-contained, already implemented in Java) 2. Tracked file data model + Parquet manifest reader + content stats read path → **read-only v4 support** 3. Manifest writers and commit path → **v4 write support** 4. Column updates, `file` type, constraints, default value expressions If anything is missing or you have suggestions, feel free to comment or open additional sub-issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
