mbutrovich opened a new issue, #2792: URL: https://github.com/apache/iceberg-rust/issues/2792
## Is your feature request related to a problem or challenge? Iceberg format version 3 stores positional deletes as deletion vectors instead of position delete files. A deletion vector is a roaring bitmap of deleted row positions, written as a `deletion-vector-v1` Puffin blob and referenced from a delete manifest entry by `content_offset` and `content_size_in_bytes`. iceberg-rust cannot read them. `CachingDeleteFileLoader::load_file_for_task` branches only on `DataContentType` (`PositionDeletes` / `EqualityDeletes` / `Data`), and the `PositionDeletes` arm unconditionally calls `parquet_to_batch_stream(...)` on the file path (`crates/iceberg/src/arrow/caching_delete_file_loader.rs`). A V3 deletion vector arrives as `PositionDeletes` content but points at a Puffin file, so the read fails when the loader parses that file as Parquet. The DV branch is stubbed: `DeleteFileContext` carries `// TODO: Delete Vector loader from Puffin files` (`caching_delete_file_loader.rs:54`). Any V3 table with positional deletes is therefore unreadable. Most of the surrounding machinery already exists, so this is wiring rather than a from-scratch feature: - `DataFile` parses the V3 manifest fields `referenced_data_file`, `content_offset`, `content_size_in_bytes` (`crates/iceberg/src/spec/manifest/data_file.rs`). - `DeleteVector` (a `RoaringTreemap` wrapper) exists (`crates/iceberg/src/delete_vector.rs`). - `ParsedDeleteFileContext::DelVecs` is already a loader variant. - Positional deletes already flow `DeleteVector` -> `DeleteFilter` -> `ArrowReader`, so DV application reuses that path unchanged. - `PuffinReader`, `PuffinWriter`, and the `DELETION_VECTOR_V1` blob-type constant exist (`crates/iceberg/src/puffin/`). Missing: the blob codec, propagation of the DV coordinates onto the scan task, and the loader branch connecting them. ## Describe the solution you'd like Add DV read support as a sequence of independently reviewable PRs. Scope is the read path only. DV writing (a `BaseDVFileWriter` equivalent) is out of scope. ### Task list - [ ] **Task 1: DV blob codec.** Add serialize and deserialize between deleted positions and the `deletion-vector-v1` blob payload (byte layout below). Reuses the existing `DELETION_VECTOR_V1` constant. No scan or reader API changes, so it merges first. Include serialize as well as deserialize: it enables a hermetic round-trip test and is the byte-layer foundation for a future DV writer. Tests: a round-trip through `PuffinWriter` / `PuffinReader`, plus a golden Puffin file produced by Iceberg-Java to check cross-implementation compatibility. - [ ] **Task 2: Carry DV coordinates on the scan task.** Add `content_offset`, `content_size_in_bytes`, and `referenced_data_file` to `FileScanTaskDeleteFile` (`crates/iceberg/src/scan/task.rs`), populated from `DataFile` during scan planning. This matches Iceberg-Java, where a DV is a `DeleteFile` (`content=POSITION_DELETES`, `format=PUFFIN`) carrying `contentOffset()`, `contentSizeInBytes()`, and `referencedDataFile()`, attached to the task via `FileScanTask.deletes()`. - [ ] **Task 3: Loader branch and end-to-end read.** In `load_file_for_task`, route delete entries with `content_offset.is_some()` to a DV path: read the blob at `content_offset` / `content_size_in_bytes` via `PuffinReader`, decode with Task 1, and return `ParsedDeleteFileContext::DelVecs`. Validate that the decoded cardinality equals `record_count` and the blob length equals `content_size_in_bytes`, as the Java reader does. Add an end-to-end test reading a V3 table whose deletes are DVs. Two read-path rules from the table spec must hold. A DV applies to a data file when the file path equals the DV's `referenced_data_file`, the file's data sequence number is `<=` the DV's, and the partitions match. When a DV applies to a data file, readers ignore any position delete files that would otherwise match it, because the DV subsumes them. At most one DV exists per data file per snapshot. Iceberg-Java planning enforces these, which is what feeds Comet, but a native iceberg-rust planner must too. ### Task 1 byte layout From the specs, confirmed against Iceberg-Java `BitmapPositionDeleteIndex` and `RoaringPositionBitmap`: - Blob: `[length: u32 BE][magic: D1 D3 39 64][vector][crc: u32 BE]`, where `length = 4 + vector_size` and the CRC-32 covers the magic and vector, not the length prefix. Java writes the magic as the little-endian int `1681511377` (`0x6439D3D1`); writing the four bytes `D1 D3 39 64` directly is equivalent. - Vector (portable 64-bit layout): an 8-byte little-endian bitmap count, then per ascending 32-bit key a 4-byte little-endian key followed by a standard 32-bit Roaring bitmap. Java builds this over 32-bit `RoaringBitmap`, not Roaring's native treemap serialization. The Rust codec should do the same: build the framing on `roaring::RoaringBitmap` (32-bit) `serialize_into` / `deserialize_from`, which implement the 32-bit RoaringFormatSpec, not `RoaringTreemap::serialize_into`, which uses a crate-native format. Add `crc32fast` for the CRC-32. The codec validates the magic and CRC; cardinality and blob-length validation live in Task 3, where the `DeleteFile` metadata is available. Open question to resolve before implementing: whether roaring 0.11 exposes per-key 32-bit bitmap access (`bitmaps()` / `from_bitmaps` or equivalent). ### References - Iceberg table spec `format/spec.md`, "Deletion Vectors": semantics, at-most-one-per-data-file, DV-subsumes-position-deletes, apply-scope rules, and the `referenced_data_file` / `content_offset` / `content_size_in_bytes` fields (required for DVs, must match the Puffin footer). - Iceberg Puffin spec `format/puffin-spec.md`: the `deletion-vector-v1` blob byte layout. - Iceberg-Java: `core/.../deletes/BitmapPositionDeleteIndex.java` (framing, length, magic, CRC) and `core/.../deletes/RoaringPositionBitmap.java` (portable 64-bit layout). ## Willingness to contribute I can contribute to this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
