mbutrovich opened a new pull request, #2866: URL: https://github.com/apache/iceberg-rust/pull/2866
## Which issue does this PR close? - Part of #2792. ## What changes are included in this PR? Adds `DeleteVector::deserialize`, which decodes a `deletion-vector-v1` Puffin blob into a `DeleteVector` (a `RoaringTreemap` of deleted row positions). This is the first, self-contained piece of deletion-vector read support (epic #2792): it adds no scan, reader, or planning changes and can be reviewed in isolation. The blob layout follows the Iceberg Puffin spec and matches Iceberg-Java's `BitmapPositionDeleteIndex` / `RoaringPositionBitmap`: ```text [length: u32 big-endian][magic: D1 D3 39 64][vector][crc32: u32 big-endian] ``` - `length` counts the magic and vector bytes (not itself or the CRC). - The CRC-32 (via `crc32fast`) covers the magic and vector. - `vector` is a roaring bitmap in the portable 64-bit format, read directly by `RoaringTreemap::deserialize_from` (roaring 0.11.x implements this official format). Validation order is length prefix, then CRC (before decoding any payload), then magic, then the roaring decode, so a corrupt blob yields a single clear error rather than an opaque roaring failure. Cardinality is intentionally not checked here; the caller validates the decoded length against the delete file's `record_count` once the manifest metadata is available (a later PR in the epic). Also adds `crc32fast` as a dependency. This overlaps with #2414, which implemented the same decode. That PR has been idle since review feedback, so this is submitted in its place to keep the epic moving; thanks to @Shekharrajak for the original contribution and for informing this implementation. ## Are these changes tested? Yes, unit tests in `delete_vector.rs`: - Round-trip decode of empty, small, and 64-bit-key-spanning vectors. - A run-optimized bitmap (via `RoaringTreemap::optimize`) so the RUN-container decode path is exercised, since Iceberg-Java run-length-encodes deletion vectors before writing. - Error paths: too-short blob, length-prefix mismatch, bad magic, and bad CRC. This decode path is also exercised end to end against deletion vectors written by Spark / Iceberg-Java in a draft DataFusion Comet PR (apache/datafusion-comet#4887), which reads real V3 merge-on-read tables through this code and verifies the deleted rows match Spark. That gives cross-implementation confidence that it decodes what Iceberg actually writes, not just what these tests encode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
