zhaoxuan1994 opened a new pull request, #777:
URL: https://github.com/apache/iceberg-cpp/pull/777
feat(puffin): support deletion-vector-v1 blob read/write
Add end-to-end support for the `deletion-vector-v1` Puffin blob type on top
of the existing PuffinReader/PuffinWriter and RoaringPositionBitmap.
Encoding (puffin/deletion_vector.{h,cc}):
- SerializeDeletionVectorBlob / DeserializeDeletionVectorBlob implement the
DV blob framing: 4-byte big-endian length, 0xD1D33964 magic, the portable
Roaring vector, and a trailing big-endian CRC-32. Reads validate the magic
and checksum.
- MakeDeletionVectorBlob builds a spec-compliant Blob: fields set to the
row-position metadata column id, snapshot-id/sequence-number = -1, no
compression, and the required referenced-data-file/cardinality properties.
Writing (data/deletion_vector_writer.{h,cc}):
- DeletionVectorWriter accumulates deleted positions per data file,
run-length
encodes each bitmap before serializing, writes one blob per data file, and
produces a DataFile per blob carrying content_offset/content_size_in_bytes,
referenced_data_file and record_count for manifest registration.
Reading (data/delete_loader.cc):
- Implement DeleteLoader::LoadDV: read the blob bytes referenced by
content_offset/content_size_in_bytes, validate referenced_data_file, the
2GB
limit and cardinality == record_count, then apply positions to the index.
This wires DV deletes into DeleteFilter::ComputeAliveRows.
Puffin writer convenience:
- PuffinWriter::Make now fills a default `created-by` property when the
caller
does not provide one, matching the Java writer.
Behavior matches the Java implementation (BaseDVFileWriter /
BaseDeleteLoader /
BitmapPositionDeleteIndex): magic, byte order, CRC coverage, blob fields, RLE
and the read-side validations are all aligned.
Tests:
- DV blob framing round-trip and error cases (bad magic, corrupted CRC,
truncated blob, size mismatch).
- DeletionVectorWriter -> DeleteLoader write/read round trip and guard cases.
- delete_loader: load DV, referenced-file filtering, cardinality mismatch,
mixed DV + position-delete loading.
- delete_filter: end-to-end ComputeAliveRows filtering with a real Puffin DV
file over Arrow FileIO.
- PuffinWriter created-by default and caller-precedence.
Not included (deletion-vector upper layer, follow-up): merging with
previously
written DVs (compaction), the bulk delete(PositionDeleteIndex) API, and
per-data-file partition/spec in a single DV file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
