dina-kar commented on issue #2580: URL: https://github.com/apache/iceberg-rust/issues/2580#issuecomment-6321056037
I'd like to work on the deletion-vector writer half of this issue, building on #3232 (`DeleteVector::serialize`). I'll leave the commit half to #2203 (RowDelta). @vikas-magar, you offered to work on this in July. If you have something in progress, please say so and I'll review it or help with it instead. **Plan:** a `DeletionVectorWriter` in `writer/base_writer`, next to `PositionDeleteFileWriter`, with the same `(file_path, pos)` input, so callers can choose by format version. - It collects the positions per referenced data file into a `DeleteVector`. On `close`, it writes one Puffin file with one `deletion-vector-v1` blob per data file. The blobs are uncompressed, use field `2147483645` (`_pos`) and snapshot id and sequence number `-1`, and carry the `referenced-data-file` and `cardinality` properties, as Iceberg Java does. - It returns one `DataFile` per blob. Each has content `PositionDeletes`, format `Puffin`, `referenced_data_file`, `content_offset`, `content_size_in_bytes`, `record_count` set to the cardinality, and the partition. - The spec allows at most one DV per data file per snapshot, so the writer also takes a data file's existing DV and merges it into the new one. The caller then removes the old DV in the same commit (via RowDelta). **Tests:** - Unit tests read the Puffin file back with `PuffinReader` and `DeleteVector::deserialize`, and scan a data file through `ArrowReader` with the written DV. - An integration test uses a Spark-provisioned v3 table with DVs. It reads Spark's blobs, merges new positions into them, and checks the result. - Outside this repository's CI, we check that Spark 4.0.4 (Iceberg 1.12.0) and Trino 483 read tables whose DVs were written this way, committed through Iceberg Java's RowDelta. The work is on https://github.com/ostrium-labs/iceberg-rust/tree/loams/dv-writer, in small PRs. I'll open upstream PRs against this issue once each piece is green, after #3232 merges, and keep each one under the size guideline. Feedback on the shape of the API is welcome before then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
