dina-kar commented on PR #3232: URL: https://github.com/apache/iceberg-rust/pull/3232#issuecomment-6321054868
Thanks for working on this. We need deletion-vector writes in iceberg-rust, and we'd like to build a `DeletionVectorWriter` on top of this PR rather than duplicate it. The writer would emit the Puffin file and its `DataFile` entries (`referenced_data_file`, `content_offset`, `content_size_in_bytes`). I've described the plan on #2580. Some things that might help here: - We have DV tables written by Spark 4.0.4 with Iceberg 1.12.0: unpartitioned and partitioned tables, `DELETE`, merge-on-read `UPDATE` and `MERGE`, and a v2 table with position deletes upgraded to v3. For every Spark-written blob in them, we can check that `deserialize` → `serialize` → `deserialize` returns the same positions, and that Spark and Trino read the blobs this PR writes once they're committed. If useful, I can contribute those checks as tests, either here or in a follow-up PR, whichever you prefer. - The branch currently conflicts with `main`: `crates/iceberg/src/test_utils.rs` was moved in #3107. - Your suggestion to `optimize()` before serializing matches what we saw: Spark's blobs are run-optimized. Our own builds carry this PR's two commits unchanged, so nothing here needs to change on our account. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
