Indian-boult commented on issue #339: URL: https://github.com/apache/incubator-xtable/issues/339#issuecomment-7004495572
I would like to pick up the Delta deletion vector conversion from RFC-2. Repro on main (`e91662f`, Spark 3.4.2, Delta 2.4.0, Iceberg 1.9.2): a DV-enabled Delta table with 1000 rows in 4 files, then `DELETE WHERE id < 100` (one DV, cardinality 100). Delta reads 900 rows. Iceberg reads 1000 after an incremental sync, a full sync onto the existing target, and a full sync into a fresh target, through both the Standalone and the Kernel source, and every sync reports SUCCESS. A second DELETE, a MERGE and a RESTORE keep the gap. Only OPTIMIZE and REORG PURGE close it, because they rewrite the files. Plan, following RFC-2: 1. `InternalDeletionVector` as its own `InternalFile` linked to its data file, with getters and a lazy ordinals iterator and no eager byte field, carried in the files diff and in snapshots (#640), plus the Standalone source. This builds on the #661 commit with its authorship kept and addresses the open review comments there. @ashvina, I hope that is fine with you. 2. Kernel source parity. 3. Iceberg target: v2 Parquet position deletes committed with `RowDelta`, replaced when a DV changes and removed with their data file. Hudi and Delta targets stay fail-closed, as in #938. 4. Possibly later: v3 deletion vectors for v3 tables. On the v3 point above: I checked it locally. Copying the on-disk Delta DV bytes into a Puffin `deletion-vector-v1` blob works. Iceberg's deserializer accepts them, its own serialization of the same positions is byte-identical, and Spark reads the same count as Delta on a v3 table. Inline DVs need the length prefix and CRC added. I still propose v2 first, because XTable creates v2 tables today and RFC-2 targets v2. One detail for the delete-file location: Delta VACUUM removes untracked files under `<table>/deletion-vectors` (checked on Delta 2.4), so I would use an underscore-prefixed directory instead. The same rule deletes XTable's Iceberg metadata, which I filed as #1021. Two questions: - Is v2 first and v3 second the order you want, or should v3 come first given #954? - Once the Iceberg target writes deletes, the check from #938 would need to move from the source to the targets that cannot represent DVs. Is a per-target check that keeps #938's config key acceptable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
