Indian-boult commented on issue #339:
URL: 
https://github.com/apache/incubator-xtable/issues/339#issuecomment-7004495572

   I would like to pick up the Delta deletion vector conversion from RFC-2.
   
   Repro on main (`e91662f`, Spark 3.4.2, Delta 2.4.0, Iceberg 1.9.2): a 
DV-enabled Delta table with 1000 rows in 4 files, then `DELETE WHERE id < 100` 
(one DV, cardinality 100). Delta reads 900 rows. Iceberg reads 1000 after an 
incremental sync, a full sync onto the existing target, and a full sync into a 
fresh target, through both the Standalone and the Kernel source, and every sync 
reports SUCCESS. A second DELETE, a MERGE and a RESTORE keep the gap. Only 
OPTIMIZE and REORG PURGE close it, because they rewrite the files.
   
   Plan, following RFC-2:
   1. `InternalDeletionVector` as its own `InternalFile` linked to its data 
file, with getters and a lazy ordinals iterator and no eager byte field, 
carried in the files diff and in snapshots (#640), plus the Standalone source. 
This builds on the #661 commit with its authorship kept and addresses the open 
review comments there. @ashvina, I hope that is fine with you.
   2. Kernel source parity.
   3. Iceberg target: v2 Parquet position deletes committed with `RowDelta`, 
replaced when a DV changes and removed with their data file. Hudi and Delta 
targets stay fail-closed, as in #938.
   4. Possibly later: v3 deletion vectors for v3 tables.
   
   On the v3 point above: I checked it locally. Copying the on-disk Delta DV 
bytes into a Puffin `deletion-vector-v1` blob works. Iceberg's deserializer 
accepts them, its own serialization of the same positions is byte-identical, 
and Spark reads the same count as Delta on a v3 table. Inline DVs need the 
length prefix and CRC added. I still propose v2 first, because XTable creates 
v2 tables today and RFC-2 targets v2.
   
   One detail for the delete-file location: Delta VACUUM removes untracked 
files under `<table>/deletion-vectors` (checked on Delta 2.4), so I would use 
an underscore-prefixed directory instead. The same rule deletes XTable's 
Iceberg metadata, which I filed as #1021.
   
   Two questions:
   - Is v2 first and v3 second the order you want, or should v3 come first 
given #954?
   - Once the Iceberg target writes deletes, the check from #938 would need to 
move from the source to the targets that cannot represent DVs. Is a per-target 
check that keeps #938's config key acceptable?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to