slachiewicz commented on issue #339: URL: https://github.com/apache/incubator-xtable/issues/339#issuecomment-5385202926
Iceberg v3 has changed the target this RFC was designed against, and I think it affects the plan here. RFC-2 specifies conversion into v2 position delete files. The current Iceberg spec now says: > Position delete files must not be added to v3 tables, but existing position delete files are valid. and, under the v3 changes: > Writers are not allowed to add new position delete files to v3 tables with the migration direction stated explicitly: > Existing position deletes must be written to delete vectors when updating the position deletes for a data file. So implementing RFC-2 as written produces output that cannot be added to a v3 table. Since #661 is currently conflicting and would need a rebase anyway, it seemed worth raising before that work resumes rather than after. The v3 replacement looks structurally closer to the source than v2 was. A Delta deletion vector is a serialized 64-bit RoaringBitmap, and Iceberg v3 stores deletion vectors as the Puffin `deletion-vector-v1` blob, which wraps the same portable RoaringBitmap payload. That suggests the conversion could be closer to re-framing the bitmap bytes than to materializing `(file_path, position)` rows — which would also avoid the delete-file cleanup problem the RFC notes. I have not verified the byte-level round trip, so I would treat that as a lead rather than a finding. Two other things I ran into that may be worth scoping explicitly, since the RFC reads as though the targets are symmetric: **Hudi is a different problem, not a harder version of the same one.** Hudi has no position-based delete mechanism to translate into. RFC-103 in `apache/hudi` contrasts its key-based delete blocks with Delta and Iceberg's position-based vectors, and states that cross-format interoperability is out of scope for it. Converting a Delta deletion vector to Hudi would mean resolving row positions to record key values, which requires reading the data files rather than the deletion vector's own side file. That is a materially larger commitment than the Iceberg case and might deserve a separate decision. **Paimon may be nearly free.** Its deletion vectors are documented as using the same bitmap layout when the table is configured with `bitmap64 = true`, which would make that path closer to a byte copy than a conversion. Happy to help verify the Puffin round trip against a real deletion-vector fixture if that would be useful to the RFC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
