slachiewicz commented on issue #339:
URL:
https://github.com/apache/incubator-xtable/issues/339#issuecomment-5385398076
I said I would treat the Puffin round trip as a lead rather than a finding
until it was verified
against a real fixture. That is now done, and the result is stronger than
the re-framing I guessed
at: for an on-disk deletion vector the bytes are identical.
The fixture is a Delta table written by Spark 3.4.2 + delta-core 2.4.0 with
`delta.enableDeletionVectors=true`, 1000 rows, then `DELETE WHERE id < 100`:
- the deletion vector's magic number decodes to `1681511377` (`0x6439D3D1`),
as `PROTOCOL.md` specifies
- a `roaring64` decode of the bitmap gives cardinality 100 at positions
`0..99`, matching the
descriptor's own `cardinality`
Delta stores each vector in its file as `[dataSize: 4B BE][magic +
serialized bitmap][checksum: 4B BE]`.
That is a Puffin `deletion-vector-v1` blob's payload as-is — the Puffin spec
says the choice was
deliberate:
> Big endian values were chosen for compatibility with existing deletion
vectors in Delta tables
So for `storageType` `u` or `p`, a converter can copy `sizeInBytes + 8`
bytes starting at the
descriptor's `offset` into a Puffin blob, without decoding the bitmap at all.
Two limits on that, both narrower than the headline:
- **Inline vectors (`storageType` `i`) are not a byte copy.** They hold the
bare serialized bitmap
with no length prefix and no checksum, so the framing has to be added.
- **This is a spec comparison plus a decode of real Delta bytes, not an
end-to-end interop test.**
I have not fed the resulting blob to an Iceberg Puffin reader.
One practical note for anyone building a fixture to check this
independently: `delta-rs` cannot
produce one. Version 1.6.3 does not write deletion vectors with or without
the table property, and
raises no error — `DeltaTable.delete()` and `merge().when_matched_delete()`
both rewrite the file
instead. In current `main`, every construction of a
`DeletionVectorDescriptor` in
`crates/core/src/operations/delete.rs` sits inside `#[cfg(test)]`; the only
production use reads an
existing vector's cardinality. Spark is the practical way to generate one,
and the property has to be
set at `CREATE TABLE` time rather than added by `ALTER TABLE` afterwards.
*This comment was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
