ebyhr opened a new pull request, #3575: URL: https://github.com/apache/iceberg-python/pull/3575
# Rationale for this change Prepares PuffinFile for `apache-datasketches-theta-v1` blob support by removing the hard constraint that all blobs are `deletion-vector-v1` and by adding zstd blob decompression. Two fixes were required to make this work correctly: Fix: blob offsets were off by 8 _payload stored puffin[8:], but blob offset values in the Puffin footer are measured from byte 0 of the file. For the existing deletion-vector-v1 case this was harmless — the 8-byte shift accidentally cancelled the 8-byte blob framing ([length:4][magic:4]) that was never stripped. With arbitrary blob types and real offsets the cancellation breaks, so _payload now holds the full file bytes. Fix: PuffinBlobMetadata.type widened from Literal["deletion-vector-v1"] to str Required to parse Puffin files containing non-DV blobs without a validation error. ## Are these changes tested? Yes. Copied Puffin files from https://github.com/apache/iceberg/tree/main/core/src/test/resources/org/apache/iceberg/puffin/v1 ## Are there any user-facing changes? No. PuffinFile is internal. The type: str widening is backwards-compatible. <!-- In the case of user-facing changes, please add the changelog label. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
