Akash3121 opened a new pull request, #10070:
URL: https://github.com/apache/paimon/pull/10070
### Purpose
Fixes #9992.
Deletion vectors already include a CRC32 checksum when they are serialized
by both `BitmapDeletionVector` and `Bitmap64DeletionVector`. However, the read
path previously skipped the final four checksum bytes without reading or
validating them:
```java
dis.skipBytes(4); // skip crc
As a result, a deletion vector with a corrupted checksum—or corrupted bitmap
data that remained structurally readable—could be accepted and used. In
addition, skipBytes(4) does not guarantee that all four bytes are available,
so a truncated checksum could also go undetected.
This change validates deletion-vector checksums before deserializing the
bitmap:
- Reconstructs the exact serialized bitmap-data range covered by the
writer's CRC32 checksum.
- Reads the stored checksum with DataInputStream.readInt() instead of
skipping it.
- Computes CRC32 over the magic number and serialized bitmap payload.
- Rejects checksum mismatches with an IOException containing the stored
and computed checksum values.
- Rejects truncated bitmap payloads or checksums through the strict
readFully and readInt operations.
- Rejects bitmap lengths smaller than the required magic-number size before
allocating or reading the payload.
- Preserves the existing length semantics for both deletion-vector
encodings:
- 32-bit BitmapDeletionVector
- 64-bit Bitmap64DeletionVector
- Performs checksum validation before bitmap deserialization, so corrupted
data is never exposed as a usable deletion vector.
A package-private DeletionVectorChecksum helper centralizes the checksum
read and validation logic without adding public API. It computes the checksum
incrementally while retaining only the bitmap payload required by the existing
deserializers, avoiding an additional copy of potentially large 64-bit bitmap
data.
Existing valid deletion-vector index fixtures remain readable, preserving
compatibility with previously written 32-bit and 64-bit deletion vectors.
### Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
