jackylee-ch opened a new pull request, #727: URL: https://github.com/apache/paimon-rust/pull/727
A BTree global index written by Java with LZ4 block compression cannot be read at all: every query fails with `Compression type Lz4 not supported`, even though the file holds the same data as the zstd and uncompressed variants. The repo already carries the Java-written fixture for it, used only to assert that the read fails. Paimon's LZ4 block is not the LZ4 frame format and not a bare block either. It is the block writer's var-int uncompressed length, then `Lz4BlockCompressor`'s 8-byte header of two little-endian `i32`s (compressed and uncompressed length), then the raw LZ4 block. Reading the fixture's bytes confirms the layout: the var-int says 261 and the header repeats 261 alongside a compressed length of 203. **Fix**: decode that frame in `decompress_block`. The uncompressed length is stored twice, so the two copies are cross-checked and a mismatch is rejected as corrupt, as are negative lengths and a payload shorter than the header claims. `lz4_flex` is already a dependency. Scope is the read path only. The writer still falls back to no compression for LZ4, unchanged — the comment is reworded to say so. A round-trip test would have passed either way, so the two new tests read the Java fixture directly and assert all three codecs decode to the same rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
