linliu-code opened a new pull request, #836:
URL: https://github.com/apache/hudi-rs/pull/836
## Description
<!--- Describe your changes in detail -->
Count HFile checksum chunks over header + data, matching the writer.
Counting over the on-disk size includes the checksums themselves and can
overcount near a checksum-chunk boundary. The reader then cuts the compressed
payload four bytes short, losing the GZIP trailer and failing with `GZIP
decompression failed: unexpected end of file`.
For example, 163819 bytes of header + data require 10 chunks of 16384 bytes
and 40 checksum bytes. Counting over the 163859-byte on-disk size instead gives
11 chunks and 44 checksum bytes.
Use `on_disk_data_size_with_header` to calculate the chunk count. Add a
header-only regression and a synthetic GZIP block generated with `GzEncoder` at
`Compression::none()`. The synthetic test searches for N=16383/C=16350, asserts
that including checksums increases the chunk count, and checks that parsing
returns the original input. It uses no external fixture.
<!--- If it fixes an open issue, please link to the issue here. -->
<!--- Please link any related issues and PRs as well. -->
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Tests were added before changing the reader. On the unchanged calculation,
`cargo test -p hudi-core hfile::block` produced:
```text
test_checksum_bytes_count_excludes_the_checksum_bytes ... FAILED
left: 44
right: 40
test_parse_gz_block_at_checksum_chunk_boundary ... FAILED
DecompressionError("GZIP decompression failed: unexpected end of file")
test result: FAILED. 17 passed; 2 failed; 0 ignored
```
With the fix, `cargo test -p hudi-core` produced:
```text
test_checksum_bytes_count_excludes_the_checksum_bytes ... ok
test_parse_gz_block_at_checksum_chunk_boundary ... ok
test result: ok. 1485 passed; 0 failed; 5 ignored
```
All executed integration tests and doctests also passed. Rust checks use
`make check-rust` (workspace Clippy with warnings denied and formatting), plus
the CI no-default-features Clippy command. Python style checks use `make
check-python`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
