linliu-code opened a new pull request, #686:
URL: https://github.com/apache/hudi-rs/pull/686
> **Stacked on #682, which is stacked on #678. Review those first.** This
targets `main`, so it shows their 11 commits alongside its own 1. The delta
that belongs to this PR is the last commit: 4 files, +300/-6.
First slice of #684. Does not close it: the header walk is still
synchronous, so the blocking bridge stays for now.
## Description
Pass 3 issued one ranged read per admitted log block, serially, through the
blocking bridge. By that point the gates have already chosen the set, so every
range is known before any byte is read and they can go out together.
`get_ranges` coalesces ranges that sit close together, so a run of adjacent
blocks costs one round trip rather than one each. Nothing in the crate called
`get_ranges` or `get_byte_ranges` before this.
The fetch moves into the async layer and Pass 3 keeps doing no I/O. That
split is not only tidiness: making Pass 3 itself `async` held the record buffer
across an await, and `HoodieFileGroupRecordBuffer` is not `Sync`, so the scan
future stopped being `Send`. Fetching in the caller and passing the bytes down
avoids that entirely.
Batches are bounded by `hoodie.memory.dfs.buffer.max.size` rather than
issued all at once. Reading a whole slice's admitted log content up front would
give back the peak-memory property #682 was justified on (+200 MB to +40 MB on
a 91 MB file). A block larger than the budget still goes out on its own, since
its content has to be whole to decode.
Ranges are grouped per file before batching, because a `LogBlockFetcher`
reads one file.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Two tests, and they exist because of a specific hazard: output is identical
whether the prefetch works or not, since Pass 3 falls back to fetching per
block. Stubbing the prefetch to return nothing passed all 1309 existing tests.
So the batching needed tests that assert the mechanism rather than the result.
- `test_read_contents_matches_reading_each_range_alone` asserts a batched
read is byte-identical to reading the same ranges one at a time. Fewer round
trips must not mean different bytes.
- `test_prefetch_returns_content_for_every_admitted_block` asserts every
admitted block comes back with exactly its own range. This is the one that
kills the stub-it-out mutation.
Both need a log file with more than one block, and no shipped fixture has
one: every file under `crates/core/tests/data/log_files` holds exactly one.
They concatenate a one-block fixture with itself, which is a valid two-block
file since log blocks are self-contained.
Verified: `cargo fmt --check`, clippy with and without default features,
1311 lib tests, 1291 with `--no-default-features`, the gold parity sweep green
across all fixtures and both reader versions, and `hudi-cpp` plus `hudi-python`
still compile.
## What is not covered
The round-trip saving itself is **not measured**. `memory_bench` measures
memory, not latency, and a local filesystem will not show the win at all, since
it is a round-trip cost. Quoting a number needs a benchmark against real object
storage, which is worth doing before this is claimed as a performance
improvement rather than a structural one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
