linliu-code opened a new pull request, #691:
URL: https://github.com/apache/hudi-rs/pull/691
## Description
**Stacked on #689, review that one first.** Until it merges, the diff here
shows the stack's commits as well as this one; the delta is `a744c12..HEAD`,
four files. Replaces #690, which implemented the same capability by holding the
whole file.
The version two file group reader selects a base file reader by format, but
`create_base_file_reader` refuses HFile, so a metadata table file slice cannot
be read through it. This adds an `HFileBaseFileReader` and selects it inside
the version two reader. The shared factory arm is left refusing, so the legacy
reader and the two tests that pin that refusal are unaffected.
The decoder now reads through a byte source rather than one buffer. A
resident source slices bytes already in hand, which is what an HFile arriving
as a log block's content is; a ranged source reads the trailer and the
load-on-open section up front, then fetches data blocks as they are read. Data
blocks are grouped into runs that stay under
`hoodie.memory.dfs.buffer.max.size` and read one request per run, so a full
scan costs a request per window rather than one per block, and peak memory
tracks one window rather than the file. The multi-level index walk fetches a
level per request for the same reason.
Scope is the base file only. Merging HFile log blocks, key and prefix
pushdown, and decoding the record value into typed columns are each separate
changes; this reader emits the key and the raw value bytes as an HFile stores
them. `MetadataTableFileGroupReader` and the log block decode path still read
resident bytes.
No latency measurement is claimed. The window sizing follows from
request-count arithmetic and from the pattern the log file path already uses,
not from a benchmark; proving it on real data needs a production metadata table
HFile with a multi-level index, which no fixture here provides.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Three additions. Every fixture under `crates/core/tests/data/hfile/` is read
both ways and the records compared, which is the guard on the two multi-level
index walks being separate code paths; the deep index fixture is the one that
exercises the split. A metadata table `files` partition base file is read
through the version two reader and its key set asserted equal to what
`MetadataTableFileGroupReader` returns for the same base file only slice,
failing before this change with `UnsupportedBaseFileFormat`. Window planning is
covered directly, including that a block larger than the budget is still read
whole.
Run locally: 1331 pass with default features, 1311 with
`--no-default-features`, clippy clean on both feature sets with warnings
denied, and `cargo check --workspace --all-targets --all-features` clean
including the Python and C++ bindings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
