linliu-code opened a new pull request, #702:
URL: https://github.com/apache/hudi-rs/pull/702
## Description
**An integration branch, not a replacement for review.** This squashes the
eighteen open PRs #678–#700 into one commit so the stack gets a single CI run.
Every one of those PRs stays open, and the per-PR diff remains the reviewable
unit — please review them there, not here.
Why it exists: CI has never run on any of #678–#700, because a fork PR needs
a committer to approve the workflow, and eighteen approvals is eighteen chances
to stop. One branch is one approval and one signal on the tree that would
actually merge.
What it contains, in the order the stack built it:
- **Correctness in the log path** — #678 gate the log-block scan on instant
state; #679 fold partial updates in both ordering directions; #680 cover
event-time ordering losing in the gold corpus; #682 walk log block headers
before the gates decide
- **The reader made asynchronous** — #686 read log files asynchronously in
batches; #687 the merge iterator owns the base-file pull; #688 merge base and
log as a stream, deleting the synchronous bridge; #689 bound the merge chunk
- **HFile support** — #691 ranged, window-coalesced base file reads; #692
merge HFile log blocks instead of dropping them; #694 seek to the blocks a key
predicate can satisfy; #697 copy a record rather than its whole block, making
block iteration linear instead of quadratic
- **The metadata table** — #693 merge its records by the payload's own rule;
#695 read the files partition through the v2 reader; #696 decode a log block
holding more than one batch, and switch the read seam over
- **Two leaves** — #699 resolve the base file format from the path rather
than config alone; #700 read the metadata table as Arrow through a plain C ABI,
so a JVM caller reaches it without Gluten or Velox
- **Build ergonomics** — #681 let consumers opt out of the merge map's
on-disk tier
Two things worth knowing about how it was assembled. **#692 was stranded**:
everything above #691 had been built on #693, so `4816cc7` was an ancestor of
no other PR and the stack tip did not contain the log-block merge fix. It is
rebased in here, and its two tests needed one adaptation —
`resolve_reader_context` gained the per-path format argument from #699. **#694
and #697 conflict** in `crates/core/src/hfile/key.rs` over `content_offset`
visibility against `from_content`; resolved by keeping both, with
`content_offset` private.
The diff is far over this repo's 1000-line guidance. That is the point of
the branch rather than an oversight: it is eighteen separately reviewed changes
in one commit, not one change of that size.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Verified on this tree, after the squash:
```
cargo test -p hudi-core --lib 1397 passed, 0
failed
cargo test -p hudi-core --lib --no-default-features 1377 passed, 0
failed
cargo test --workspace --all-targets --all-features 92
table_read_tests, 4 gold parity,
39 + 21 + 12
hudi-datafusion,
20 statistics,
6 hudi-jvm-ffi, all green
cargo clippy -p hudi-core --all-targets -- -D warnings clean
cargo clippy -p hudi-core --lib --no-default-features -- -D warnings clean
cargo fmt --all -- --check clean
```
The two tests ported from #692 were run by name after the rebase, since they
were the ones that could have rotted against the newer API:
`v2_reads_every_metadata_partition_with_its_log_blocks` and
`v2_reads_a_files_slice_matching_the_metadata_table_reader`, both pass.
**Not covered, and stated rather than implied.** The JVM half of #700's
ownership contract — that the Arrow stream is freed exactly once including on
an exception path, and that a Rust panic does not abort a real JVM — is
asserted on the Rust side only. Proving it from Java needs a test module this
repo does not have yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
