yihua opened a new pull request, #765:
URL: https://github.com/apache/hudi-rs/pull/765
## Description
The TPC-H harness only built copy-on-write tables, so it could not measure
what the merge path costs on top of a plain base file read. This adds
`--table-type mor` to `create-tables`, `bench-*` and `validate`: the same CTAS
creates merge-on-read tables, and one `UPDATE` per table in the same Spark
session follows it, so that every file group carries exactly one log file and
every read goes through the file group reader's merge path.
The update rewrites `--update-fraction` of each table's rows (default 0.001)
with their own values, plus the smallest key in every file group so the small
tables get a log file too. Query results are identical to the parquet source,
which keeps `validate` meaningful on the merge-on-read tables. Tables without a
real ordering column (everything but `lineitem` and `orders`) now merge by
commit time instead of using their own key as `preCombineField`.
Because the values are unchanged, `validate` proves the merge through
`_hoodie_commit_time`: for each MOR table it counts the rows stamped with the
update commit and requires the count to equal `numUpdateWrites` from that
commit's metadata. `create-tables` prints the file layout per table and fails
if any file group is missing its log file. Merge-on-read tables default to
`data/sf{N}-hudi-mor` and persist under the `hudi-mor` format label, so both
types can be built from one parquet set and compared. `--reader-version 1` pins
the previous file group reader as a baseline (results get a `-r1` suffix). That
reader has no commit-time merge and would read the default merge-on-read tables
append-only, so the harness refuses the pin on them and says how to build
tables both readers merge. One side effect on the existing Spark leg: a
`--format parquet` run is now labelled `spark+parquet` instead of `spark+hudi`.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Unit tests cover the CTAS and UPDATE renderers (table type, ordering mode,
hash modulus, composite keys, first key per file group, fraction bounds).
End to end at SF 0.01 with Spark 3.5.8 and `hudi-spark3.5-bundle_2.12:1.1.1`
on a table version 9 layout: `create-tables --table-type mor` produced 8
tables, each with 1 file group and 1 log file; `validate --table-type mor`
passed 22/22 queries against parquet and the merge check matched on all 8
tables (for example 77 update records written, 77 rows merged for `lineitem`).
With `RUST_LOG=hudi_core::file_group::reader=debug`, every one of the 93 file
slice reads in that run was served by file group reader version 2 and none by
version 1. `bench-datafusion` with `--table-type mor` and the existing COW path
both ran and persisted under distinct labels, and `compare --runs` charts them
side by side; `--reader-version 1` on the merge-on-read tables is refused with
the message above. The COW path was re-run (`create-tables`, `validate`,
`bench-datafusion`) to confirm the ordering-mode change did not alter it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
