yihua opened a new pull request, #880:
URL: https://github.com/apache/hudi-rs/pull/880
## Description
closes #879
Adds `benchmark/upsert`, a single-node benchmark for the write path that
anyone can rerun from the repository. It drives the Python write API (#878)
through five mutation workloads: random UUID keys, TPC-DS `web_returns` with a
composite key (#871), NYC taxi trips, a chunk table with 384-dimension
embeddings re-synced document by document, and a training set read
incrementally and by time travel. Each runs on copy-on-write and merge-on-read
tables, and on copy-on-write without the record-level index to isolate it.
Every run's final table is compared, column types included, against a plain
Parquet copy of the expected result built by replaying the same commits, and
incremental and time-travel reads are checked by key and value. A headroom gate
refuses timings taken on a loaded machine, runs hold a lock so two cannot share
a work directory, and `--repeat` reports the median and spread. The README's
Reproduce section gives the exact commands; it needs a hudi build that includes
#878.
The diff is about 4,300 added lines: the harness itself is about 2,500, its
tests about 1,100, and the README about 500. No results are committed; numbers
come from a full run on an idle machine.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
`benchmark/upsert/tests` (69 tests, `pytest`) covers the oracle comparison
including type mismatches, the read-back, incremental and time-travel checks,
the headroom gate, the run lock, repetition statistics, and workload
determinism. A smoke run of every scenario on all three legs passed both the
sampled and the oracle checks with a build of #868, #869, #871, #873, #877 and
#878 together. `ruff check` and `ruff format --check` pass with the repo's
config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
