linliu-code opened a new pull request, #706:
URL: https://github.com/apache/hudi-rs/pull/706
## Description
Adds the file-group benchmark harness to the repository and extends it into
a gate for one requirement: **given a fixed amount of memory, a read may be
slower but must not fail.**
`fg-bench` existed as untracked local work and, in an older form, on an
internal branch. The two had drifted; the local copy is imported because it
carries a macOS `ru_maxrss` fix the other lacks — the BSDs report bytes where
Linux reports kilobytes, so reading it as kilobytes overstates peak RSS by
1024×, turning a 21 MB process into a 21 GB one. In a harness whose job is
measuring memory that is not cosmetic.
It is ported to the public `FileGroupReader` surface. The untracked copy
reached into `file_group::reader_v2`, which is `pub(crate)`, so it compiled
against neither repository — likely why it was never committed. The knobs
survive the move because they are Hudi config keys rather than constructor
arguments. What does not survive is `HoodieReadStats`; widening an internal
module to suit a benchmark is the wrong trade, so spill is instead observed
from outside.
### What it adds
- **`--slice-concurrency`** reads slices through `buffer_unordered`,
mirroring the DataFusion fan-out. Default 1 is sequential with no coordination
cost — the baseline a bounded-memory claim must not regress.
- **`--max-rss-bytes`** fails the run when peak RSS exceeds a declared
budget, with a non-zero exit.
- **`FG_BENCH_ALLOC_CAP_BYTES`** installs a global allocator that refuses
past a ceiling. This is the half `--max-rss-bytes` cannot do: an assertion made
after the allocation answers *"how much did this read want"*, never *"does it
survive on a small machine"*. Measured at no throughput cost (579ms against
621ms — noise).
- **`fg-gen`** generates a table of a target size, because a gigabyte cannot
be committed — the checked-in fixtures are tens of kilobytes. Table version 6,
whose commit metadata is JSON; version 8 encodes it as Avro, which a generator
would have to reimplement for no gain to the read path under test. With
`--log-files` it writes Hudi log files carrying Avro data blocks, so the table
is merge-on-read and the read builds a real merge map.
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
This is a benchmark harness, so its correctness is in whether its
instruments can fail. Both were verified in both directions:
| instrument | positive case | negative case |
| --- | --- | --- |
| RSS gate | 1 MiB budget on a 21 MB read → exit 1 | 4 GiB budget → exit 0 |
| allocation ceiling | 256 MiB on a 1 GiB eager read → exit 134, `memory
allocation of 32000000 bytes failed` | 4 GiB → exit 0 |
| spill detector | MOR with a 1 MiB merge budget → `spilled=true`, 1032 MB
peak | COW with no merge → `spilled=false` |
The spill detector was wrong twice, and both ways reported "never spilled"
for a read that spilled a gigabyte. It scanned one level deep while RocksDB
writes into a subdirectory; and once that was fixed it still saw nothing,
because RocksDB removes its directory when the reader closes — before the read
call returns — so a before/after sample finds an empty directory at both ends.
It now samples on a thread during the read and keeps the high-water mark. A
detector that only ever says "no" is indistinguishable from a broken one, which
is why the negative case is asserted too.
### What it measured
```
1 GiB / 10 files, concurrency 1
copy-on-write eager wall=709ms rss=419MB streaming wall=615ms
rss=61MB
merge-on-read eager wall=3801ms rss=770MB streaming wall=3776ms
rss=769MB
```
Streaming is 7× smaller on copy-on-write and **saves nothing** on
merge-on-read: the merge map must hold log records keyed for lookup before it
can emit a row, so streaming avoids retaining the output, which on COW was
everything and on MOR is a minority. With the merge budget honoured at 1 MiB,
the map spills **1032 MB** to disk and the resident set is still **1467 MB** —
spilling moves the accounted bytes out and RSS does not follow.
A related trap found while measuring: `hoodie.memory.merge.max.size` is a
**table-level** config, so passing it as a read option is dropped silently by
design (`TableBuilder`, "dropping rather than erroring"). It has no effect and
reports no error. Every measurement above sets it in `hoodie.properties`
instead.
**Not covered:** `fg-gen` writes a gigabyte and does not clean up after
itself. A harness that fills a laptop's disk will fill a CI runner's too, and
it filled mine while this was being built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
