yihua opened a new pull request, #931:
URL: https://github.com/apache/hudi-rs/pull/931
## Description
A merge-on-read table written by hudi-rs today only grows log files: nothing
folds them back into base files unless Spark runs compaction on the table, and
the metadata table's log files (the record index in particular) grow the same
way. This adds experimental native compaction for both.
- **Data table compaction.** `Table::compact` (and `HudiTable.compact` in
Python) folds every latest file slice that has log files into a new base file
under one `compaction` instant. The requested instant carries Java's
`HoodieCompactionPlan`, and the instant completes as a `commit` marked
`compacted`, so Spark sees an ordinary compaction. It can also run after delta
commits once `hoodie.compact.inline.max.delta.commits` (default 5) is reached,
either in the write call (`hoodie.compact.inline`) or on a background task
while writes continue (`hoodie.datasource.compaction.async.enable`, awaited
with `await_compaction`). Both triggers are off by default.
- **Metadata table compaction, inline.** After a data write completes,
inside its completion critical section, the metadata table is compacted once it
holds `hoodie.metadata.compact.max.delta.commits` (default 10, as in Java)
delta commits since its last compaction, and only while no data instant is
pending. Base records no log block touches are copied as raw HFile key and
value bytes without an Avro decode, and record index merges take the newest
record's bytes, so a record index compaction decodes nothing. This is on by
default, matching Java.
- `HFileWriter` now sits on a streaming multi-block `HFileBuilder` (1 MiB
data blocks for compacted bases). Its existing single-block output is
byte-identical to before.
Compaction plans under the table lock, so a delta commit that completes
while a compaction is planning is never left on the superseded slice. A crashed
compaction of either kind is rolled back before the next one plans; Java would
re-run the plan instead. Pending data compaction plans that another engine
scheduled are refused rather than rolled back. Metadata records written with a
different Avro schema are also refused rather than resolved, because generic
resolution would retype stats bounds. `docs/writer-design.md` describes both
compactions.
The diff is about 3,300 added lines, roughly 40% of them tests, over the
1,000-line guideline: the data and metadata compactions share the commit and
rollback plumbing, so splitting them would land the first half without the part
the record index benchmark needs. It also fixes `table_write_runtime_tests`,
which stopped compiling on main because #884 still calls `with_metadata`, which
#927 renamed to `with_metadata_table_enabled`.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Unit tests cover the plan's Avro round trip, a delta commit landing while a
compaction waits for the lock (it fails if planning moves back outside the
lock), the multi-block HFile builder, compaction output and timeline shape,
inline and background triggers, crash rollback of both compactions, a pending
data write holding metadata compaction back, and every metadata partition's
merged view being unchanged by compaction on COW and MOR. A unit test feeds
base values that are not valid Avro and checks they come through compaction
byte for byte. Python tests cover `compact` and `await_compaction`. Spark
parity scenario K (Spark 3.5.5, Hudi 1.1.1) has rs compact an rs-written MOR
table while its handles compact the metadata table every two delta commits,
then checks Spark reads (snapshot and time travel, with and without the
metadata table), Spark upserts on the compacted base files, rs compaction of
Spark's log files, and a pending rs compaction plan. Spark parity results for
the fina
l commit will follow in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
