linliu-code opened a new pull request, #704:
URL: https://github.com/apache/hudi-rs/pull/704
## Description
The required schema keeps a merge's own inputs even when the caller did not
ask for them — record key fields, ordering fields under event-time ordering,
the delete markers, `_hoodie_operation`. It never consulted the **custom
merger**, although `prepare_required_schema` had already resolved one into
`self.custom_merger`.
The metadata payload merger dispatches on a `type` column
(`metadata_merger.rs`, `record_type`), and `type` appeared in none of those
lists. So a requested schema that omitted `type` would have been honoured, the
base read would have projected it away, and the merge would have failed with
```
A metadata record must carry a 'type' column to be merged.
```
on a perfectly valid metadata table.
**That refusal is a correct guard and is left exactly as it is.** What was
wrong is that a projection could remove the column it guards.
The fix puts the knowledge where it belongs: `CustomMerger` now declares the
columns it reads, and the schema handler asks it, rather than the handler
carrying a column name owned by the merger. A second custom merger added later
cannot be forgotten at that one call site.
**Not reachable today, and the PR says so rather than overselling.** No
production code sets a requested schema — every caller of
`with_requested_schema` is under `#[cfg(test)]` (`schema_handler.rs`'s test
module, plus `harness.rs` and `harness_tests.rs`, both `#[cfg(test)]` in
`reader_v2/mod.rs`). With it unset, `generate_required_schema` falls back to
the full data schema, which always carried `type`. The protection was
accidental rather than designed, and the trigger is close: `hudi-datafusion`'s
`TableProvider` already does projection pushdown at the table level, and the C
ABI added for JVM callers exists specifically to let an outside caller drive
reads. The first caller that threads a requested schema into a metadata read
turns a correct guard into a spurious failure.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
One test, `the_required_schema_keeps_the_custom_merger_s_own_columns`. The
discriminating shape is a requested schema of `["key"]` against a table schema
of `["key", "type", "filesystemMetadata"]` on a `HoodieMetadataPayload` table —
`type` present in the table and absent from the request is the only shape that
distinguishes *adding it back* from *never having dropped it*. The test asserts
the column is present in the resulting required schema, and separately asserts
both of its own premises (that a merger really resolved, and that the request
really omits `type`) so it cannot pass vacuously.
Mutation-checked with a positive control, on both halves of the fix:
| mutation | result |
| --- | --- |
| remove the inclusion from `generate_required_schema` | FAILED — required
schema is `["key"]` |
| make `CustomMerger::required_field_names` return `&[]` | FAILED — required
schema is `["key"]` |
| unmutated | passes |
```
cargo test -p hudi-core --lib 1393
passed, 0 failed
cargo test -p hudi-core --lib --no-default-features 1373
passed, 0 failed
cargo test --workspace --all-targets --all-features 0
failing suites
cargo clippy -p hudi-core --all-targets -- -D warnings clean
cargo clippy -p hudi-core --lib --no-default-features -- -D warnings clean
cargo fmt --all -- --check clean
```
**Not covered:** there is no end-to-end test reading a metadata table
through a projection, because no production path can express one yet. When
projection pushdown reaches this reader, that test becomes possible and should
be added with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
