yihua opened a new pull request, #815:
URL: https://github.com/apache/hudi-rs/pull/815
## Description
closes #814
A parquet log block reaches the merge in its file schema, and the drain
reconcile refused the pairings Java repairs with `SchemaRepair`: tz-aware
micros labelling millis values, and a plain long the table declares as a local
timestamp. MOR reads with such log blocks failed with `reconcile: column
'ts_millis' type Timestamp(us, "UTC") incompatible with target Timestamp(ms,
"UTC")`.
Reconcile now relabels exactly those leaf pairings, at any depth, keeping
the stored i64, which matches the base file path and Java. Every other leaf
difference is still refused. Because a PK-safe row filter pushed into such a
block would be evaluated on the mislabelled values and drop matching rows
before the merge, the decoder now withholds the row filter from a parquet block
whose footer needs the repair against the table schema, and from every parquet
block when the table schema is unknown, as the base file read already does for
a mislabelled base file.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Unit tests cover the reconcile relabel for top-level, struct, list and map
leaves (values, nulls and slice offsets preserved, and the pairings outside
Java's repair still refused), the `schema_needs_logical_type_repair` gate, a
parquet block decode showing the pushed predicate would drop a matching row and
is withheld when the repair is needed or the table schema is absent, and that
the log record reader hands the decoder the table schema rather than the
required schema. `cargo test -p hudi-core --lib`, `cargo fmt --check` and
`cargo clippy -D warnings` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
