yihua opened a new issue, #814: URL: https://github.com/apache/hudi-rs/issues/814
## Problem The base file read applies the same logical type repair Hudi's Java reader applies (`SchemaRepair` for parquet, `AvroSchemaRepair` for Avro): a column stored as tz-aware microseconds but declared by the table as milliseconds is relabelled, keeping the stored i64, and a plain `long` the table declares as a timezone-less timestamp is read as that timestamp. A parquet data log block does not get that treatment. It reaches the merge in its own file schema, and the drain-side reconcile in `reader_v2/buffer/row_extraction.rs` refuses any leaf type that differs from the table's. A merge-on-read read of a file group with such a log block fails with ``` reconcile: column 'ts_millis' type Timestamp(us, "UTC") incompatible with target Timestamp(ms, "UTC") ``` for the same table the base-file-only path reads fine, and the same table Java reads fine. ## Expected Reconcile relabels exactly the leaf pairings Java repairs (tz-aware micros labelling millis values; a plain long the table declares as a local timestamp), at any nesting depth, keeping the stored i64 so the log side agrees with the base side. Every other leaf difference is still refused. There is a second, quieter consequence. A primary-key-safe row filter pushed into such a parquet log block is evaluated on the mislabelled physical values and drops rows that actually match, before the merge sees them. The base file read already withholds the row filter from a file that needs the repair; the log block decoder should do the same for a block whose footer needs the repair against the table schema, and for every parquet block when no table schema is available to check against. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
