voonhous opened a new pull request, #20513: URL: https://github.com/apache/hudi/pull/20513
### Describe the issue this Pull Request addresses Closes #19445 `HoodieSparkTypeUtils.isCastPreservingOrdering` decides whether data skipping may re-apply a `Cast` to a column's min/max stats. It rejected a list of known-bad casts and returned `true` for everything else, and every soundness bug found in it so far (#19475, #20458) came through that fallthrough. One still open: `CAST(ts AS TIMESTAMP_NTZ)` takes the local wall-clock time, so across a DST fall-back 05:30Z becomes 01:30 but the later 06:10Z becomes 01:10, and a file holding a matching row is pruned. ### Summary and Changelog The function is now deny-by-default: only casts known to preserve ordering are accepted, and new types (Spark 4 collations, `TimeType`) are safe by construction. - Allowlist: identity, numeric up-casts, decimal to float/double, boolean to integral/float/double, timestamp to bigint/float/double, date to/from timestamp and timestamp_ntz, default string to/from binary. - String to date/timestamp stays allowed. Unpadded strings parse out of order, but the optimizer rewrites `to_date(col)` / `to_timestamp(col)` without a format into this cast, and expression indexes on them would stop pruning. Left to #19481. - Tests: `TestHoodieCatalystExpressionUtils` pins the new allowed and rejected pairs; the DST test in `TestDataSkippingUtils` adds the timestamp_ntz case (Spark 3.4+). ### Impact No public API change. Filters over casts outside the allowlist (for ex, numeric to timestamp, date to string) no longer prune. ### Risk Level low. The only new behavior is falling back to no pruning. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
