ump45nose opened a new pull request, #20468: URL: https://github.com/apache/hudi/pull/20468
### Describe the issue this Pull Request addresses Closes #20460 Spark data skipping treats `Add`, `Multiply`, and `ShiftLeft` as order-preserving and re-applies them to column min/max stats. For integral types that wrap on overflow (non-ANSI mode), and for `ShiftLeft` which is integral-only and always wraps, that can invert the bound and prune files that still contain matching rows. IEEE float/double saturate to +/- Inf and keep ordering, so those remain safe. ### Summary and Changelog - Restrict `Add` / `Multiply` matching in `OrderPreservingTransformation` to result types that do not wrap: `FloatType` and `DoubleType` only (via `isNonWrappingNumericType`) - Remove `ShiftLeft` from the order-preserving matcher (always wraps) - Leave `Divide`, `BitwiseOr`, and `ShiftRight` unchanged (out of scope for this overflow case; negative factors are covered by #20457) - Unit tests: float/double Add/Multiply still match; integral Add/Multiply/`ShiftLeft` return `None`; parameterized data-skipping cases for long overflow and a float control ### Impact Queries that filter on expressions like `col + 1`, `col * 2`, or `col << n` over integral columns will no longer translate those expressions into column-stats predicates. Data skipping for those shapes becomes more conservative (may read more files) but no longer incorrectly drops matching rows. Float/double filters are unchanged. ### Risk Level low — matcher is tightened only for wrapping numeric types; existing float/double paths keep working; covered by targeted unit tests. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable ### AI assistance AI-assisted (Cursor/Grok Bot harness). I reviewed the change against #20460, aligned it with the issue's suggested fix, and ran the unit tests below myself. ### Tests ``` mvn -pl hudi-spark-datasource/hudi-spark surefire:test \ -Dtest=TestHoodieCatalystExpressionUtils,TestDataSkippingUtils \ -Dspark3 -Dscala-2.12 ``` Results: Tests run: 70, Failures: 0, Errors: 0, Skipped: 0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
