voonhous opened a new issue, #20460:
URL: https://github.com/apache/hudi/issues/20460
### Describe the problem
Data skipping treats `Add`, `Multiply` and `ShiftLeft` with a constant as
order-preserving and re-applies them to a column's min/max stats. Without ANSI,
integral arithmetic wraps around on overflow, so the transformed max can land
below the transformed min and files holding matching rows are pruned. A
positive factor does not help: `Long.MaxValue * 2L` wraps to `-2`.
### Repro
| Filter | File values | Translated bound | Matching row |
| --- | --- | --- | --- |
| `l * 2L > 100` (bigint) | `{51, Long.MaxValue}` | `max * 2L = -2 > 100`:
pruned | `51 * 2 = 102` |
| `i + 1 > 0` (int) | `{0, Int.MaxValue}` | `max + 1 = Int.MinValue > 0`:
pruned | `0 + 1 = 1` |
| `shiftleft(i, 1) > 0` (int) | `{1, Int.MaxValue}` | `max << 1 = -2 > 0`:
pruned | `1 << 1 = 2` |
The oracle test in #20439 reproduces this as the `add_overflow`,
`multiply_overflow` and `shiftleft_overflow` families (column and partition
stats).
### Suggested fix
In `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation`, match
`Add`, `Multiply` and `ShiftLeft` only when the result cannot wrap:
floating-point results saturate to infinity and keep the ordering. Integral
results, and decimal results that can overflow into NULL, fall back to no
pruning. This costs pruning for filters such as `int_col + 1 > 5`, and
`ShiftLeft` (integral only) stops pruning entirely.
Split out of #19445 (one issue per failure mechanism, see #20449). The sign
check for `Multiply`/`Divide` is in #20457; this fix builds on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
