voonhous opened a new issue, #19445:
URL: https://github.com/apache/hudi/issues/19445
### Describe the problem
`HoodieSparkTypeUtils.isCastPreservingOrdering` (hudi-spark-common) only
rejects `String <-> Numeric` pairs and returns `true` for every other cast,
including narrowing numeric casts.
`BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation` uses it to
decide whether a `Cast` over a column can be mapped back to the source
attribute for data skipping, and
`DataSkippingUtils.translateIntoColumnStatsIndexFilterExpr` then rewrites the
predicate over the column's min/max through the cast. A non-monotonic cast
makes `[cast(min), cast(max)]` an invalid bound for the cast values, so files
containing matching rows can be pruned.
### Repro sketch
bigint column `a`, one file with values `{1, 2147483647, 4294967297}` -> col
stats `min=1`, `max=4294967297`. Query filter `cast(a as int) > 100` in
non-ANSI mode:
- the matcher accepts the cast, so the filter is translated to
`cast(a_maxValue as int) > 100`
- `cast(4294967297L as int)` wraps to `1`, the translated filter is false,
the file is pruned
- but the file holds `a=2147483647` whose cast is `2147483647 > 100` ->
silently missing row
### Related gaps in the same whitelist
- The `Multiply`/`Divide` arms of `OrderPreservingTransformation` match any
literal operand, including negative literals, which reverse ordering.
- On Spark 4.x `StringType` is collation-parameterized; casts to/from a
non-default-collation string do not equal the `StringType` companion in the
match and fall through to `case _ => true`.
### Suggested fix
For numeric-to-numeric pairs require `Cast.canUpCast(from, to)`; keep the
`String <-> Numeric` rejections; handle the collation cases explicitly.
Found while reviewing #19405, which pins the current behavior with a TODO
referencing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
