fhan688 commented on PR #19169:
URL: https://github.com/apache/hudi/pull/19169#issuecomment-6309889254
While comparing #19169 with #20392, I noticed two correctness edge cases
that may be worth addressing here or in a follow-up:
1. **Missing min/max statistics**
The generated predicate currently evaluates to `NULL` when either bound
is missing:
```text
max >= prefix AND (min < prefix OR startsWith(min, prefix))
Since NULL is treated as false by the index-table filter, the corresponding
file is pruned. This is safe when the column is known to contain only nulls,
but missing bounds may also mean that usable statistics are unavailable. In
that case, pruning could introduce another false negative.
Would it be safer to conservatively retain such files?
```text
min IS NULL OR max IS NULL OR
(max >= prefix AND (min < prefix OR startsWith(min, prefix)))
````
2. Spark 4 non-binary collations
The current pattern accepts any UTF8String-backed StartsWith expression. On
Spark 4, that may include strings with a non-binary collation. The StartsWith
and comparison semantics of a collated string may not match the binary ordering
represented by the column-stat min/max values.
Unless the column-stat ordering is guaranteed to use the same collation, it
may be safer to translate only binary-string expressions and fall back
toTrueLiteral for non-binary collations. For example, the translation could
require:
```text
sourceExpr.dataType == StringType && value.dataType == StringType
```
I have coverage for both cases in #20392. If you prefer to keep this PR
focused on the reported regression, I can move these checks and tests into a
separate follow-up after #19169 merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
