yihua opened a new pull request, #19169:
URL: https://github.com/apache/hudi/pull/19169
### Describe the issue this Pull Request addresses
Closes #18791.
With data-skipping (column-stats index) enabled, a `LIKE 'prefix%'`
predicate (Spark Catalyst `StartsWith`) silently drops rows in 1.x/master when
a file's minimum value is lexicographically greater than the literal prefix,
e.g. a file whose values all start with `abc` has `minValue = "abc123" >
"abc"`. This is a regression from 0.15.x and produces empty/short results with
no error.
### Summary and Changelog
The `StartsWith` translation reused `genColumnValuesEqualToExpression`,
producing `colMin <= prefix AND colMax >= prefix`. That checks whether the
prefix falls inside `[min, max]`, which wrongly prunes files whose values all
start with the prefix (their `min > prefix`).
- Add `genColumnStartsWithExpression` in `ColumnStatsExpressionUtils`,
producing `colMax >= prefix AND (colMin < prefix OR colMin startsWith prefix)`.
This keeps a file iff its `[min, max]` range overlaps the range of all strings
carrying the prefix (`[prefix, upperBound(prefix))`), expressed without
materializing the successor string.
- Point the `StartsWith` case in `DataSkippingUtils` at the new helper and
correct the misleading comment.
- Add a regression case to `TestDataSkippingUtils` (`file_4`,
`minValue="abc123"`, `maxValue="abc345"`, query `B.startsWith("abc")`) that the
old predicate pruned and the new one keeps.
### Impact
Correct results for `LIKE 'prefix%'` queries when data-skipping is enabled.
No API or config change. The `NOT(StartsWith)` path is unchanged.
### Risk Level
low - localized predicate-translation change covered by a red/green unit
test in `TestDataSkippingUtils`.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
