wombatu-kun opened a new pull request, #20526:
URL: https://github.com/apache/hudi/pull/20526
### Describe the issue this Pull Request addresses
Closes #20525
Part of #20449.
### Summary and Changelog
Data skipping now keeps the files that span an instant at which the local
date ran backwards, so filters on a timestamp's local date stop pruning files
that hold matching rows around historical fall-backs across midnight.
- `DataSkippingUtils`: a translated leaf predicate that renders a timestamp
in a time zone (`CAST(ts AS DATE)`, `to_date(ts)`, `date_format`,
`from_unixtime`) is ORed with `min < T AND max >= T` over the rendering's input
for each instant T at which the zone's local date runs backwards, since only a
file spanning such an instant can hold rows that render outside of [f(min),
f(max)]. The new `localDateReversals` reads the instants from the zone's
`ZoneRules` (resolved with Spark's `DateTimeUtils.getZoneId`, cached per zone
id); a zone whose current rules keep doing it gets no translation. Compound
predicates are composed from guarded leaves, `RuntimeReplaceable` expressions
such as `to_date` are looked through, and the bounds are evaluated through
`TryEval`, so that a bound the predicate itself never evaluates (for ex, an
unparseable string min under `f(col) >= v` with ANSI) cannot fail the lookup.
- Tests: `TestDataSkippingUtils` covers `CAST`, `to_date`, `date_format`,
`from_unixtime`, `date_add` over the cast, `IN`, `NOT` and `AND` across the
`America/St_Johns` fall-back of 2009-11-01, with files that do not span it, or
that another conjunct rules out, still pruned, plus an unparseable min under
ANSI. It also pins the instants: St_Johns, midnight fall-backs that repeat a
date without reversing it (`Asia/Beirut`, `Pacific/Apia` in 1892), zones
without any, and a current rule that reverses the date every year.
### Impact
Correctness fix: such queries could silently miss rows through the column
stats index; partition stats skip these filters and expression indexes strip
the function. Pruning is only lost for files whose [min, max] spans one of the
instants, all historical in tzdb 2026a (the issue lists the zones). Out of
scope: Spark's own rewrite of `CAST(ts AS DATE) = DATE'...'` into a range on
`ts` (`UnwrapCastInBinaryComparison`), which has the same problem, and
renderings through `CAST(ts AS TIMESTAMP_NTZ)`, a cast that breaks at every DST
fall-back and that #20513 stops treating as order-preserving. Once this lands,
the NOTEs in #20521 ("A date-only pattern is assumed unaffected") and #20513
("the date does not repeat") can point to this guard.
### Risk Level
low. It only keeps more files, and only those that span such an instant. The
new St_Johns cases fail without the fix (except the one pinning that another
conjunct still prunes), each of twelve mutants of the guard fails at least one
case, and the SQL repro from the issue returns the expected rows with it.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
