wombatu-kun opened a new pull request, #20521: URL: https://github.com/apache/hudi/pull/20521
### Describe the issue this Pull Request addresses Closes #20520 Part of #20449. ### Summary and Changelog Data skipping now treats `date_format` and `from_unixtime` as order-preserving only when the session time zone cannot reorder what the pattern renders, so time-of-day filters in a DST session zone stop pruning files that hold matching rows. - `BaseHoodieCatalystExpressionUtils`: the `DateFormatClass`/`FromUnixTime` arms now require `isZoneSafePattern`. Either the literal pattern has only date fields (era, year, quarter, month, week, day; quoted text and non-ASCII characters ignored), or the session zone has a fixed offset, checked by the new `isFixedOffsetZone`, which resolves it with Spark's `DateTimeUtils.getZoneId` as the functions do when evaluated. - Tests: `TestDataSkippingUtils` covers both functions across the `America/New_York` fall-back next to a date-only pattern that still prunes; `TestHoodieCatalystExpressionUtils` pins time-of-day, offset, date-only, quoted, non-ASCII, fixed-zone, non-literal, null, unknown and unresolved cases. ### Impact Correctness fix: such queries could silently miss rows through the column stats index; partition stats skip these functions. With a session zone whose offset ever changed, such as `America/New_York`, or `Asia/Shanghai` and `Asia/Kolkata` through historical changes, patterns with a time-of-day or offset field no longer prune; date-only patterns and fixed-offset session zones such as `UTC` or `+08:00` still do. Date-only patterns keep the assumption that `CAST(ts AS DATE)` also makes, that the local date never runs backwards: tzdb 2026a records historical exceptions where the clocks fell back across midnight, the last in 2010 (`America/St_Johns` at 00:01). Guarding them would end date-only pruning for every date in those zones, `America/Phoenix` (1944) and the Alaska zones (1867) included, so they are left out. Whether a pattern sorts chronologically (#19481) is unchanged. With #20519, this is the mechanism behind the remaining `date_format` patterns of the `dst_fall_back` family in the #20439 oracle. Shares the fixed-offset zone check with #20519; whichever lands second will reuse the other's. ### Risk Level low. It only removes pruning that was unsound. The new DST test cases fail without the fix, each of eight mutants of the guard fails at least one of them, and the SQL repro from the issue returns the expected rows with it. Only column stats pruning reaches these arms: the expression index strips the function before translating, and partition stats skip it. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
