wombatu-kun opened a new issue, #20520: URL: https://github.com/apache/hudi/issues/20520
### Bug Description **What happened:** With the column stats index enabled, a filter on `date_format(ts, fmt)` or `from_unixtime(sec, fmt)` can prune a file that holds matching rows when `fmt` has a time-of-day or offset field and the session time zone changes its offset, as it does at daylight saving transitions. Data skipping treats both functions as order-preserving for any pattern and zone (`BaseHoodieCatalystExpressionUtils`) and re-applies them to the file's min/max stats, but they render the time in the session time zone, where local time runs backwards across a fall-back: in `America/New_York`, 05:30Z renders as `01:30` and the later 06:10Z as `01:10`. Date-only patterns such as `yyyy-MM-dd` are unaffected under current rules, since the local date only runs backwards where the clocks fall back across midnight; tzdb last records that in 2010 (`America/St_Johns` at 00:01), and that historical case, which `CAST(ts AS DATE)` shares, is not covered here. The partition stats index skips these functions, so only file pruning is affected. Part of #20449 (time-zone conversions across DST transitions), next to #20518. Patterns that do not sort chronologically, such as `MM/dd/yyyy`, are a separate problem tracked in #19481. **What you expected:** The matching row returned. **Steps to reproduce:** ```sql SET spark.sql.session.timeZone = UTC; CREATE TABLE t (id INT, ts TIMESTAMP, sec BIGINT) USING hudi TBLPROPERTIES (primaryKey = 'id', 'hoodie.metadata.index.column.stats.enable' = 'true'); INSERT INTO t VALUES (1, TIMESTAMP'2026-11-01 05:30:00', 1793511000), (2, TIMESTAMP'2026-11-01 06:10:00', 1793513400); SET spark.sql.session.timeZone = America/New_York; SELECT id FROM t WHERE date_format(ts, 'yyyy-MM-dd HH:mm') > '2026-11-01 01:20'; -- expected: 1 row (id = 1) -- actual: no rows SELECT id FROM t WHERE from_unixtime(sec, 'yyyy-MM-dd HH:mm') > '2026-11-01 01:20'; -- expected: 1 row (id = 1) -- actual: no rows ``` With `hoodie.enable.data.skipping=false`, both queries return the row. ### Environment **Hudi version:** master (1.3.0-SNAPSHOT); `release-1.2.1` and `release-1.1.2` carry the same matcher arms **Query engine:** Spark 3.5 **Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`, `hoodie.enable.data.skipping=true` (default), a `spark.sql.session.timeZone` with DST -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
