wombatu-kun opened a new issue, #20518:
URL: https://github.com/apache/hudi/issues/20518

   ### Bug Description
   
   **What happened:**
   
   With the column stats index enabled, a filter on `from_utc_timestamp(ts, 
tz)` or `to_utc_timestamp(ts, tz)` can prune a file that holds matching rows 
when the offset of `tz` changes, as it does at daylight saving transitions. 
Data skipping treats both functions as order-preserving for any zone 
(`BaseHoodieCatalystExpressionUtils`) and re-applies them to the file's min/max 
stats, but they shift by the zone's offset at that time, so they keep the 
ordering only in a zone whose offset never changes. In `America/New_York`, 
`from_utc_timestamp` runs backwards across the fall-back (05:30Z becomes 01:30, 
the later 06:10Z becomes 01:10), and `to_utc_timestamp` shifts the 
spring-forward gap past later times (02:30 read as New York time becomes 
07:30Z, the later 03:00 becomes 07:00Z). The partition stats index goes through 
the same translation.
   
   Part of #20449 (time-zone conversions across DST transitions). 
`date_format(ts, 'yyyy-MM-dd HH:mm')` in a DST session zone breaks the same 
way, but through the format arms, and is not covered here.
   
   **What you expected:**
   
   The matching row returned.
   
   **Steps to reproduce:**
   
   ```sql
   SET spark.sql.session.timeZone = UTC;
   
   CREATE TABLE t (id INT, ts TIMESTAMP) USING hudi
   TBLPROPERTIES (primaryKey = 'id', 
'hoodie.metadata.index.column.stats.enable' = 'true');
   
   INSERT INTO t VALUES (1, TIMESTAMP'2026-11-01 05:30:00'), (2, 
TIMESTAMP'2026-11-01 06:10:00');
   
   SELECT id FROM t WHERE from_utc_timestamp(ts, 'America/New_York') > 
TIMESTAMP'2026-11-01 01:20:00';
   -- expected: 1 row (id = 1)
   -- actual: no rows
   ```
   
   `to_utc_timestamp` fails the same way across the spring-forward gap:
   
   ```sql
   CREATE TABLE t2 (id INT, ts TIMESTAMP) USING hudi
   TBLPROPERTIES (primaryKey = 'id', 
'hoodie.metadata.index.column.stats.enable' = 'true');
   
   INSERT INTO t2 VALUES (1, TIMESTAMP'2026-03-08 02:30:00'), (2, 
TIMESTAMP'2026-03-08 03:00:00');
   
   SELECT id FROM t2 WHERE to_utc_timestamp(ts, 'America/New_York') > 
TIMESTAMP'2026-03-08 07:15:00';
   -- expected: 1 row (id = 1)
   -- actual: no rows
   ```
   
   With `hoodie.enable.data.skipping=false`, both queries return the row.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT); `release-1.2.1` and 
`release-1.1.2` carry the same matcher arms
   **Query engine:** Spark 3.5
   **Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`, 
`hoodie.enable.data.skipping=true` (default)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to