wombatu-kun opened a new issue, #20525:
URL: https://github.com/apache/hudi/issues/20525
### Bug Description
**What happened:**
With the column stats index enabled, a filter on a timestamp's local date
can prune a file that holds matching rows when the session time zone once fell
back across midnight. Data skipping treats `CAST(ts AS DATE)` and
`date_format`/`from_unixtime` with a date-only pattern as order-preserving and
re-applies them to the file's min/max stats, which assumes the local date never
runs backwards. It does where the clocks fell back across midnight: in
`America/St_Johns` on 2009-11-01 they went from 00:01 back to 23:01 of the
previous day, so 02:30:30Z renders as `2009-11-01` but the later 02:40Z as
`2009-10-31`, and a file with those two rows is pruned for either date.
The window is narrow: a file is pruned wrongly only if it holds a row from
between midnight and the fall-back (one minute in St_Johns) and a row from the
repeated hour, one of them being its min or max. Only historical timestamps are
affected: tzdb 2026a records such fall-backs in `America/St_Johns` and
`America/Goose_Bay` every year from 1987 to 2010, in `America/Moncton` from
1993 to 2006, and in `Antarctica/Casey` (2010), `Pacific/Guam` (1969),
`America/Phoenix` (1944) and the Alaska zones (1867). Partition stats skip
these filters, so only file pruning is affected.
`to_date(ts)` reaches data skipping as `CAST(ts AS DATE)`, and
`date_add(CAST(ts AS DATE), 1) = DATE'2009-11-01'` is pruned the same way.
Spark itself rewrites a comparison of the cast with a date literal into a range
on `ts` (`UnwrapCastInBinaryComparison`), so the cast reaches data skipping in
shapes such as `IN` with several dates; that rewrite makes the same assumption,
and `CAST(ts AS DATE) = DATE'2009-10-31'` returns no rows here even with data
skipping off, which is for Spark to fix.
Follow-up to #20520, whose fix (#20521) keeps date-only patterns on this
assumption, as #20513 does for `CAST(ts AS DATE)` ("the date does not repeat").
Part of #20449.
**What you expected:**
The matching row returned.
**Steps to reproduce:**
```sql
SET spark.sql.session.timeZone = UTC;
CREATE TABLE t (id INT, ts TIMESTAMP, sec BIGINT) USING hudi
TBLPROPERTIES (primaryKey = 'id',
'hoodie.metadata.index.column.stats.enable' = 'true');
INSERT INTO t VALUES (1, TIMESTAMP'2009-11-01 02:30:30', 1257042630), (2,
TIMESTAMP'2009-11-01 02:40:00', 1257043200);
SET spark.sql.session.timeZone = America/St_Johns;
SELECT id FROM t WHERE date_format(ts, 'yyyy-MM-dd') = '2009-10-31';
-- expected: 1 row (id = 2)
-- actual: no rows
SELECT id FROM t WHERE date_format(ts, 'yyyy-MM-dd') = '2009-11-01';
-- expected: 1 row (id = 1)
-- actual: no rows
SELECT id FROM t WHERE from_unixtime(sec, 'yyyy-MM-dd') = '2009-10-31';
-- expected: 1 row (id = 2)
-- actual: no rows
SELECT id FROM t WHERE CAST(ts AS DATE) IN (DATE'2009-10-30',
DATE'2009-10-31');
-- expected: 1 row (id = 2)
-- actual: no rows
```
With `hoodie.enable.data.skipping=false`, each query returns its row.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT); `release-1.2.1` and
`release-1.1.2` carry the same matcher arms and cast rule
**Query engine:** Spark 3.5
**Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`,
`hoodie.enable.data.skipping=true` (default), a `spark.sql.session.timeZone`
such as `America/St_Johns`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
