voonhous opened a new issue, #20538:
URL: https://github.com/apache/hudi/issues/20538
### Describe the problem
A column stats expression index on a function that depends on the session
time zone (for ex, `hour`, `date_format`) stores values rendered in the
writer's session time zone, and the zone is not recorded anywhere. A query run
in a different session time zone strips the function off its filter
(`ExpressionIndexSupport.extractQueryAndLiterals`,
`ExpressionIndexSupport.scala:445-489`) and compares the literal with those
stored values. Files holding matching rows are pruned, so the query silently
returns too few rows.
`git grep -i "timezone\|zoneId"` over `HoodieIndexDefinition`,
`ExpressionIndexSparkFunctions` and `ExpressionIndexSupport` finds nothing.
### Repro
```sql
SET spark.sql.session.timeZone=UTC;
CREATE TABLE hudi_table_expr_index (
ts TIMESTAMP, uuid STRING, rider STRING, driver STRING, fare DOUBLE,
city STRING
) USING HUDI
PARTITIONED BY (city)
TBLPROPERTIES (
primaryKey = 'uuid',
preCombineField = 'ts',
hoodie.metadata.index.column.stats.column.list = 'fare'
);
INSERT INTO hudi_table_expr_index VALUES
(TIMESTAMP'2023-09-19
23:40:49','334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
(TIMESTAMP'2023-09-18
21:45:54','e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70,'san_francisco'),
(TIMESTAMP'2023-09-18
19:14:22','9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90,'san_francisco'),
(TIMESTAMP'2023-09-21
16:34:26','1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
(TIMESTAMP'2023-09-23
09:42:17','e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'),
(TIMESTAMP'2023-09-22
09:53:40','7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40,'sao_paulo'),
(TIMESTAMP'2023-09-19
02:38:07','3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06,'chennai'),
(TIMESTAMP'2023-09-19
01:33:19','c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
CREATE INDEX ts_hour ON hudi_table_expr_index USING column_stats(ts)
options(expr='hour');
SET spark.sql.session.timeZone=America/Los_Angeles;
SELECT city, uuid FROM hudi_table_expr_index WHERE hour(ts) = 18;
-- returns no rows; chennai c8abbe79 (01:33:19Z, 18:33 PDT) matches
```
One base file per partition. Spark 3.5, master `0b696f915b67`,
`hoodie.enable.data.skipping=true`. Expected = the same filter over an
unindexed copy of the rows. In UTC every case below is correct.
| index | filter, session zone America/Los_Angeles | rows returned |
expected |
| --- | --- | --- | --- |
| `hour` | `hour(ts) = 18` | none | chennai/c8abbe79 |
| `hour` | `hour(ts) = 2` | none | sao_paulo/7a84095f, sao_paulo/e3cf430c |
| `date_format`, `format='yyyy-MM-dd'` | `date_format(ts, 'yyyy-MM-dd') =
'2023-09-18'` | san_francisco x2 | san_francisco x2, chennai x2 |
The other zone-dependent index functions (`year`, `month`, `day`,
`from_unixtime`, `unix_timestamp`, `to_date`, `to_timestamp`) take the same
path; only `hour` and `date_format` were run. Every commit also computes index
values in its own writer's session time zone, so writers in different zones mix
zones within one index (from reading the code, not run).
### Suggested fix
Record the session time zone in the index definition when the index is
created, and compute index values in that zone on every write. On read, use the
index only when the query expression's `timeZoneId` resolves to the same zone
(or to the same fixed offset), and otherwise skip it.
Related: #20537 (same matching step), #20255 / #20256, #20449.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
