voonhous opened a new issue, #20537:
URL: https://github.com/apache/hudi/issues/20537
### Describe the problem
A column stats expression index prunes files only for `=` (and for `IN` when
the source column needs no cast). Range predicates on the indexed expression
(`>`, `>=`, `<`, `<=`, `BETWEEN`) never use it, so every file is scanned.
`ExpressionIndexSupport.extractQueryAndLiterals` picks the filter to
evaluate through `RecordLevelIndexSupport.filterQueryWithRecordKey`
(`RecordLevelIndexSupport.scala:210-281`). That helper was written for record
key lookups and accepts only `EqualTo`, `In`, and `And` of those. Any other
filter falls to `case _ => (Option.empty, false)`, so no expression index query
is found and both file pruning (`computeCandidateFileNames`) and partition
pruning (`prunePartitions`) are skipped.
`IN` has a second gap: its branch (`RecordLevelIndexSupport.scala:228-250`)
only accepts a bare `AttributeReference`, while `EqualTo` goes through
`getAttributeLiteralTuple`, which also unwraps a `Cast`. With `ts STRING`,
`hour(ts)` is `hour(cast(ts as timestamp))`, so `hour(ts) = 16` prunes but
`hour(ts) IN (16, 19)` does not.
The stats filter compiler already handles ranges, so this looks limited to
the extraction step (plus `composeIndexSchema`, which takes the stats type from
the literal only for `EqualTo` and `In`).
### Repro
```sql
CREATE TABLE hudi_table_expr_index (
ts STRING, uuid STRING, rider STRING, driver STRING, fare DOUBLE, city
STRING
) USING HUDI
PARTITIONED BY (city)
TBLPROPERTIES (
primaryKey = 'uuid',
preCombineField = 'ts',
hoodie.metadata.index.column.stats.column.list = 'fare'
);
INSERT INTO hudi_table_expr_index VALUES
('2023-09-19
23:40:49','334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
('2023-09-18
21:45:54','e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70,'san_francisco'),
('2023-09-18
19:14:22','9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90,'san_francisco'),
('2023-09-21
16:34:26','1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
('2023-09-23
09:42:17','e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'),
('2023-09-22
09:53:40','7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40,'sao_paulo'),
('2023-09-19
02:38:07','3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06,'chennai'),
('2023-09-19
01:33:19','c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
CREATE INDEX ts_hour ON hudi_table_expr_index USING column_stats(ts)
options(expr='hour');
SELECT city, _hoodie_file_name, fare FROM hudi_table_expr_index WHERE
hour(ts) > 12;
```
One base file per partition. `hour(ts)` ranges: chennai [1, 2],
san_francisco [16, 23], sao_paulo [9, 9]. Files scanned (`numFiles` of the
scan) with `hoodie.enable.data.skipping=true`, Spark 3.5, master `0b696f915b67`:
| filter | files that can match | scanned, `ts STRING` | scanned, `ts
TIMESTAMP` |
| --- | --- | --- | --- |
| `hour(ts) = 16` | 1 | 1 | 1 |
| `hour(ts) IN (16, 19)` | 1 | 3 | 1 |
| `hour(ts) > 12` | 1 | 3 | 3 |
| `hour(ts) BETWEEN 12 AND 24` | 1 | 3 | 3 |
| `hour(ts) >= 16` | 1 | 3 | 3 |
| `hour(ts) < 5` | 1 | 3 | 3 |
| `NOT hour(ts) <= 12` | 1 | 3 | 3 |
Results are correct in every case; only pruning is lost.
### Suggested fix
Give the expression index its own filter extraction instead of reusing the
record key helper: accept binary comparisons on the indexed expression, unwrap
a `Cast` on the source column for `In` as `EqualTo` already does, and derive
the stats type in `composeIndexSchema` for comparisons too.
Related: #20255 / #20256 (the same method matches filters on a different
function of the column), #17489.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
