voonhous opened a new pull request, #20540: URL: https://github.com/apache/hudi/pull/20540
### Describe the issue this Pull Request addresses Closes #20539 `ExpressionIndexSupport` picked the filters for an expression index with the record key helper and an `attributeFetcher` that stripped any unary function, then guessed the stats type from the literal. So: - a filter on another function of the column, or on the bare column, read the index's stats (#20255, #17489), - `IN` over a cast column did not prune, - only the last matching index was used. ### Summary and Changelog The expression index becomes a `RangeSource` of the data skipping stages (#20524): - `IndexedExpression.matches(term)`: is the term the index's `f(col)`? One case per function, checking the function name and options. The column may carry Spark's implicit cast. - `ExpressionIndexSource`: a matching term reads the index's min / max; any other term has no range, so the file is kept. - Each matching index prunes, and the candidate sets intersect. The stats type comes from the matched term. - `TermRange`: a range whose off-chain column the source has no stats for is dropped. - Deleted: `attributeFetcher`, `extractQueryAndLiterals`, `getExpressionIndexPartitionAndLiterals`, `filterQueriesWithFunctionalFilterKey`. Only `=` and `IN` read an expression index, as before; #20537 lifts that. ### Impact Fixes wrong results for filters on another function or the bare column. `IN` over a cast column and multi-index filters prune more. ### Risk Level low. See details. <details> <summary>Tests and risk</summary> - `TestExpressionIndex#Test Expression Index Reads Only Filters On Its Own Expression` (the #20537 table): `=`, `IN` over a STRING column, two indexes intersecting, `minute(ts)` and bare `ts` not reading the hour stats, ranges still not reading the index. On master it fails at `hour(ts) IN (16, 19)`: expected 1 file scanned, was 3. - `TestDataSkippingStages`: matching per function and options, identity, compiled filters, the off-chain guard. - Spark 3.5: `TestExpressionIndex` (27), `TestDataSkippingStages` (27), `TestDataSkippingUtils` (143), `TestDataSkippingGoldenFilters` pass. Golden output is unchanged, so column stats translation is untouched. - Overlaps #20256 (same method). Whichever lands second rebases; its tests stay. </details> ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
