voonhous opened a new issue, #20539: URL: https://github.com/apache/hudi/issues/20539
## Summary Make the Spark expression index a `RangeSource` of the data skipping stages from #20524, and delete the separate filter extraction in `ExpressionIndexSupport`. This is the "expression index as range source" follow-up in the plan of #20527, and it lands before the fix for #20537. ## Problem today Before the stats filter is built, `ExpressionIndexSupport.extractQueryAndLiterals` decides which filters reach the index, through code that was not written for it: - **Which filter forms:** it reuses `RecordLevelIndexSupport.filterQueryWithRecordKey`, written for record key lookups: only `=`, `IN` and `AND` of them. Every range is dropped (#20537), and `IN` needs a bare column, so `hour(ts) IN (...)` on a STRING `ts` is dropped too. - **Which index:** `attributeFetcher` strips any `UnaryExpression` and checks per-function options, never the function name (#20255); its catch-all lets a bare column through (#17489). Two loops keep only the last matching filter and the last matching index. - **Which stats type:** `composeIndexSchema` guesses it from the `=` / `IN` literal. ## Design The compiler from #20524 already asks its `RangeSource` about a whole term before taking any rule hop (`TermRange.derive`). So the index only has to answer one question: is this term my `f(col)`? - `IndexedExpression(definition).matches(term)`: one case per supported function, naming the function and checking its options; the column may carry Spark's implicit cast. Anything else, including a bare column for a non-identity index, has no range, so the file is kept. - `ExpressionIndexSource(indexed)`: returns the index's min / max stats for a matching term. AND / OR / NOT, sides and bound checks stay in the compiler, as for column stats. - Every expression index a filter term matches prunes, and the candidate sets intersect. The stats type comes from the matched term. Under RFC-88 section 4.3, this becomes the `ExpressionIndexLookup`, and the Catalyst cases in `matches` become `SparkTranslator` cases.   ## Plan 1. This refactor, keeping today's operators (`=` and `IN`) behind a gate. Filters on another function or the bare column stop using the index, `IN` over a cast column prunes, and several indexes combine. 2. #20537: delete the gate, add range tests. 3. #20538 (session time zone) and #19480 (to_date / to_timestamp shapes) each become a case in `matches`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
