voonhous opened a new issue, #19480: URL: https://github.com/apache/hudi/issues/19480
**Describe the problem you faced** The expression-index filter matcher has the same optimizer-boundary bug that #19446/#19474 fixed for column-stats data skipping. `ExpressionIndexSupport.extractQueryAndLiterals`'s `attributeFetcher` matches the analysis-time `ParseToDate` / `ParseToTimestamp` shapes, but the `queryFilters` it receives are `HoodieFileIndex` dataFilters -- post-optimizer, because Hudi's pruning rule is injected via `injectOptimizerRule` and runs after the FinishAnalysis batch (ReplaceExpressions). For the explicit-format form the real shape is `GetTimestamp(col, fmt)`, which falls through to `case other => other` (the UnaryExpression arm does not match a BinaryExpression), and `RecordLevelIndexSupport.getAttributeLiteralTuple` then requires a bare `AttributeReference`, so no index match happens. Net effect: an expression index created with `expr='to_date', format=...` or `expr='to_timestamp', format=...` cannot be selected by a real SQL query. **Why no test catches it** Every case in `TestExpressionIndex` (around lines 1681-1714) builds the filter with `resolveExpr` -- analyzer only, no optimizer -- and calls `fileIndex.listFiles` directly via `verifyFilePruning`. The harness therefore feeds analysis-time shapes that production never delivers. This is the exact blind spot that hid #19446. **Suggested fix** 1. Lift the `fullyOptimize` helper added in #19474 (session optimizer over a `HoodieDummyExpressionHolder`) into a shared test util and run `verifyFilePruning` filters through it. Expect several existing `TestExpressionIndex` cases to go red -- that is the point. 2. Add a `GetTimestamp` arm to `attributeFetcher`, with the same soundness considerations #19474 applied to the data-skipping whitelist (literal format, ANSI/failOnError semantics, null on unparseable input). **Environment**: master (found during review of #19474; matcher logic verified by reading, end-to-end consequence deduced, not executed). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
