rangareddy commented on issue #17399: URL: https://github.com/apache/hudi/issues/17399#issuecomment-5351245701
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9343). **Findings: substantial progress landed, but the specific refactor this ticket tracks is not done. Keeping it open.** What has landed since the ticket was filed, via the MDT write-path index abstraction (parent HUDI-9176): - #18348 (`a001ea87da7b`, merged 2026-07-08) introduced the `Indexer` abstraction and moved expression index initialisation into the engine-agnostic `ExpressionIndexer` in `hudi-client-common`. - #18372 (`2baa29b14d37`, merged 2026-07-08) moved the update/clean/restore flows onto the same abstraction. Before that, `getExpressionIndexRecords` was an abstract method on `HoodieBackedTableMetadataWriter` overridden separately in `SparkHoodieBackedTableMetadataWriter`, `SparkHoodieBackedTableMetadataWriterTableVersionSix`, `FlinkHoodieBackedTableMetadataWriter` and `JavaHoodieBackedTableMetadataWriter`. Today the orchestration lives once in `ExpressionIndexer`, and only record generation is delegated per engine through `EngineIndexerSupport#generateExpressionIndexRecords` (`SparkIndexerSupport` implements it, `UnsupportedEngineIndexerSupport` throws `HoodieNotSupportedException` for Flink/Java). However, that delegating interface is exactly the interim shim this ticket was filed to remove. From the review comment that created HUDI-9343 ([#12983 (comment)](https://github.com/apache/hudi/pull/12983#discussion_r2060984146)): > Makes sense. I rename the class to `ExpressionIndexRecordGenerator`. Eventually there should only be an engine-based expression evaluator interface that should exist; given that requires more refactoring on the expression index (HUDI-9343), I'll keep this interface to help isolate the refactoring of the core MDT writer logic. `ExpressionIndexRecordGenerator` was renamed into `EngineIndexerSupport#generateExpressionIndexRecords`, but the shape is the same: the engine still returns finished `HoodieRecord`s rather than just evaluating an expression. Current state on `master`: - `SparkIndexerSupport#generateExpressionIndexRecords` still builds a Spark `Dataset<Row>` and calls `SparkMetadataWriterUtils.getExprIndexRecords(...)`, i.e. the whole column-stats/bloom-filter record-construction path remains Spark-side. - The only implementation of `HoodieExpressionIndex<S, T>` (now `hudi-common`, `org.apache.hudi.core.index.expression`) is `HoodieSparkExpressionIndex`, typed on Spark `Column`, backed by `ExpressionIndexSparkFunctions`. - There is no engine-agnostic expression evaluator; Flink and Java still cannot build an expression index. (`org.apache.hudi.source.ExpressionEvaluators` in `hudi-flink` is an unrelated read-path/source-pruning class.) Remaining work: introduce the engine-based expression evaluator interface so the record construction moves into common code and only expression evaluation is engine-specific, then retire `EngineIndexerSupport#generateExpressionIndexRecords` and replace `UnsupportedEngineIndexerSupport` with real Flink/Java support. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
