voonhous opened a new issue, #20633: URL: https://github.com/apache/hudi/issues/20633
Part of #20625. Line references are against master `db90a0da`. ### Problem The merge path reads the base file with `enablePredicatePushDown=false` (`HudiPageSourceProvider.java:258`), so the Parquet tuple domain is `TupleDomain.all()`: static predicates and dynamic filters prune no row groups or pages. Partition-level dynamic filtering in `HudiSplitSource` still applies to MOR splits; the loss is pruning inside the file. ### Direction Enable base-file pushdown only where it cannot change the merged result, with Trino re-applying the filter after the merge: - record-key-only filters (what Spark and Flink push under MOR); - any filter under `COMMIT_TIME_ORDERING` with full-row log records. Keep it off for `EVENT_TIME_ORDERING`, partial updates and custom mergers, where a pruned base row can let a log record surface as a spurious row. ### Exit criteria - Pushdown enabled only for the safe cases; the eligibility decision is unit-tested. - Correctness tests where an update moves a row into and out of the predicate, and where a delete targets a row whose base row group would be pruned; merged results equal a no-pushdown read. - Benchmark evidence on multi-row-group base files (generator option) that physical input bytes drop for a selective predicate on scan `_rt`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
