zhuqi-lucas opened a new issue, #24563: URL: https://github.com/apache/datafusion/issues/24563
Follow-up to #23696, per @adriangb's [decomposition proposal](https://github.com/apache/datafusion/pull/23696#issuecomment-5346992228) (PR 2). `RowFilterGenerator::build()` currently redoes `split_conjunction` + `FilterCandidateBuilder::build` + `reassign_expr_columns` **for every row group**. On any multi-row-group scan with a pushdown predicate, that per-RG tree walk and column resolution is wasted work. Move it to **once per file**: prebuild the row-filter candidate list when the file is opened, then instantiate a `RowFilter` per row group cheaply from the prebuilt list. Proposed shape (extracted from #23696, decoupled from `fully_matched`): - add `PrebuiltRowFilterCandidate` / `prebuild_row_filter_candidates` (once per file: tree walk + candidate construction) and `row_filter_from_prebuilt` (per RG: cheap bind against the fresh array readers); - delete `RowFilterGenerator`; - **reimplement the existing public `build_row_filter` on top of the two new functions** (its signature must stay — `parquet_nested_filter_pushdown` / `parquet_struct_filter_pushdown` benches depend on it). Parity to preserve: same conjunct order (`sort_unstable_by_key(required_bytes)` when `reorder_predicates`), same metric wiring (every predicate shares `pushdown_rows_pruned`, only the last predicate counts `pushdown_rows_matched`). One intended behavioural delta: `reassign_expr_columns` errors surface once at open time instead of being swallowed per-build by `log::debug!`. Needs nothing from `fully_matched`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
