zhuqi-lucas commented on issue #23263: URL: https://github.com/apache/datafusion/issues/23263#issuecomment-4899758481
Thanks for the deep-dive @RatulDawar — the framing is right. The 0.9s narrow-column baseline is effectively the theoretical lower bound: the filter/sort/limit phase only needs the filter and sort keys to decide *which* rows survive, so its cost should be independent of the SELECT list width. The 4s vs 0.9s gap is exactly the wide-decode work being paid for rows that never make it into the final 10. Before jumping straight to a full late-materialization rewrite, a couple of orthogonal knobs to layer in first — the remaining gap tells us how much LM actually buys: 1. **`datafusion.execution.parquet.pushdown_filters = true`** (default is `false`). This enables Parquet `RowFilter`, which decodes the filter column (`URL`) first, applies `LIKE '%google%'`, and only decodes the other 104 columns for surviving rows within each row group. For ~1% match rate this eliminates the wide-decode on ~99% of rows. 2. **TopK dynamic row-group pruning** ([#22450](https://github.com/apache/datafusion/pull/22450), merged) — the TopK threshold on `EventTime` is pushed back to the scan as a `DynamicFilter`, so row groups whose `EventTime` range is fully below the current top-10 threshold get skipped. Applies automatically for `SortExec + fetch` over Parquet. Splitting the 3.1s gap (4.0s − 0.9s): | Wasted work | Rows | Recovered by | |-------------|------|--------------| | Wide decode of filter-rejected rows | ~99M | `pushdown_filters=true` | | Wide decode of filter-survived but LIMIT-rejected rows | ~1M | Late materialization | So the sequence would be: turn on (1) and (2), re-measure Q23. If we land at ~1.5s, LM is worth ~0.6s. If we land at ~1.0s, LM buys ~0.1s — probably not worth the architectural weight. Concrete numbers before committing to the design. If we do proceed with LM, your direction is exactly what DuckDB does — [`src/optimizer/late_materialization.cpp`](https://github.com/duckdb/duckdb/blob/main/src/optimizer/late_materialization.cpp) ([PR #15692](https://github.com/duckdb/duckdb/pull/15692)) rewrites `SELECT * FROM t ORDER BY x LIMIT N` into `SELECT * FROM t WHERE rowid IN (SELECT rowid FROM t ORDER BY x LIMIT N)`, with `rowid` for Parquet being `(file_index, file_row_number)`. Once rewritten, the semi-join predicate becomes a `TableFilter` on `file_row_number` and inherits all existing row-group / page-index / RowFilter pushdown for free. DataFusion doesn't yet expose `file_row_number` as a projectable virtual column, so that plumbing is the first prerequisite. I've been working across all the moving parts here — Parquet reader / `ParquetOpener`, `DataSourceExec`, the physical optimizer, and the dynamic-filter path from #22450 — so I'm happy to drive the whole thing end-to-end: `file_row_number` virtual column exposure, the optimizer rewrite rule, plumbing the `file_row_number IN (...)` filter into the Parquet access plan, and the cost heuristic. Would be great to have you and @saadtajwar as reviewers and on integration testing / cost-model validation — let's confirm the pushdown numbers first, then I'll start opening PRs. Given the scope (Parquet reader plumbing → virtual column plumbing → physical optimizer rule → cost heuristic → benchmarks), I'd suggest **promoting this issue to an EPIC** with sub-issues for each work item — similar to [#23036](https://github.com/apache/datafusion/issues/23036) (Sort Pushdown) and [#22715](https://github.com/apache/datafusion/issues/22715) (GroupValuesColumn coverage). Happy to help draft the sub-issue breakdown once we agree on the direction. cc @alamb for visibility. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
