zhuqi-lucas commented on PR #23696:
URL: https://github.com/apache/datafusion/pull/23696#issuecomment-5352863385
Thanks @adriangb — this decomposition is great, and the lazy-metric point in
particular is a real win (it kills the semver flag *and* the ~34 EXPLAIN
ANALYZE baseline regenerations in one move). Let's do it.
**PR 1 is up: #24509** — landed `strip_empty_row_groups` with the generic
signature only (no `fully_matched` parameter), wired into
`PreparedAccessPlan::prepare`, and updated the `sort.rs` reverse test to assert
the all-skipped plan strips to an empty plan. Closes #24287. Added two focused
unit tests for the strip itself. No `fully_matched`, no public API, no new
metric, no slt churn.
Plan for the rest, following your dependency order:
- **PR 2** (new issue) — prebuild row-filter candidates once per file;
reimplement the public `build_row_filter` on top of the two new functions with
the parity you called out (same conjunct order under `reorder_predicates`, same
metric wiring: shared `pushdown_rows_pruned`, only the last predicate counts
`pushdown_rows_matched`), and surface `reassign_expr_columns` errors once at
open time. I'll file the issue and put it up next.
- **PR 3** — extract `InitialDecoderState` (#24286), before the feature so
it adds fields to a struct instead of reshaping a tuple.
- **PR 4** (#23696) — the feature, rebased on all three, with the lazy
metric registration + the diff-noise revert + the single `Vec<{index,
fully_matched}>` you suggested.
One note: `ParquetAccessPlan::fully_matched` (and the
`OverallRowSelectionCursor` refactor) already appear to have landed on `main`,
so some of the groundwork is in place. I'll keep PR 4 focused on what's
genuinely new.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
