adriangb opened a new pull request, #25683: URL: https://github.com/apache/datafusion/pull/25683
## Which issue does this PR close? - Part of #22883 (experimental "optional filters" stack, design notes: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt). - **Depends on #25673 and #25674.** Review only the top two commits. - A smaller re-implementation of #22698 on top of the shared optional filter pieces. #22698 stays open for its review history. ## Rationale for this change `FilterExec` is the second consumer of optional filters, and the place where conjunct order matters most. The review of #22698 asked for a much simpler version: one AND evaluator (ordinary `BinaryExpr`), a cost model that matches `BinaryExpr`'s real pre-selection rule, state scoped to one execution, deterministic tests with a mocked clock, and no unsafe unwraps. This PR starts from those points. ## What changes are included in this PR? **Commit 1: `FilterExec` skips non-selective optional conjuncts** (`datafusion.execution.optional_filter_mode`, from #25674) - `always` (default): the predicate is not split; exactly today's behavior. - `pruning_only`: optional conjuncts are not evaluated (`FilterExec` cannot prune). - `adaptive`: required conjuncts run as one ordinary `BinaryExpr` AND chain; each optional conjunct then runs behind its own `OptionalFilterGate`, and a paused gate costs nothing. - Pooled gate statistics are tied to the current `TaskContext`, so a new execution of the same plan starts fresh (the state-leak concern on #22698). `reset_state` also clears them. - Wrapped conjuncts do not feed equivalence classes or constants (`Optional(a = b)` is ignored by predicate analysis). - Metrics `optional_filter_rows_skipped` and `optional_filter_pauses`, registered only when a stream gates optional filters. **Commit 2: adaptive conjunct reordering** (`datafusion.execution.adaptive_filter_reordering`, default `false`), about 425 lines of logic versus 1621 in #22698: - Each stream runs the written predicate for 8 warm-up batches with per-conjunct counters, and does not change the tree shape during warm-up. - It ranks conjuncts by rows dropped per nanosecond (the #22698 key). It adopts a new order only if the estimate is at least 5% cheaper, using `BinaryExpr`'s pre-selection rule (≤ 20% true and no nulls) for the cost estimate. `PRE_SELECTION_THRESHOLD` is made `pub` (`#[doc(hidden)]`) so the two cannot drift. - An adopted order is built as a plain right-nested `BinaryExpr` chain. Otherwise the original predicate object is kept. - Volatile predicates are never reordered. A new `adaptive_reorders` metric. - The clock is injectable; state is per stream. ## What is the testing strategy for this PR? - Commit 1: identical results in all modes; a redundant optional conjunct pauses with exact pause and skip counts; a selective one never pauses; `NOT(Optional(x))` is never skipped; wrapped equalities add no equivalences; statistics are per context and reset by `reset_state`. - Commit 2: 9 mock-clock scenario tests that check the strategy on each batch, an end-to-end test, and `adaptive_filter_reordering.slt`. - `cargo test -p datafusion-physical-plan filter`, physical-plan and physical-expr lib tests, proto tests and the full sqllogictest suite pass. Known gaps: in `adaptive` optional mode, optional conjuncts run after the (possibly reordered) required chain and are not ranked with it. Reordering decides once per stream. ## Are there any user-facing changes? One new config option (`adaptive_filter_reordering`, default off) and new metrics. Default behavior is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
