zhuqi-lucas commented on PR #23701: URL: https://github.com/apache/datafusion/pull/23701#issuecomment-5016276377
Followup based on local Q9 profile: Confirmed regression root cause is HashJoin **Partitioned-mode** dynamic filter running as a per-batch **PostScanFilter** when `pushdown_filters=false`. The predicate is a `CASE hash(col) % N WHEN pid THEN bounds ELSE lit(false) END` — per-row hash + modulo + CASE branch — and on high-match-rate joins it prunes almost nothing (downstream HashJoin hash lookup would eliminate the same rows). Local samply on TPC-H SF1 Q9 (release-nonlto): - baseline (HEAD~2): 0% self-time in `case::PartialResultIndex::merge_n`, 0% in `filter_native` - this PR before fix: **1.11%** and **1.34%** self-time respectively — regression signature **Fix (5a8ca02)**: skip `DynamicFilterPhysicalExpr`-containing conjuncts from `post_scan_conjuncts` in the `pushdown_filters=false` branch. `RowGroupPruner` (from #22450) still sees the full predicate via `prepared.predicate`, so **RG-level dynamic pruning continues to fire**. Downstream operator does the exact-equivalent filtering (hash lookup / sort heap). No correctness change. Local TPC-H SF1 (3 iters, release-nonlto), sum across Q1-Q22: - baseline (HEAD~2, pre-#22384): **608 ms** - PR before fix (per CI benchmark): **1032 ms** (1.70x slower) - PR + this fix: **577 ms** — 5% net ahead of baseline Q9 individually: 40 ms baseline → 80 ms PR → **34 ms PR + fix** (matches baseline warm-cache). Waiting on CI `run benchmarks` to confirm ClickBench pushdown / TPC-DS are also clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
