zhuqi-lucas commented on PR #23420: URL: https://github.com/apache/datafusion/pull/23420#issuecomment-4958364417
Update on the TPCH regression from my previous comment (@alamb): Opened #23532 — caches the remapped expression inside `DynamicFilterPhysicalExpr::current()`. That method is called per batch on the RowFilter path and, for filters carrying a large `InListExpr` (HashJoin pushing down `col IN build_keys`), was dominated by `InListExpr::with_new_children` cloning the whole list on every call. samply on Q17 had that + its `drop_glue` at the top. **TPCH SF1 (10 iterations, `pushdown_filters=true`):** | | HEAD | #23532 | `pushdown=false` baseline | |---|---|---|---| | Q17 | 128 ms | **53 ms** (2.4×) | 55 ms | | Q18 | 69 ms | 56 ms | 49 ms | | Total (22) | 743 ms | 644 ms (**−13%**) | 531 ms | Q17 reaches parity with `pushdown=false` — DynamicFilter tax fully eliminated on that shape. Row counts identical pre/post across all 22 queries. This doesn't resolve the whole TPCH regression from the earlier comment. Remaining gap between #23532 (644 ms) and `pushdown=false` baseline (531 ms) is the fundamental two-phase-read tax on queries where selectivity + narrow non-filter projection don't repay the coordination overhead (Q12 is the archetype). That looks like it needs the full adaptive-filter direction — @adriangb's stack (#22384 → #22237) is the right shape but is stalled; happy to help push #22384 forward first since it's the foundation. ClickBench simulate-latency and tpcds numbers I'll re-run against #23532 as follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
