zhuqi-lucas commented on PR #23369: URL: https://github.com/apache/datafusion/pull/23369#issuecomment-4904400037
🎉 The adaptive gate transformed the ClickBench picture: | | Pushdown default only | + Adaptive gate | |---|---|---| | Queries Faster | 6 | 5 | | Queries Slower | **20** | **4** | | Queries No Change | 17 | **34** | | Total Time | 17231 ms | **16735 ms** (14% below HEAD's 19549 ms) | | Avg Time | 400 ms | **389 ms** | Concretely for the queries the gate targeted (via `non_filter_projected_cols < 3` → decline pushdown): - Q10, Q11, Q12, Q13, Q14, Q24, Q26, Q27, Q40 — all previous 1.20-1.46× regressions are now **no change** vs HEAD. - Q23 — still **+21.12× faster**. - Q22 — win lost (835 → 992 ms, back to no change vs HEAD) since it has `non_filter=1` but the compound `Title LIKE + URL NOT LIKE + SearchPhrase <> ''` is jointly selective. Heuristic doesn't see this yet. Remaining regressions: - Q19 (1.31×), Q26 (1.09×), Q29 (1.10×), Q41 (1.33×) — Q19/Q26/Q29 have very high variance windows in this run (±11-15 ms on ~30-40 ms queries) so are likely noise. Q41 is real and worth investigating. Given this shape, I think we're getting close to "flip default true" being defensible. Next steps I'm considering: 1. Confirm Q41 root cause (likely still Row Filter overhead with `DontCountHits = 0 AND IsRefresh = 0` etc. selectivity). 2. A better heuristic for Q22-shape queries (multiple selective conjuncts) — perhaps use PruningPredicate's selectivity estimate per conjunct, or a runtime adaptive path. 3. Tune the threshold (currently hardcoded to 3) — maybe expose as a config knob. Interested in whether reviewers see any obvious extension of the heuristic that would recover Q22 without reopening the Q10-shape regressions. cc @alamb @adriangb — this may be complementary to the adaptive placement work in #22883. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
