zhuqi-lucas commented on PR #23420: URL: https://github.com/apache/datafusion/pull/23420#issuecomment-4932872849
> This is great. I think we are very close @zhuqi-lucas > > I think we need to do the following : > > 1. To make this a config setting that can be disabled (so that people who want to force filter pushdown on (to keep the current behavior) can do so > 2. Add a note to the upgrade guide explaining how to get the original behavior > 3. Add some metric so it is clear from the profiling that this heuristic has happened > > I would also like to do some profiling of the query that appears consistently get slower > > ``` > │ QQuery 29 │ 41.17 / 45.16 ±7.39 / 59.93 ms │ 41.71 / 55.51 ±9.80 / 71.56 ms │ 1.23x slower │ > ``` Thanks @alamb , addressed all three points in 1ef39907: 1. Config opt-out: added datafusion.execution.parquet.pushdown_filter_narrow_projection_gate (default true). Set to false to restore the pre-existing unconditional pushdown. Proto + serde support round-trips through the physical plan. 2. Plan-display marker: when the gate declines, ParquetSource's fmt_extra output now includes pushdown_declined=narrow_projection so EXPLAIN / EXPLAIN ANALYZE shows it without needing to correlate config against per-query symptoms. 3. Upgrade guide: docs/source/library-user-guide/upgrading/55.0.0.md documents the new behavior and the opt-out. Kept the heuristic itself untouched (single hardcoded PUSHDOWN_MIN_NON_FILTER_COLS = 3, no tuning knob, no byte-weighting) per your complexity-budget note. I will also profile the Query 29. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
