zhuqi-lucas commented on PR #23369: URL: https://github.com/apache/datafusion/pull/23369#issuecomment-4904061095
Added an adaptive gate (`c61e14b`) — `ParquetSource::try_pushdown_filters` now declines pushdown when the projection has fewer than 3 columns not referenced by the filter. When it declines, the filter falls back to a `FilterExec` above the scan and the predicate is still injected into `ParquetSource` for stats/bloom/page-index pruning. Local Q10 EXPLAIN ANALYZE + timings (5 iters, 12 partitions): | | HEAD (default off) | Pushdown-default only | + gate | |---|---|---|---| | Q10 (non-filter=1) | 85 ms | 95 ms (1.11x slower) | **82 ms** | | Q11 (non-filter=2) | 81 ms | 116 ms (1.43x slower) | **73 ms** | | Q23 (non-filter=104) | 3107 ms | 318 ms (**21× faster**) | 340 ms (still pushed) | | Q41 (non-filter=6) | 14 ms | 20 ms (1.51x slower) | **11.5 ms** | | Q22 (non-filter=1) | 1007 ms | 835 ms (**+21% faster**) | 1150 ms — win **lost** (heuristic misses this) | `run benchmarks` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
