alamb commented on PR #24426: URL: https://github.com/apache/datafusion/pull/24426#issuecomment-5443606081
## Local Q25 results for the five combinations Five `--profile=profiling` builds of `datafusion-cli` (one per combination in https://github.com/apache/datafusion/pull/24426#issuecomment-5438472525), each run at its own defaults against `hits_partitioned` with the harness-matched setup (`binary_as_string=true`, `hits` view with the `EventDate` cast). 3 interleaved rounds × 10 measured runs per binary after 2 warmups; per-round averages agreed within ~2ms, stdev ≤3.5ms. Q25: `SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;` | Combination | min ms | avg ms | vs main | |---|---|---|---| | 1. main (merge-base `a38bb10`) | 81.0 | 84.2 | 1.00x | | 2. + `progressive_io`, `pushdown_filters` on | 105.0 | 108.4 | 1.29x | | 3. + `reorder_filters` on | 102.0 | 106.6 | 1.27x | | 4. + gate from #23420 | 110.0 | 113.1 | **1.34x** | | 5. + [arrow-rs#10852](https://github.com/apache/arrow-rs/pull/10852) (via [arrow-rs#10885](https://github.com/apache/arrow-rs/pull/10885) pin) | 97.0 | 101.9 | **1.21x** | Step-by-step isolation: 1. **Pushdown itself costs ~24ms (1.29x) on Q25** — consistent with the earlier profiling (selection construction + cache-replay concat + assembly, see [arrow-rs#10774](https://github.com/apache/arrow-rs/issues/10774) / [arrow-rs#10776](https://github.com/apache/arrow-rs/issues/10776)). 2. **`reorder_filters` is neutral here** (-2ms, borderline noise): Q25 has a single static conjunct. 3. **The gate costs ~6.5ms on Q25** (106.6 → 113.1, reproduced in all three rounds). Q25 takes the gate's *dynamic-filter bypass* path, so the heuristic itself decides nothing here — the earlier samply profiles showed `FilterExec` frames on the gate binary that are absent without the gate, suggesting the gate's per-filter parent responses change where the TopK dynamic filter lands (`FilterExec` above the scan vs absorbed). Needs investigation in `try_pushdown_filters` — this looks like a fixable interaction, not an inherent cost of the heuristic. 4. **arrow-rs#10852 recovers 11.2ms** (113.1 → 101.9), about 40% of the pushdown penalty — matching the prediction from the profile (~100ms CPU/query of boolean→selectors→mask churn spread over ~10 effective cores). Nice work @hhhizzz! Remaining ~18ms gap vs main: ~6ms from the gate/TopK placement issue above; the bulk of the rest matches the `CachedArrayReader` replay concat ([arrow-rs#10774](https://github.com/apache/arrow-rs/issues/10774), ~170ms CPU/query). Next: the same five-way sweep across all 43 ClickBench queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
