adriangb commented on PR #24086: URL: https://github.com/apache/datafusion/pull/24086#issuecomment-5805453159
# Benchmark summary: `16c5ba3` (`scan_plan` revision) `DF_FETCH_POLICY=streaming`, `DF_FETCH_BUDGET=100MB`. GKE runs: adriangbot `c4a-highmem-16`, compared with merge-base `95bb0a0`. "Query total" is the sum of per-query times from the bot. Bot wall times are rounded to 5 s. ## GKE, with `SIMULATE_LATENCY` | suite | query total base → branch | speedup | faster / slower / same | wall base → branch | peak memory base → branch | previous revision (wall) | |---|---|---|---|---|---|---| | tpch_sf1 | 18.91 s → 10.90 s | **1.73x** | 21 / 0 / 1 | 110 s → 65 s | 825 MiB → 944 MiB | 1.69x | | tpch_sf10 | 119.19 s → 16.93 s | **7.04x** | 22 / 0 / 0 | 625 s → 95 s | 4.0 GiB → 3.6 GiB | 6.35x | | tpcds_sf1 | 65.59 s → 65.09 s | 1.01x | 11 / 3 / 85 | 370 s → 370 s | 1.0 GiB → 1.1 GiB | 1.00x | | clickbench_partitioned | 86.33 s → 44.07 s | **1.96x** | 36 / 3 / 4 | 450 s → 235 s | 17.0 GiB → 17.4 GiB | 1.01x (`pipelined`: 1.33x) | Slower queries: tpcds Q30 1.08x, Q76 1.06x, Q83 1.05x. clickbench Q36 1.11x, Q38 1.12x, Q41 1.07x. ## GKE, without latency | suite | query total base → branch | change | faster / slower / same | wall base → branch | peak memory base → branch | |---|---|---|---|---|---| | tpch_sf1 | 753.8 ms → 749.2 ms | 1.01x | 0 / 0 / 22 | 5 s → 5 s | 1.3 GiB → 1.4 GiB | | tpcds_sf1 | 9.30 s → 9.51 s | 0.98x | 0 / 1 / 98 | 50 s → 50 s | 2.2 GiB → 2.0 GiB | | clickbench_partitioned | 19.55 s → 20.18 s | 0.97x | 3 / 6 / 34 | 105 s → 105 s | 16.6 GiB → 18.5 GiB | Slower queries: tpcds Q72 1.10x. clickbench Q4 1.16x, Q22 1.05x, Q23 1.13x, Q31 1.18x, Q34 1.14x, Q35 1.06x. ## Local ClickBench: files as published vs rewritten with a page index Files: `hits_0..2.parquet`, 10 row groups. The rewrite uses pyarrow `write_page_index=True` and keeps the same row groups. Simulated store: 50 ms per request + 500 MB/s. 8 partitions, 100 MB window. Each query ran 5 iterations of 4 variants: `off`, `streaming`, `off`, `streaming`. The order rotates every iteration. Values are medians. q36–q42 fail under both policies (`EventDate` is `UInt16` in these files) and are excluded, which leaves 36 queries. A/A (same policy twice): | dataset | `off` A/A total | `streaming` A/A total | per-query A/A deviation, median / p95 | |---|---|---|---| | no page index | 0.96 | 1.09 | 5.6% / 23.9% | | page index | 0.98 | 1.04 | 2.9% / 16.5% | A/B (`off` vs `streaming`, both runs of each policy pooled): | dataset | `off` total | `streaming` total | speedup | per-query median | faster / slower / same (beyond A/A p95) | GETs `off` / `streaming` | MB fetched `off` / `streaming` | previous revision | |---|---|---|---|---|---|---|---|---| | no page index | 7.13 s | 5.54 s | **1.29x** | 1.51x | 30 / 0 / 6 | 509 / 509 | 1673 / 1678 | 1.00x | | page index | 6.36 s | 5.00 s | **1.27x** | 1.52x | 31 / 1 / 4 | 512 / 533 | 1583 / 1680 | 1.54x | Slower: page index q28 0.80x (549 → 690 ms). Below 1.0x on both datasets: q28, q33, q34 (0.85x–0.97x). Per-query speedup, page index: | q0 | q1 | q2 | q3 | q4 | q5 | q6 | q7 | q8 | q9 | q10 | q11 | |---|---|---|---|---|---|---|---|---|---|---|---| | 1.36 | 1.88 | 1.76 | 1.94 | 1.72 | 1.37 | 1.55 | 1.95 | 1.64 | 1.51 | 1.80 | 1.71 | | q12 | q13 | q14 | q15 | q16 | q17 | q18 | q19 | q20 | q21 | q22 | q23 | |---|---|---|---|---|---|---|---|---|---|---|---| | 1.35 | 1.22 | 1.28 | 1.69 | 1.37 | 1.37 | 1.37 | 1.76 | 1.53 | 1.50 | 1.16 | 1.10 | | q24 | q25 | q26 | q27 | q28 | q29 | q30 | q31 | q32 | q33 | q34 | q35 | |---|---|---|---|---|---|---|---|---|---|---|---| | 1.55 | 1.55 | 1.57 | 1.52 | 0.80 | 1.69 | 1.41 | 1.31 | 1.41 | 0.97 | 0.92 | 1.68 | ## Notes - Files without a page index now improve too: the plan has one entry per column chunk, so read-ahead overlaps decode across row groups. This is the main change behind ClickBench going from 1.01x to 1.96x on GKE (its published files have no page index). - Streaming loads the page index; `off` skips it when pruning cannot use it. This is included in the GET and MB columns. - Open regressions: 3–18% on some ClickBench queries without latency, and higher peak memory on ClickBench without latency (+11%). The window is per stream, not a `MemoryPool` reservation. Runs: [tpch](https://github.com/apache/datafusion/pull/24086#issuecomment-5805055558), [tpch10](https://github.com/apache/datafusion/pull/24086#issuecomment-5805151834), [tpcds](https://github.com/apache/datafusion/pull/24086#issuecomment-5805151513), [clickbench](https://github.com/apache/datafusion/pull/24086#issuecomment-5805157506) with latency; [tpch](https://github.com/apache/datafusion/pull/24086#issuecomment-5805030160), [tpcds](https://github.com/apache/datafusion/pull/24086#issuecomment-5805043339), [clickbench](https://github.com/apache/datafusion/pull/24086#issuecomment-5805078251) without. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
