simonvandel commented on PR #9346:
URL:
https://github.com/apache/arrow-datafusion/pull/9346#issuecomment-1968588865
I was curious on how this affected planning performance. Here are my results
comparing 0c46d7fa105fddc4a35a4c99e4aa2a063d967abb (parent of first commit in
this PR) with a8fac85429f519ea5ae258a2a6425eaa9ab333c8.
Seems like there are quite big regressions in physical planning. But I may
have messed up.
<details>
<summary>Results</summary>
```
mold -run cargo bench --bench sql_planner --profile release-nonlto --
--baseline=0c46d7fa105fddc4a35a4c99e4aa2a063d967abb
Running benches/sql_planner.rs
(/home/svs/code/arrow-datafusion/target/release-nonlto/deps/sql_planner-355e1e15827c5f11)
Gnuplot not found, using plotters backend
logical_select_one_from_700
time: [576.63 µs 577.55 µs 578.47 µs]
change: [+1.0656% +1.7066% +2.3733%] (p = 0.00 <
0.05)
Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
1 (1.00%) low severe
3 (3.00%) high mild
2 (2.00%) high severe
physical_select_one_from_700
time: [4.0361 ms 4.0488 ms 4.0629 ms]
change: [-1.9349% -1.3951% -0.9091%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe
logical_trivial_join_low_numbered_columns
time: [586.18 µs 587.20 µs 588.35 µs]
change: [-0.4208% +1.0724% +3.2859%] (p = 0.32 >
0.05)
No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high severe
logical_trivial_join_high_numbered_columns
time: [627.61 µs 628.75 µs 629.94 µs]
change: [-1.5180% -1.0025% -0.5763%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
Benchmarking logical_aggregate_with_join: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 5.5s, enable flat sampling, or reduce sample count to 60.
logical_aggregate_with_join
time: [1.0842 ms 1.0866 ms 1.0893 ms]
change: [-1.5018% -0.9371% -0.4389%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
physical_plan_tpch_q1 time: [7.8999 ms 7.9398 ms 7.9805 ms]
change: [+1113.8% +1122.0% +1129.1%] (p = 0.00 <
0.05)
Performance has regressed.
physical_plan_tpch_q2 time: [11.743 ms 11.806 ms 11.878 ms]
change: [+1622.6% +1632.4% +1644.4%] (p = 0.00 <
0.05)
Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe
physical_plan_tpch_q3 time: [3.8676 ms 3.8778 ms 3.8886 ms]
change: [+625.22% +628.60% +631.62%] (p = 0.00 <
0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
9 (9.00%) high mild
4 (4.00%) high severe
physical_plan_tpch_q4 time: [2.9474 ms 2.9549 ms 2.9635 ms]
change: [+442.21% +445.51% +448.46%] (p = 0.00 <
0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
physical_plan_tpch_q5 time: [5.8592 ms 5.8881 ms 5.9182 ms]
change: [+903.01% +912.33% +920.23%] (p = 0.00 <
0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild
Benchmarking physical_plan_tpch_q6: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 9.3s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q6 time: [1.8279 ms 1.8300 ms 1.8322 ms]
change: [+358.95% +361.52% +365.24%] (p = 0.00 <
0.05)
Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) high mild
3 (3.00%) high severe
physical_plan_tpch_q7 time: [8.2906 ms 8.3330 ms 8.3757 ms]
change: [+950.61% +957.43% +963.46%] (p = 0.00 <
0.05)
Performance has regressed.
physical_plan_tpch_q8 time: [11.821 ms 11.870 ms 11.920 ms]
change: [+1331.7% +1339.7% +1347.4%] (p = 0.00 <
0.05)
Performance has regressed.
physical_plan_tpch_q9 time: [8.8488 ms 8.8885 ms 8.9294 ms]
change: [+1161.1% +1169.6% +1177.0%] (p = 0.00 <
0.05)
Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
physical_plan_tpch_q10 time: [6.0025 ms 6.0278 ms 6.0543 ms]
change: [+829.09% +837.51% +845.36%] (p = 0.00 <
0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
7 (7.00%) high mild
physical_plan_tpch_q11 time: [4.8230 ms 4.8421 ms 4.8622 ms]
change: [+716.64% +721.76% +726.08%] (p = 0.00 <
0.05)
Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe
physical_plan_tpch_q12 time: [3.8354 ms 3.8458 ms 3.8573 ms]
change: [+553.06% +556.86% +560.04%] (p = 0.00 <
0.05)
Performance has regressed.
Found 15 outliers among 100 measurements (15.00%)
10 (10.00%) high mild
5 (5.00%) high severe
physical_plan_tpch_q13 time: [2.4938 ms 2.4991 ms 2.5052 ms]
change: [+420.96% +422.92% +424.70%] (p = 0.00 <
0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
physical_plan_tpch_q14 time: [3.2362 ms 3.2430 ms 3.2510 ms]
change: [+527.89% +530.83% +533.51%] (p = 0.00 <
0.05)
Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) high mild
3 (3.00%) high severe
physical_plan_tpch_q16 time: [4.9522 ms 4.9715 ms 4.9919 ms]
change: [+840.97% +846.02% +850.48%] (p = 0.00 <
0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild
physical_plan_tpch_q17 time: [4.7371 ms 4.7556 ms 4.7757 ms]
change: [+848.11% +853.56% +858.89%] (p = 0.00 <
0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe
physical_plan_tpch_q18 time: [5.4681 ms 5.5619 ms 5.6655 ms]
change: [+851.77% +869.08% +890.15%] (p = 0.00 <
0.05)
Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
8 (8.00%) high mild
2 (2.00%) high severe
physical_plan_tpch_q19 time: [9.6241 ms 9.6606 ms 9.6985 ms]
change: [+1401.0% +1409.1% +1416.4%] (p = 0.00 <
0.05)
Performance has regressed.
physical_plan_tpch_q20 time: [6.1281 ms 6.1519 ms 6.1765 ms]
change: [+914.60% +921.14% +926.79%] (p = 0.00 <
0.05)
Performance has regressed.
physical_plan_tpch_q21 time: [9.0428 ms 9.0806 ms 9.1191 ms]
change: [+991.69% +997.40% +1003.3%] (p = 0.00 <
0.05)
Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
physical_plan_tpch_q22 time: [4.3348 ms 4.3436 ms 4.3534 ms]
change: [+532.03% +534.80% +537.16%] (p = 0.00 <
0.05)
Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe
physical_plan_tpch_all time: [12.755 ms 12.775 ms 12.796 ms]
change: [-0.8907% -0.6753% -0.4745%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
